Linux process details

 
Here is the execution ps -ax in my 2.6 64 bit Linux. Here we try to explain what these processes are.

PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 init [5]        
    2 ?        S<     0:00 [migration/0]
    3 ?        SN     0:00 [ksoftirqd/0]
    4 ?        S<     0:00 [watchdog/0]
    5 ?        S<     0:00 [migration/1]
    6 ?        SN     0:00 [ksoftirqd/1]
    7 ?        S<     0:00 [watchdog/1]
    8 ?        S<     0:00 [events/0]
    9 ?        S<     0:00 [events/1]
   10 ?        S<     0:00 [khelper]
   27 ?        S<     0:00 [kthread]
   32 ?        S<     0:00 [kblockd/0]
   33 ?        S<     0:00 [kblockd/1]
   34 ?        S<     0:00 [kacpid]
  132 ?        S<     0:00 [cqueue/0]
  133 ?        S<     0:00 [cqueue/1]
  136 ?        S<     0:00 [khubd]
  138 ?        S<     0:00 [kseriod]
  210 ?        S      0:00 [khungtaskd]
  213 ?        S<     0:10 [kswapd0]
  214 ?        S<     0:00 [aio/0]
  215 ?        S<     0:00 [aio/1]
  364 ?        S<     0:00 [kpsmoused]
  396 ?        S<     0:00 [ata/0]
  397 ?        S<     1:22 [ata/1]
  398 ?        S<     0:00 [ata_aux]
  405 ?        S<     0:00 [scsi_eh_0]
  406 ?        S<     3:52 [scsi_eh_1]
  413 ?        S<     0:00 [kstriped]
  426 ?        S<     0:00 [ksnapd]
  441 ?        S<     0:28 [kjournald]
  474 ?        S<     0:00 [kauditd]
  516 ?        S<s    0:00 /sbin/udevd -d
 1577 ?        S<     0:00 [hd-audio0]
 1786 ?        S<     0:00 [kmpathd/0]
 1787 ?        S<     0:00 [kmpathd/1]
 1788 ?        S<     0:00 [kmpath_handlerd]
 1813 ?        S<     0:00 [kjournald]
 2007 ?        S<     0:00 [kondemand/0]
 2008 ?        S<     0:00 [kondemand/1]
 2025 ?        S<     0:00 [iscsi_eh]
 2069 ?        S<     0:00 [ib_addr]
 2079 ?        S<     0:00 [ib_mcast]
 2080 ?        S<     0:00 [ib_inform]
 2081 ?        S<     0:00 [local_sa]
 2085 ?        S<     0:00 [iw_cm_wq]
 2089 ?        S<     0:00 [ib_cm/0]
 2090 ?        S<     0:00 [ib_cm/1]
 2094 ?        S<     0:00 [rdma_cm]
 2112 ?        Ssl    0:00 brcm_iscsiuio
 2118 ?        Ss     0:00 iscsid
 2119 ?        S<Ls   0:00 iscsid
 2220 ?        Ss     0:00 mcstransd
 2502 ?        Ss     0:02 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0
 2553 ?        S<sl   0:00 auditd
 2555 ?        S<sl   0:00 /sbin/audispd
 2567 ?        S      0:02 stardict
 2571 ?        Ss     0:00 /usr/bin/esd -terminate -nobeeps -as 2 -spawnfd 24
 2573 ?        Ss     0:00 /usr/sbin/restorecond
 2584 ?        Ss     0:01 syslogd -m 0
 2587 ?        Ss     0:00 klogd -x
 2623 ?        Ss     0:00 portmap
 2640 ?        S      0:13 /usr/bin/python2.4 /usr/share/meld/meld
 2648 ?        Ss     0:00 rpc.statd
 2674 pts/9    S+     0:01 ssh rh8
 2675 pts/10   S+     0:01 ssh rh8
 2682 ?        S<     0:00 [rpciod/0]
 2683 ?        S<     0:00 [rpciod/1]
 2690 ?        Ss     0:00 rpc.idmapd
 2709 ?        Ssl    0:18 dbus-daemon --system
 2721 ?        Ss     0:00 /usr/sbin/hcid
 2727 ?        Ss     0:00 /usr/sbin/sdpd
 2750 ?        S<     0:00 [krfcommd]
 2792 ?        Ssl    0:01 pcscd
 2808 ?        Ss     0:00 /usr/bin/hidd --server
 2827 ?        Ssl    0:00 automount
 2860 ?        Ss     0:00 /usr/sbin/acpid
 2871 ?        Ss     0:00 ./hpiod
 2876 ?        S      0:00 python ./hpssd.py
 2891 ?        Ss     0:00 /usr/sbin/sshd
 2902 ?        Ss     0:00 cupsd
 2916 ?        SLs    0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
 2936 ?        Ss     0:00 sendmail: accepting connections
 2945 ?        Ss     0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
 2957 ?        Ss     0:00 gpm -m /dev/input/mice -t exps2
 2968 ?        Ss     0:00 crond
 3003 ?        Ss     0:00 xfs -droppriv -daemon

init: created in the system start up which is used to monitor and manager all other processes
migration: ?
ksoftirqd: kernel threads to manage the softirq
watchdog: ?
events: predefine kernel threads to manage work queues
khelper:?
kthread:?
kblockd: special work queue management threads for block device
kacpid:
khubd:
kseriod:
khungtaskd:
kswapd: the kernel thread to mange the memory swap
aio
kpsmoused:
ata:
ata_aux:
scsi_eh_0:
kstriped
ksnapd
kjournald
kauditd
/sbin/udevd -d: event manage daemon to handle device management (udev)
hd-audio0
kmpathd
kmpath_handlerd
kondemand
iscsi_eh
ib_addr
ib_mcast
ib_inform
local_sa
iw_cm_wq
ib_cm
rdma_cm
cqueue:

eicar signature in clamav

(1) Get the clamav signature and then unpack it:
sigtool --unpack-current main.cvd

(2) Inside unpack main.ndb
Eicar-Test-Signature:0:0:58354f2150254041505b345c505a58353428505e2937434329377d2445494341522d5354414e444152442d414e544956495255532d544553542d46494c452124482b482a

==>

malware-name:any file: absolute offset: hex signature:(decode hex)X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

computer virus

I am trying to understand how the anti-virus software create/manage the virus signatures.

This article gives me some idea how the signature is created. When a new virus is found, you can run a 'decoy' program to collect as much as possible variance of the virus, and then compare them to get the candidate signatures. In order to avoid 'false positive', the candidate signature may run against a large volume of the good data to obtain unique signature for the virus.

Here is another blog regarding how to create the clamav signatures. A detail explain to eicar virus test file.

A nice web site regarding the virus analysis.

Linux FAQ

(1) What is the maximum memory for 32 bit Linux?
  Normally, Linux split 4G address space into two parts: 3G for user space virtual address and 1G for kernel virtual address.Here are some solutions:
  a). HIGHMEM solution for up to 4G
  Use the kmap to map the memory in ZONE_HIGHMEM to ZONE_NORMAL
  b). HIGHMEM solution for using 64G memroy (36bit bus address)
  This is enabled via PAE(Physical Address Extension) extension of the Pentiumpro processor. Then map ZONE_HIGHMEM to ZONE_NORMAL

(2) What is the maximum memory for 64 bit linux?
Based on this, it is 16T, which is controlled by:
  arch/x86/include/asm/sparsemem.h:
  # define MAX_PHYSMEM_BITS     44

Obtained current logon user name in Windows

We had an application to retreive current logon user name in Windows. Since our application is an HTTP proxy, there are two approaches:

(1) Obtain user name using TCP connection
a). tcptable=GetExtendedTcpTable(...)
b). Get the process id of the socket
    foreach(tcptable->dwNumEntries)
            if (tcptable->table[i].dwLocalPort==port)
                processid=tcptable->table[i].dwOwningpid;
c). Open process to get the process token
    hrpocess=OpenProcess(PROCESS_QUERY_INFORMATION, FALSE, processid)
    OpenProcessToken(hprocess, TOKEN_QUERY, &hToken)
d) Then use the token to user info
    GetTokenInformation(hToken, TokenUser, PUserInfo, dwSize, &dwSize)
e) Look up the user name from the user info
    LookupAccountSidW(NULL, pUserInfo->User.sid, Name, &dwSize, lpDomain, &dwSize, &SidType)


(2) Obtain the user name using active session
a) enumerate the sessions
   WTSEnumerateSessions(WTS_CURRENT_SERVER_HANDLE, 0 ,1, &pSessionInfo, &dwCount)
b) foreach dwCount
        if (pSessionInfo[i].State==WTSActive)
              dwSessionId=pSessionInfo[i].SessionId;
c). Then query the user token use the session id
      WTSQueryUserToken(dwSessionId, &hToken) 
After that following the d) and e) in approach (1)

large file cache use mmap

We had a 3G file and fields position can be identified using hash value. We use the mmap to deal with the cache.
(1) open the file
(2) caculate the position, if the data is not in memory, then call:
mmap2(NULL, 16384, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0x210e8)
which loads the 16K into memory.
(3) for the data in the next request, call the mmap2 again (the mmap can be called multiple times).
(4) When you run out of cache (let's say 300M), find the least used fields, then run
munmap(....)
to cache the fileds out.

Perl provides a module Cache:FastMmap to share memory cross processes.

process virtual memory layout in 32 bit and 64 bit Linux

I did some experiment to compare the process virtual memory layout in 32 bit linux (2.6.9) and 64 bit linux(2.6.18).

This program is used:

const size_t arraySize=0x1000000; //16M
char global[arraySize*2];
int main(int argc, char** argv)
{
        int i=0;
        char local[arraySize];
        char* heap=(char*)malloc(arraySize*3);
        for (i=0; i        for (i=0; i        for (i=0; i        printf("local %lx, global %lx, heap %lx\r\n",local,global, heap);
}

(1) In linux 32 bit, the output is:

local bee40ef0, global 08049800, heap 4000a008

The memory map is:
00734000-00749000 r-xp 00000000 03:03 32066      /lib/ld-2.3.4.so
00749000-0074a000 r--p 00015000 03:03 32066      /lib/ld-2.3.4.so
0074a000-0074b000 rw-p 00016000 03:03 32066      /lib/ld-2.3.4.so
00753000-0075c000 r-xp 00000000 03:03 38761      /lib/libgcc_s-3.4.6-20060404.so.1
0075c000-0075d000 rw-p 00009000 03:03 38761      /lib/libgcc_s-3.4.6-20060404.so.1
00774000-00898000 r-xp 00000000 03:03 32151      /lib/tls/libc-2.3.4.so
00898000-00899000 r--p 00124000 03:03 32151      /lib/tls/libc-2.3.4.so
00899000-0089c000 rw-p 00125000 03:03 32151      /lib/tls/libc-2.3.4.so
0089c000-0089e000 rw-p 0089c000 00:00 0
008a0000-008c1000 r-xp 00000000 03:03 38754      /lib/tls/libm-2.3.4.so
008c1000-008c3000 rw-p 00020000 03:03 38754      /lib/tls/libm-2.3.4.so
00af9000-00bb9000 r-xp 00000000 03:03 195327     /usr/lib/libstdc++.so.6.0.3
00bb9000-00bbe000 rw-p 000bf000 03:03 195327     /usr/lib/libstdc++.so.6.0.3
00bbe000-00bc4000 rw-p 00bbe000 00:00 0
08048000-08049000 r-xp 00000000 00:14 34996225   /mnt/centos/ut/a.out
08049000-0804a000 rw-p 00000000 00:14 34996225   /mnt/centos/ut/a.out
0804a000-0a04a000 rw-p 0804a000 00:00 0
40008000-4300d000 rw-p 40008000 00:00 0
bee40000-c0000000 rw-p bee40000 00:00 0
ffffe000-fffff000 ---p 00000000 00:00 0

The exeutable is start from 0x08048000, and then the data and bss section for global variable (32M):
0804a000-0a04a000 rw-p 0804a000 00:00 0

And then the heap starts from 1/3 of the user space (0x4000000), which is 48M:
40008000-4300d000 rw-p 40008000 00:00 0

The stack begins at 0xc0000000 and grows to low address, which is 18M:
bee40000-c0000000 rw-p bee40000 00:00 0

(2) in Linux 64 bit:

The output is:
local 7fffa88b2e40, global 00600b20, heap 2ab801210010

The memory map is:
00400000-00401000 r-xp 00000000 fd:00 34996225                           /home/bliu/ut/a.out
00600000-00601000 rw-p 00000000 fd:00 34996225                           /home/bliu/ut/a.out
00601000-02601000 rw-p 00601000 00:00 0
314ec00000-314ec1c000 r-xp 00000000 fd:00 11796527                       /lib64/ld-2.5.so
314ee1b000-314ee1c000 r--p 0001b000 fd:00 11796527                       /lib64/ld-2.5.so
314ee1c000-314ee1d000 rw-p 0001c000 fd:00 11796527                       /lib64/ld-2.5.so
314f000000-314f14d000 r-xp 00000000 fd:00 11796545                       /lib64/libc-2.5.so
314f14d000-314f34d000 ---p 0014d000 fd:00 11796545                       /lib64/libc-2.5.so
314f34d000-314f351000 r--p 0014d000 fd:00 11796545                       /lib64/libc-2.5.so
314f351000-314f352000 rw-p 00151000 fd:00 11796545                       /lib64/libc-2.5.so
314f352000-314f357000 rw-p 314f352000 00:00 0
315ae00000-315ae0d000 r-xp 00000000 fd:00 11796784                       /lib64/libgcc_s-4.1.2-20080825.so.1
315ae0d000-315b00d000 ---p 0000d000 fd:00 11796784                       /lib64/libgcc_s-4.1.2-20080825.so.1
315b00d000-315b00e000 rw-p 0000d000 fd:00 11796784                       /lib64/libgcc_s-4.1.2-20080825.so.1
3cb2600000-3cb26e6000 r-xp 00000000 fd:00 18000239                       /usr/lib64/libstdc++.so.6.0.8
3cb26e6000-3cb28e5000 ---p 000e6000 fd:00 18000239                       /usr/lib64/libstdc++.so.6.0.8
3cb28e5000-3cb28eb000 r--p 000e5000 fd:00 18000239                       /usr/lib64/libstdc++.so.6.0.8
3cb28eb000-3cb28ee000 rw-p 000eb000 fd:00 18000239                       /usr/lib64/libstdc++.so.6.0.8
3cb28ee000-3cb2900000 rw-p 3cb28ee000 00:00 0
3ecfe00000-3ecfe82000 r-xp 00000000 fd:00 35454980                       /lib64/libm-2.5.so
3ecfe82000-3ed0081000 ---p 00082000 fd:00 35454980                       /lib64/libm-2.5.so
3ed0081000-3ed0082000 r--p 00081000 fd:00 35454980                       /lib64/libm-2.5.so
3ed0082000-3ed0083000 rw-p 00082000 fd:00 35454980                       /lib64/libm-2.5.so
2ab8011f6000-2ab8011f7000 rw-p 2ab8011f6000 00:00 0
2ab80120d000-2ab804213000 rw-p 2ab80120d000 00:00 0
7fffa88b2000-7fffa98b4000 rw-p 7ffffeffd000 00:00 0                      [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]


Here the executable is start from 0x00400000, then 32M is allocated for the global bss section:
00601000-02601000 rw-p 00601000 00:00 0

After load the library at low address, the heap is started at 0x2ab80120d000, which is 48M:
2ab80120d000-2ab804213000

The stack is started at 7fffa98b4000, which is 16M:
7fffa88b2000-7fffa98b4000


Then VDSO(Virtual dynamic shared object):
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vdso]
the vdso mapped to linux-gate.so.1, which is a virtual shared object exposed by kernel. It is used to make virtual system call using sysenter/sysexit, and this is faster than regular interruption.