set sqlite3 for django in centos 5.3

I upgraded the python in my centos from 5.4 to 5.5. However, when I try to set up the django to use the sqlite3 database, and it stop working: I cannot get sqlite3 for python 5.5 as the rpm already installed for 5.4. And the one for 5.4 give me some unresolved symbol errors.

I workaround this by download the source from pysqlite. and then run:

python setup.py build_static install 

Now: 

manage.py syncdb

works.

recover delete file from Linux

Refer to this page

$ sudo yum install ext3grep

Create a file called: strangefile.txt

$ rm strangefile.txt

$ sudo ext3grep  /dev/mapper/VolGroup00-LogVol00  --restore-file home/username/strangefile.txt

combine bing map

I want to combine bing birdview map to create a bigger map. In Linux, a few commands can help:
djpeg: this will convert the jpg to ppm format.
pnmcat: this will combine the images
cjpeg: this will convert the ppm back to jpg.

Also, the bing images in such a format:
http://ecn.t2.tiles.virtualearth.net/tiles/cmd/ObliqueHybrid?a=03022313120-1715-20-120&g=671
Each row has 16 small tiles.

The code looks like this:
#!/usr/bin/perl

#965    1715      1989
#    2590 703
my $url="http://ecn.t2.tiles.virtualearth.net/tiles/cmd/ObliqueHybrid?a=03022313120-";
my $colnum=16;
my $startcut=$colnum;
my $curtilenum=1715;
my $rownum=(191-$startcut)/$colnum-1;
#my @tilenum=(965, 1715, 1989, 2590, 703);
my @tilenum=(1715);

sub callCmd($)
{
    my ($cmd) = @_;
    open(CMD, "$cmd 2>&1 |") or die "Couldn't open CMD file to execute command";
    my $commandOutput= <CMD>;
    close(CMD);
}

sub combineTile($)
{
    my ($curtilenum) = @_;
    my $combinexfile="";
    my $combineyfile="";
    for ($row=0;$row<$rownum;$row++)
    {
       for ($col=0; $col<$colnum; $col++)
       {
         my $x=$col;
         my $y=$row;
         my $num=$startcut+$x+$y*$colnum;
         my $cmd="curl \"$url$curtilenum-20-$num&g=671\" -o tmp/t.$x.$y.jpg";
         print $cmd . "\n";
         system($cmd);
         #Convert JPEGs into PPM bitmaps
         callCmd("djpeg tmp/t.$x.$y.jpg > tmp/$x.$y.ppm");
         $combinexfile = $combinexfile . "tmp/$x.$y.ppm ";
       }
       #combine the cols
       callCmd("pnmcat -leftright $combinexfile > tmp/row.$row.ppm");
       $combinexfile="";
       $combineyfile = $combineyfile . "tmp/row.$row.ppm ";
     }
     callCmd("pnmcat -topbottom $combineyfile > tmp/final.ppm");
     callCmd("cjpeg -quality 100 tmp/final.ppm > final$curtilenum.jpg");
     callCmd("rm tmp/*.ppm tmp/*.jpg");
}



foreach $tile (@tilenum){
    combineTile($tile);
}

Error prone C

These tips taken from "writting solid code":misspell errors

•  Misspell errors

 /*This file is used to demonstrate some error prone function in C.
 *Taken from the "Wirting Solid Code"
 */
#include <stdio.h>
int main()
{
     char ch;
     int word;
     short bHigh=4, bLow=1;
     char* pb=NULL;
     /*Two possible errors here
      * a). char can be unsigned even if x86 always uses signed char
      *     gcc can also change the behavior using -funsigned-char flag
      * b). the != has higher precedence than =, so this is equal to
      *     ch=(getchar() != EOF)
     if (ch=getchar() != EOF)
     {
          printf("Not real char: %c\n", ch);
     }



     /* Similar error:
      * It is bHigh<<(8+bLow);
      */
     word=bHigh<<8 + bLow;

     /* compiler warnning:
      * comparison is always false due to limited range of data type
      */
     ch=0xFF;
     if (ch==0xFF)
            printf("Happy\n");

     /* condition error
      * a). 063: octal number
      * b). & instead of && : compiler warnning: comparison is always true due to limited range of data type
      * c). /* comment, should use / *
      */
     if (word==063
         || (pb!=NULL & *pb !=0xFF )
//         || word=bHigh/*bLow
     )
     {
             printf("Condition matched\n");
     }

}
~                         

•  Make assumption that long is 4 always bytes. how about 64 bit machine?
• realloc() too flexible interface
• getchar() handle the errors and results in the same return value
• step through code use debugger (data flow)
• use the size instead of pointer to control your loop (overflow....)
• the risk to modify static variables in a function

        a). You have to make sure nobody  will call directly/indirectly that function before
        b). multiple threads
        c). volatile

• the risk to refer the memory just released:

         a). internal memory manager may use that for free chain private information
         b). other thread may used it.

• Compiler will help you to optimize for such code (high risk no return):

         size >> 2 instead of size/4

• --size : error prone

         while (--size >= 0) //how about unsigned int size? or signed size is MIN_INT

• ANSI type range:

          char                         0-----------------------------127
          signed char             -127(not -128)-----------127
          unsigned char         0-----------------------------255
          short                        -32767(not -32768)-----32767
          int                            -32767(not -32768)-----32767
          long                          -2147483647------------- 2147483647

         

Compared Signal handling between LinuxThread and NPTL

Versions:
$ getconf GNU_LIBPTHREAD_VERSION
linuxthreads-0.10
$getconf GNU_LIBPTHREAD_VERSION
NPTL 2.5

The main program will starts up 5 threads, and we use the USR1 Signal:
Signal                   Linux Thread                               NPTL
 ------------------------------------------------------------------------------------------------------------
non-block             6 threads captured, exit            only main thread
                                                                             get 1, not exit
 ------------------------------------------------------------------------------------------------------------
block main           no                                                no
 ------------------------------------------------------------------------------------------------------------
block main+         one kill, everyone gets/exit        round-robin until
unblock threads                                                       last one get then exit
------------------------------------------------------------------------------------------------------------
block child+         main, multiple kill, no exit           main, multiple kill
unblock main                                                            , no exit
------------------------------------------------------------------------------------------------------------
pthread_kill          as expected                                 as expected
to each other

Heruristic virus scan

I was interested in the heruristic virus scan. I just come cross an article talk about this. Here is the summary:

• Weight based (old way) vs. Rule based
• vb5 example:

    Options.VirusProtection = 0 

 Obfuscation:
     Options.VirusProtection = 1 AND 0

• Encrypted virus
• initialize the pointer with a valid memory address
• initialize a counter
• memory read operation depends on the pointer
• logical operation on the memory read result
• memory write operation with the result from the logical operation
• manipulation of the counter
• branching depends on the counter

• Emulate and decode the data 

            word7 vbscript example:

1. Line 1: Private Sub Document_Open(): Application.EnableCancelKey = wdCancelDisabled

   The definition of the private document handler Document_Open() (often
   inaccurately referred to as a macro) is not typical for common
   applications, so it should be flagged with a low priority. The next operation disables 

   the ‘ESC’ key and has the same security
   risk level as the definition of the private document handler and,
   therefore, should be flagged accordingly.

   Line 2: For d = 6 To ThisDocument.VBProject.VBComponents.Item(1).CodeModule.CountOfLines: C$ = ""

    This line simply initializes a ‘For’ loop, depending on the number of
   lines. Such strings should be flagged by heuristic engines, as a
   request to count the lines of the existing macro code is suspicious.
   Additionally a heuristic engine should remember that ‘d’ is an integer
   variable, the maximum value of which depends on the number of lines of
   code.

   Line 3:  I = (ThisDocument.VBProject.VBComponents.Item(1).CodeModule.Lines(d, 1))  

   A line of code, depending on the counter, will be read from the macro
   code. The range from the counter is chosen that way, so that every line
   of the malicious code can be accessed. Again, this can be seen as a
   memory-read operation as described above and should be flagged.
   Furthermore, the variable ‘I’ should be stored as a string variable
   containing line information.

   Line 4: f = (Mid(I, 2, 1)): For X = 3 To Len(I): B$ = Asc(Mid(I, X, 1)) - f: C$ = C$ & Chr(B$): Next X: A = C$ 

    A set of operations will be done with the read content from the
   previous line. Actually, for the heuristic, the type of encryption that
   is occurring here is not really important; the existence of such a
   routine is suspicious enough and should be flagged. For emulation
   issues, the analysis of encryption functionality has to go deeper.

   Line 5: ThisDocument.VBProject.VBComponents.Item(1).CodeModule.ReplaceLine d, A: Next d: End Sub 

   This line replaces existing code (the parameter ‘d’ defines the line
   number and ‘A’ defines the actual content) and is another critical
   operation (equivalent to the memory-write operation mentioned above),
   which has to be flagged with a high security risk level. This line also
   contains the end of the outer ‘for’ loop, which is responsible for
   accessing all lines within a certain range of the document. 

   Line 6: '6Vxo|gzk&Y{h&Jui{sktzeIruyk./@&Uvzouty4Yg|kTuxsgrVxusvz&C&6 

   This line (as well as all of the following 13 lines) contains this kind of comment with encrypted code:

   * the string is quite long (i.e., consists of more than forty characters) and contains no spaces; 

   * it is not typical to start a comment with a number; and, 

   * the string contains suspicious mixture of numbers, special characters and ordinary alphabet characters. 

• Components of a Heuristic Engine 
• variable/memory emulator;  
• parser; 
• flow analyzer; 
• analyzer; 
• disassembler/emulator; and, 
• weight-based system and/or rule based system.     
• A rule-based system simply compares found functionality with a set of rules. If a predefined rule is found within the code, the rule-based system returns with a positive result.

Linux process details

 
Here is the execution ps -ax in my 2.6 64 bit Linux. Here we try to explain what these processes are.

PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 init [5]        
    2 ?        S<     0:00 [migration/0]
    3 ?        SN     0:00 [ksoftirqd/0]
    4 ?        S<     0:00 [watchdog/0]
    5 ?        S<     0:00 [migration/1]
    6 ?        SN     0:00 [ksoftirqd/1]
    7 ?        S<     0:00 [watchdog/1]
    8 ?        S<     0:00 [events/0]
    9 ?        S<     0:00 [events/1]
   10 ?        S<     0:00 [khelper]
   27 ?        S<     0:00 [kthread]
   32 ?        S<     0:00 [kblockd/0]
   33 ?        S<     0:00 [kblockd/1]
   34 ?        S<     0:00 [kacpid]
  132 ?        S<     0:00 [cqueue/0]
  133 ?        S<     0:00 [cqueue/1]
  136 ?        S<     0:00 [khubd]
  138 ?        S<     0:00 [kseriod]
  210 ?        S      0:00 [khungtaskd]
  213 ?        S<     0:10 [kswapd0]
  214 ?        S<     0:00 [aio/0]
  215 ?        S<     0:00 [aio/1]
  364 ?        S<     0:00 [kpsmoused]
  396 ?        S<     0:00 [ata/0]
  397 ?        S<     1:22 [ata/1]
  398 ?        S<     0:00 [ata_aux]
  405 ?        S<     0:00 [scsi_eh_0]
  406 ?        S<     3:52 [scsi_eh_1]
  413 ?        S<     0:00 [kstriped]
  426 ?        S<     0:00 [ksnapd]
  441 ?        S<     0:28 [kjournald]
  474 ?        S<     0:00 [kauditd]
  516 ?        S<s    0:00 /sbin/udevd -d
 1577 ?        S<     0:00 [hd-audio0]
 1786 ?        S<     0:00 [kmpathd/0]
 1787 ?        S<     0:00 [kmpathd/1]
 1788 ?        S<     0:00 [kmpath_handlerd]
 1813 ?        S<     0:00 [kjournald]
 2007 ?        S<     0:00 [kondemand/0]
 2008 ?        S<     0:00 [kondemand/1]
 2025 ?        S<     0:00 [iscsi_eh]
 2069 ?        S<     0:00 [ib_addr]
 2079 ?        S<     0:00 [ib_mcast]
 2080 ?        S<     0:00 [ib_inform]
 2081 ?        S<     0:00 [local_sa]
 2085 ?        S<     0:00 [iw_cm_wq]
 2089 ?        S<     0:00 [ib_cm/0]
 2090 ?        S<     0:00 [ib_cm/1]
 2094 ?        S<     0:00 [rdma_cm]
 2112 ?        Ssl    0:00 brcm_iscsiuio
 2118 ?        Ss     0:00 iscsid
 2119 ?        S<Ls   0:00 iscsid
 2220 ?        Ss     0:00 mcstransd
 2502 ?        Ss     0:02 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0
 2553 ?        S<sl   0:00 auditd
 2555 ?        S<sl   0:00 /sbin/audispd
 2567 ?        S      0:02 stardict
 2571 ?        Ss     0:00 /usr/bin/esd -terminate -nobeeps -as 2 -spawnfd 24
 2573 ?        Ss     0:00 /usr/sbin/restorecond
 2584 ?        Ss     0:01 syslogd -m 0
 2587 ?        Ss     0:00 klogd -x
 2623 ?        Ss     0:00 portmap
 2640 ?        S      0:13 /usr/bin/python2.4 /usr/share/meld/meld
 2648 ?        Ss     0:00 rpc.statd
 2674 pts/9    S+     0:01 ssh rh8
 2675 pts/10   S+     0:01 ssh rh8
 2682 ?        S<     0:00 [rpciod/0]
 2683 ?        S<     0:00 [rpciod/1]
 2690 ?        Ss     0:00 rpc.idmapd
 2709 ?        Ssl    0:18 dbus-daemon --system
 2721 ?        Ss     0:00 /usr/sbin/hcid
 2727 ?        Ss     0:00 /usr/sbin/sdpd
 2750 ?        S<     0:00 [krfcommd]
 2792 ?        Ssl    0:01 pcscd
 2808 ?        Ss     0:00 /usr/bin/hidd --server
 2827 ?        Ssl    0:00 automount
 2860 ?        Ss     0:00 /usr/sbin/acpid
 2871 ?        Ss     0:00 ./hpiod
 2876 ?        S      0:00 python ./hpssd.py
 2891 ?        Ss     0:00 /usr/sbin/sshd
 2902 ?        Ss     0:00 cupsd
 2916 ?        SLs    0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
 2936 ?        Ss     0:00 sendmail: accepting connections
 2945 ?        Ss     0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
 2957 ?        Ss     0:00 gpm -m /dev/input/mice -t exps2
 2968 ?        Ss     0:00 crond
 3003 ?        Ss     0:00 xfs -droppriv -daemon

init: created in the system start up which is used to monitor and manager all other processes
migration: ?
ksoftirqd: kernel threads to manage the softirq
watchdog: ?
events: predefine kernel threads to manage work queues
khelper:?
kthread:?
kblockd: special work queue management threads for block device
kacpid:
khubd:
kseriod:
khungtaskd:
kswapd: the kernel thread to mange the memory swap
aio
kpsmoused:
ata:
ata_aux:
scsi_eh_0:
kstriped
ksnapd
kjournald
kauditd
/sbin/udevd -d: event manage daemon to handle device management (udev)
hd-audio0
kmpathd
kmpath_handlerd
kondemand
iscsi_eh
ib_addr
ib_mcast
ib_inform
local_sa
iw_cm_wq
ib_cm
rdma_cm
cqueue: