On-line inspection

Clone the disk and copy data

• Efficiency VS Accuracy
• Copying the data to a remote host for later investigation
• • Using cp -p to prserve the owner and group id, permissions modes, modification and access time

    	cp -rp directory target_directory
    

  • Copying the data to a remote host by ttcp or nc

    	Received side
            nc -p 6666 -l > output          
    
            Send side 
            cat data |nc -w 3 receved_host 6666
    
            Received side                             
            /usr/local2/bin/ttcp -r -B | tar xvpf -    
    
            Send side
            tar cf - directory  |  /usr/local/bin2/ttcp  -t receved_host
    

  • Clone to other disk by dd

            dd if=/dev/rdsk/c0t0d0s7 of=/dev/rmt/0
            dd if=/dev/rdsk/c0t0d0s7 of=/dev/rdsk/same_size_disk
    

Memory Dump

        dd if= /dev/kmem of=output
        dd if= /dev/mem  of=output

Process Investigation

• To view the running process enviroment

    /bin/ps auxeww 
          to view the running process enviroment
    
  E.G.
  
  The ./setiathome process 
  
  
  USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
  
  shlam    12485 99.9 22.7 15492 14112 ?       RN   07:42 126:28 ./setiathome TERM=vt100 
  DISPLAY=ntec5:10.0 HOME=/home/shlam SHELL=/bin/tcsh USER=shlam LOGNAME=shlam 
  PATH=/usr/kerberos/bin:/bin:/usr/bin:/usr/local/bin:/usr/bin/X11:/usr/X11R6/bin 
  HOSTTYPE=i386-linux VENDOR=intel OSTYPE=linux MACHTYPE=i386 SHLVL=1 PWD=/home/shlam/set 
  GROUP=peter HOST=ntec5 
  REMOTEHOST=ntec4.ie.cuhk.edu.hk HOSTNAME=ntec5 INPUTRC=/etc/inputrc LS_COLORS=no=00:fi=00:
  di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:
  ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:
  *.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:
  *.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;
  35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: KDEDIR=/usr LANG=en_US LESSOPEN
  =|/usr/bin/lesspipe.sh %s QTDIR=/usr/lib/qt-2.1.0 HTTP_PROXY=proxy.ie.cuhk.edu.hk:8080
  

  ---

  Assignment
  1. restart snmpd (/etc/rc.d/init.d/snmpd restart)

  2. dump the snmpd process enviroment and its current enviroment to Ass_G.html /bin/ps auxeww |grep snmpd |grep -v grep > a_text_file
  3. exam the a_text_file and try to find out the following information. Compare this information with the one you obtained from /usr/bin/env REMOTEHOST= where did you login from PWD= where did you start the process PATH= what is your search path SUDO_COMMAND= what is your sudo command

  4. Put all your finding on your web pages

  ---

• To view process opening file (lsof -p pid)

  
  E.G. list the opening file of syslod (its pid is 467)
  
  csh> lsof -p 467
  
  COMMAND PID USER   FD   TYPE     DEVICE    SIZE   NODE NAME
  syslogd 467 root  cwd    DIR        3,3    4096      2 /
  syslogd 467 root  rtd    DIR        3,3    4096      2 /
  syslogd 467 root  txt    REG        3,3   26352  96657 /sbin/syslogd
  syslogd 467 root  mem    REG        3,3  340663 208505 /lib/ld-2.1.3.so
  syslogd 467 root  mem    REG        3,3 4101324 208512 /lib/libc-2.1.3.so
  syslogd 467 root  mem    REG        3,3  246652 208543 /lib/libnss_files-2.1.3.so
  syslogd 467 root    0u  unix 0xc37c2ec0            502 /dev/log
  syslogd 467 root    1w   REG        3,3  210439 229189 /var/log/messages
  syslogd 467 root    2w   REG        3,3     114 229190 /var/log/secure
  syslogd 467 root    3w   REG        3,3    1144 229191 /var/log/maillog
  syslogd 467 root    4w   REG        3,3       0 229192 /var/log/spooler
  syslogd 467 root    5w   REG        3,3   27428 229193 /var/log/boot.log
  syslogd 467 root    6w   REG        3,3       0 226719 /var/log/news/news.crit
  syslogd 467 root    7w   REG        3,3       0 226720 /var/log/news/news.err
  syslogd 467 root    8w   REG        3,3       0 226718 /var/log/news/news.notice
  
  
         FD         is the File Descriptor number of the file or:
  
                         cwd  current working directory;
                         Lnn  library references;
                         ltx  shared library text (code and data);
                         Mxx  hex memory-mapped type number xx.
                         m86  DOS Merge mapped file;
                         mem  memory-mapped file;
                         pd   parent directory;
                         rtd  root directory;
                         txt  program text (code and data);
                         v86  VP/ix mapped file;
  
                    FD  is  followed  by  one of these characters, describing the mode under
                    which the file is open:
  
                         r for read access;
                         w for write access;
                         u for read and write access;
                         space if unknown and no lock character;
                         `-' if unknown and lock character.
  
  
  

  ---

  Assignment
  1. Exam the lsof information of your httpd to
     cat /var/run/httpd.pid to get the process ID no. Then use lsof to dump the open files information of the httpd /usr/sbin/lsof -p The_httpd_proccess_ID_no. >> a_text_file
  2. Exam the a_tex_file and identify which log files does your httpd open
  3. Post all your findings on your web pages

  ---

• To trace a process library call (ltrace -p pid)

  csh> ltrace -p 12485
  
  getrusage(0, 0xbffff820, 0x4013dd60, 0x4013dd60, 0x64646465) = 0
  free(0x40d46008)                                  = 
  sprintf("outfile.sah", "%s%s", "", "outfile.sah") = 11
  fopen("outfile.sah", "r")                         = 0x080731a8
  fseek(0x080731a8, 0, 2, 0x0805de68, 0x40d46040)   = 0
  ftell(0x080731a8, 0x0805d001, 0xbffff868, 0x0805033e, 0x080731a8) = 5208
  fclose(0x080731a8)                                = 0
  malloc(524355)                                    = 0x40d46008
  
  

• To trace a process system calls and signals (strace -p pid)

  
  csh> strace -p 12485
  
  old_mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40d46000
  getrusage(RUSAGE_SELF, {ru_utime={7512, 860000}, ru_stime={12, 290000}, ...}) = 0
  munmap(0x40d46000, 1052672)             = 0
  open("outfile.sah", O_RDONLY)           = 4
  fstat(4, {st_mode=S_IFREG|0644, st_size=5208, ...}) = 0
  old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40015000
  fstat(4, {st_mode=S_IFREG|0644, st_size=5208, ...}) = 0
  _llseek(4, 4096, [4096], SEEK_SET)      = 0
  read(4, "4 chirprate=7.926451e+00 maxpow="..., 1112) = 1112
  close(4)                                = 0
  munmap(0x40015000, 4096)                = 0
  
  

  ---

  Assignment
  1. Run "ping -i 5 your_partner_host"

  2. dump the first 50 lines of ltrace and strace of ping prcoess
     ntec23:/root> ps -ef | grep ping root 12868 12729 0 14:09 pts/1 00:00:00 ping ntec95 ntec23:/root> ltrace -tp 12868 14:11:50 --- SIGALRM (Alarm clock) --- 14:11:50 gettimeofday(0xbffff82c, 0, 12, 0x0805e700, 0x400817e4) = 0 14:11:50 gettimeofday(0xbffff800, 0, 0, 0, 192) = 0 14:11:50 memcpy(0x0804d2e8, "\246\005\001:\376)", 8) = 0x0804d2e8 14:11:50 sendmsg(3, 0x0804cff0, 0, 0, 0) = 64 14:11:50 setitimer(0, 0xbffff834, 0, 12, 0x0805e700) = 0 14:11:50 breakpointed at 0x400ddc01 (?) 14:11:50 __errno_location() = 0x4011bd60 14:11:50 recvfrom(3, 0x0805e700, 192, 0, 0xbffff9d4) = 84 14:11:50 gettimeofday(0xbffff928, 0, 0xbffff9d4, 0x0805e700, 192) = 0 14:11:50 gethostbyaddr(0xbffff8e8, 4, 2, 0x0805d300, 0xbffff930) = 0x4011de48 14:11:50 inet_ntoa(0x5f80a8c0) = "192.168.128.95" 14:11:50 sprintf("ntec95 (192.168.128.95)", "%s (%s)", "ntec95", "192.168.128.95") = 23 14:11:50 printf("%d bytes from %s: icmp_seq=%u", 64, "ntec95 (192.168.128.95)", 129) = 51 14:11:50 printf(" ttl=%d", 254) = 8 14:11:50 printf(" time=%ld.%ld ms", 4, 4) = 12 14:11:50 _IO_putc('\n', 0x40119980) = 10 14:11:50 fflush(0x40119980) = 0 14:11:50 recvfrom(3, 0x0805e700, 192, 0, 0xbffff9d4 <unfinished ...> ... ntec23:/root> strace -tp 12868 14:14:38 recvfrom(3, 0x805e700, 192, 0, 0xbffff9d4, 0xbffff968) = ? ERESTARTSYS (To be restarted) 14:14:38 --- SIGALRM (Alarm clock) --- 14:14:38 gettimeofday({973145678, 996437}, NULL) = 0 14:14:38 gettimeofday({973145678, 996606}, NULL) = 0 14:14:38 sendmsg(3, {msg_name(16)={sin_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.128.95")}}, msg_iov(1)=[{"\10\0BTD2*\1N\6\1:\3764\17\0\10\t\n\v\f\r\16\17\20\21\22"..., 64}], msg_controllen=0, msg_flags=0}, 0) = 64 14:14:38 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={1, 0}}, NULL) = 0 14:14:38 sigreturn() = ? (mask now []) 14:14:38 recvfrom(3, "E\0\0Tg\26\0\0\376\1\23\313\300\250\200_\300\250@\27\0"..., 192, 0, {sin_family=AF_INET, sin_port=htons(4288), sin_addr=inet_addr("192.168.128.95")}}, [16]) = 84 14:14:38 gettimeofday({973145678, 998321}, NULL) = 0 14:14:38 socket(PF_UNIX, SOCK_STREAM, 0) = 4 14:14:38 connect(4, {sin_family=AF_UNIX, path=" /var/run/.nscd_socket"}, 110) = -1 ECONNREFUSED (Connection refused) 1

  3. copy and paste ltrace and strace outputs to a file and post it on your web pages

  ---

• To dump a process memory image
  • Stop the process before dump
    

    	 kill -STOP pid
    	 ps ax|grep T to check its status
    

  • Dump the process image
    

    	 gcore -o core.file pid (for solaris)
    
    	 /usr/local2/bin/pcat pid > dump.file  (by using CTC)
    

  • Resume the process if necessary
    

    	 kill -CONT pid
    

  • peek the strings in the dump files (strings dump.file)

           csh> strings /tmp/syslogd.dump
           /lib/ld-linux.so.2
           __gmon_start__
           libc.so.6
           longjmp
           strcpy
           writev
           printf
           stdout
           getdtablesize
           recv
           strerror
           fdopen
           snprintf
           __ctype_b
           __rawmemchr
           __strtol_internal
           getpid
           fscanf
           fgets
           ...  
           	

• Recovering files from Kernel through /proc directory

  Each running process has a corresponding directory in /proc by the name of its pid. You can review various process information in this directory. You can also recover the process binary file even it has been deleted from the hard disk. See the folloing example.

  [root@ntec 683]# cd /tmp [root@ntec /tmp]# cp /usr/sbin/inetd /tmp/copy_inetd [root@ntec /tmp]# /tmp/copy_inetd Start another inetd process [root@ntec /tmp]# ps -ef|grep /tmp/copy_inetd root 8081 1 0 11:47 ? 00:00:00 /tmp/copy_inetd [root@ntec /tmp]# cd /proc/8081 [root@ntec 8081]# ls -l total 0 -r--r--r-- 1 root root 0 Jun 11 11:47 cmdline lrwx------ 1 root root 0 Jun 11 11:47 cwd -> / -r-------- 1 root root 0 Jun 11 11:47 environ lrwx------ 1 root root 0 Jun 11 11:47 exe -> /tmp/copy_inetd dr-x------ 2 root root 0 Jun 11 11:47 fd pr--r--r-- 1 root root 0 Jun 11 11:47 maps -rw------- 1 root root 0 Jun 11 11:47 mem lrwx------ 1 root root 0 Jun 11 11:47 root -> / -r--r--r-- 1 root root 0 Jun 11 11:47 stat -r--r--r-- 1 root root 0 Jun 11 11:47 statm -r--r--r-- 1 root root 0 Jun 11 11:47 status The soft link of exe show the location of the process binary file [root@ntec 8081]# rm /tmp/copy_inetd rm: remove `/tmp/copy_inetd'? y [root@ntec 8081]# ls -l total 0 -r--r--r-- 1 root root 0 Jun 11 11:47 cmdline lrwx------ 1 root root 0 Jun 11 11:47 cwd -> / -r-------- 1 root root 0 Jun 11 11:47 environ lrwx------ 1 root root 0 Jun 11 11:47 exe -> /tmp/copy_inetd (deleted) dr-x------ 2 root root 0 Jun 11 11:47 fd pr--r--r-- 1 root root 0 Jun 11 11:47 maps -rw------- 1 root root 0 Jun 11 11:47 mem lrwx------ 1 root root 0 Jun 11 11:47 root -> / -r--r--r-- 1 root root 0 Jun 11 11:47 stat -r--r--r-- 1 root root 0 Jun 11 11:47 statm -r--r--r-- 1 root root 0 Jun 11 11:47 status [root@ntec 8081]# cp exe /tmp/recovery_of_copy_inetd Recover the binary file even it have been deleted. [root@ntec 8081]# ls -l /tmp/recovery_of_copy_inetd /usr/sbin/inetd -rwxr-xr-x 1 root root 21200 Jun 11 11:48 /tmp/recovery_of_copy_inetd -rwxr-xr-x 1 root root 21200 Jan 30 06:59 /usr/sbin/inetd [root@ntec 8081]# cmp /usr/sbin/inetd /tmp/recovery_of_copy_inetd

Collect Network Information

• The host network connection (netstat)

  csh> netstat
  
  TCP
     Local Address        Remote Address    Swind Send-Q Rwind Recv-Q  State
  -------------------- -------------------- ----- ------ ----- ------ -------
  iesun7.telnet        banpc17.1040         32120      0 10136      0 ESTABLISHED
  iesun7.40590         ieug0.50819          64240      0  8760      0 ESTABLISHED
  iesun7.560           ieug0.nfsd           64240      0  8760      0 ESTABLISHED
  iesun7.22            iest26.1022           8760      0  8760     20 ESTABLISHED
  iesun7.40709         eng.32771             8760      0  8760      0 TIME_WAIT
  iesun7.559           iest0.nfsd            8760      0  8760      0 ESTABLISHED
  iesun7.40710         eng.32771             8760      0  8760      0 TIME_WAIT
  iesun7.558           eng.nfsd              8760      0  8760      0 ESTABLISHED
  iesun7.557           ieugb.nfsd            8760      0  8760      0 ESTABLISHED
  Active UNIX domain sockets
  

• To list the open port (lsof -i)

  ntec81:/var/log/tcpdump>  /usr/sbin/lsof -i
  COMMAND    PID USER   FD   TYPE DEVICE SIZE NODE NAME
  portmap    345 root    3u  IPv4    359       UDP *:sunrpc 
  portmap    345 root    4u  IPv4    360       TCP *:sunrpc (LISTEN)
  rpc.statd  370 root    0u  IPv4    406       UDP *:971 
  rpc.statd  370 root    1u  IPv4    409       TCP *:973 (LISTEN)
  identd     489 root    4u  IPv4    515       TCP *:auth (LISTEN)
  identd     493 root    4u  IPv4    515       TCP *:auth (LISTEN)
  identd     494 root    4u  IPv4    515       TCP *:auth (LISTEN)
  identd     495 root    4u  IPv4    515       TCP *:auth (LISTEN)
  identd     496 root    4u  IPv4    515       TCP *:auth (LISTEN)
  inetd      539 root    4u  IPv4    562       TCP *:ftp (LISTEN)
  inetd      539 root    5u  IPv4    563       TCP *:telnet (LISTEN)
  inetd      539 root    6u  IPv4    564       TCP *:shell (LISTEN)
  inetd      539 root    9u  IPv4    565       TCP *:login (LISTEN)
  inetd      539 root   10u  IPv4    566       UDP *:talk 
  inetd      539 root   11u  IPv4    567       UDP *:ntalk 
  inetd      539 root   12u  IPv4    568       TCP *:finger (LISTEN)
  inetd      539 root   13u  IPv4    569       TCP *:linuxconf (LISTEN)
  lpd        553 root    6u  IPv4    586       TCP *:printer (LISTEN)
  sendmail   601 root    4u  IPv4    631       TCP *:smtp (LISTEN)
  sshd       740 root    3u  IPv4    725       TCP *:ssh (LISTEN)
  xntpd      752 root    4u  IPv4    744       UDP *:ntp 
  xntpd      752 root    5u  IPv4    745       UDP localhost.localdomain:ntp 
  xntpd      752 root    6u  IPv4    746       UDP ntec81:ntp 
  ypbind     768 root    4u  IPv4    761       UDP *:945 
  ypbind     768 root    5u  IPv4    764       TCP *:947 (LISTEN)
  ypbind     774 root    4u  IPv4    761       UDP *:945 
  ypbind     774 root    5u  IPv4    764       TCP *:947 (LISTEN)
  ypbind     774 root    8u  IPv4   1981       UDP *:952 
  ypbind     774 root   14u  IPv4   1983       UDP *:953 
  named     1260 root    4u  IPv4   1501       UDP *:1026 
  named     1260 root   20u  IPv4   1497       UDP localhost.localdomain:domain 
  named     1260 root   21u  IPv4   1498       TCP localhost.localdomain:domain (LISTEN)
  named     1260 root   22u  IPv4   1499       UDP ntec81:domain 
  named     1260 root   23u  IPv4   1500       TCP ntec81:domain (LISTEN)
  httpd     1325 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1328 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1329 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1330 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1331 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1332 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1333 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1334 root   16u  IPv4   1596       TCP *:www (LISTEN)
  httpd     1335 root   16u  IPv4   1596       TCP *:www (LISTEN)
  sshd      3596 root    5u  IPv4  10725       TCP ntec81:ssh->fortress:1023 (ESTABLISHED)
  sshd      3596 root    9u  IPv4  10732       TCP *:6010 (LISTEN)
  

• To trace a host (traceroute)

  csh> traceroute www.ust.hk
  
  traceroute to www.ust.hk (143.89.14.34), 30 hops max, 38 byte packets
   1  ntec-fw (192.168.64.254)  0.268 ms  0.170 ms  0.148 ms
   2  router99.ie.cuhk.edu.hk (137.189.99.254)  0.833 ms  0.860 ms  0.767 ms
   3  137.189.200.250 (137.189.200.250)  1.402 ms  1.270 ms  1.383 ms
   4  globalone-FE.hkix.net (202.40.161.32)  3.292 ms  3.097 ms  3.052 ms
   5  204.59.89.14 (204.59.89.14)  3.191 ms  2.911 ms  2.840 ms
   6  192.245.196.214 (192.245.196.214)  5.038 ms  5.080 ms  4.657 ms
   7  internet-gw1.ust.hk (202.40.138.117)  5.052 ms  5.463 ms  4.940 ms
   8  www.ust.hk (143.89.14.34)  4.753 ms *  4.872 ms
  

• To trace the MAC (arp)

  csh> /sbin/arp ntec14
  Address                 HWtype  HWaddress           Flags Mask            Iface
  ntec14                  ether   00:D0:09:28:9F:79   C                     eth0
  
  csh> /sbin/arp -a
  ntec14 (192.168.64.14) at 00:D0:09:28:9F:79 [ether] on eth0
  pntec11 (172.18.1.11) at 00:D0:09:44:C2:7C [ether] on eth0
  ntec92 (192.168.128.92) at 00:D0:09:28:31:69 [ether] on eth0
  ntec93 (192.168.128.93) at 00:D0:09:2A:49:08 [ether] on eth0
  ntec15 (192.168.64.15) at 00:D0:09:4D:9F:D6 [ether] on eth0
  ntec12 (192.168.64.12) at 00:D0:09:28:3B:1C [ether] on eth0
  ntec94 (192.168.128.94) at 00:D0:09:2D:6F:03 [ether] on eth0
  victim (192.168.128.50) at 00:D0:09:27:66:18 [ether] on eth0
  ntec13 (192.168.64.13) at 00:D0:09:2D:D8:2A [ether] on eth0
  ntec95 (192.168.128.95) at 00:D0:09:4E:46:0C [ether] on eth0
  ntec88 (192.168.128.88) at 00:D0:09:4E:46:13 [ether] on eth0
  ...
  
  

References

• Read the man pages of the above commands
• Computer Forensics Analysis Class Handouts at www.fish.com
• Coroner's Toolkit (TCT)
• The Study of an Attack Know Your Enemy: A Forensic Analysis
• Forensic Computer Analysis: An Introduction (from Dr. Dobb's)
• "Root Kits" and hiding file/directories/processes after a beak-in
• Widespread Exploitation of rpc.statd and wu-ftpd Vulnerabilities
• CERT Intruder Detection Checklist