Linux Forensics

Linux boxes leave fewer “did it run” artefacts than Windows (no Prefetch, no Amcache, no ShimCache) and far more text: logs, histories, unit files, crontabs. The investigation therefore leans on logs and timestamps — journalctl, auditd, wtmp, the four stat times on every inode — and on knowing the dozen places persistence hides. This card is the live-response order, the /var/log map, the queries, the persistence hunt, and then the dead-disk workflow on SIFT Workstation with Sleuth Kit and plaso that produces the super-timeline.

[!warning] Every command you run on a live box changes it Your shell writes ~/.bash_history, atimes update on files you read (unless the filesystem is relatime/noatime), and your tools appear in ps and the journal. Run statically-linked binaries from your own media, redirect output to your own mount, note the exact time you started, and prefer capturing memory (acquisition sheet) before anything else.

Live response, in order

# 0. Your evidence mount and a log of what you run
E=/mnt/usb/evidence/$(hostname)-$(date -u +%Y%m%dT%H%MZ); mkdir -p "$E"; exec > >(tee -a "$E/collector.log") 2>&1
date -u; uptime; cat /etc/os-release; uname -a; hostnamectl 2>/dev/null

# 1. Memory (see acquisition sheet) — AVML needs no kernel module
/mnt/usb/bin/avml --compress "$E/mem.lime.compressed"

# 2. Volatile: who / what / where
w; last -Faiw | head -50; lastb -F | head -50
ps -eo pid,ppid,user,lstart,etime,stat,%cpu,%mem,cmd --forest
ss -tulpan; ss -xlp                      # sockets incl. unix
lsof -nP +L1                             # deleted-but-open files (malware unlinking itself)
lsof -nP -i                              # network by process
ip a; ip r; ip neigh; cat /etc/resolv.conf /etc/hosts
cat /proc/mounts; df -hT; lsblk -f
lsmod; cat /proc/modules                 # compare later for hidden modules
cat /proc/sys/kernel/tainted             # non-zero + no proprietary drivers = suspicious
systemctl list-units --type=service --state=running --no-pager
systemctl list-timers --all --no-pager
env; cat /proc/1/environ | tr '\0' '\n'  # LD_PRELOAD in PID 1's env?

# 3. Per-process detail for anything odd (deleted binary, weird cwd, no tty)
for p in $(pgrep -f suspicious); do
  echo "== $p"; ls -l /proc/$p/exe /proc/$p/cwd; cat /proc/$p/cmdline | tr '\0' ' '; echo
  cat /proc/$p/maps | head; ls -l /proc/$p/fd
  cp /proc/$p/exe "$E/proc-$p.exe"       # recover a deleted binary from memory
done

# 4. Users and auth surface
cat /etc/passwd /etc/group /etc/shadow /etc/sudoers; ls -la /etc/sudoers.d/
for h in /root /home/*; do echo "== $h"; ls -la "$h/.ssh" 2>/dev/null; cat "$h/.ssh/authorized_keys" 2>/dev/null; done
getent passwd | awk -F: '$3==0 || $7 !~ /(nologin|false)$/'   # UID 0 or interactive shells

# 5. Copy the artefacts you will parse offline (timestamps preserved)
tar --xattrs --acls -czpf "$E/logs.tgz" /var/log /etc /var/spool/cron /var/spool/at /root /home --exclude='*.iso' 2>/dev/null
journalctl -o export > "$E/journal.export"          # lossless journal dump
find / -xdev -newermt "$(date -d '7 days ago' +%F)" -type f 2>/dev/null > "$E/recent-files.txt"

[!tip] Automate it UAC (github.com/tclahr/uac) is the Linux/macOS/*BSD answer to KAPE: ./uac -p ir_triage /mnt/usb/evidence collects all of the above (and hundreds more artefacts) using its own statically-linked binaries, hashed, in minutes. Velociraptor does the same at fleet scale (Linux.Triage.* artifacts). CyLR is the older, smaller option. Use one of these on real cases; the manual list above is for understanding what they grab.

The /var/log map

Debian/Ubuntu on the left, RHEL/Fedora/SUSE where different.

# Binary login records
last -Faiw -f /mnt/img/var/log/wtmp          # -F full times, -a host in last column, -i IPs, -w full names
last -Faiw -f wtmp reboot                     # boots
lastb -F -f /mnt/img/var/log/btmp | awk '{print $3}' | sort | uniq -c | sort -rn | head    # top attacking IPs
utmpdump /mnt/img/var/log/wtmp                # raw records incl. terminal and pid; also: utmpdump wtmp > wtmp.txt; edit; utmpdump -r < wtmp.txt
lastlog -R /mnt/img                           # per-user last login from an image root
# Rotated logs: read them all at once
zgrep -h 'Accepted' /mnt/img/var/log/auth.log*  | sort -k1M -k2n

journalctl — the systemd journal

# Point at an image's journal dir (not the live one)
J="journalctl -D /mnt/img/var/log/journal --no-pager -o short-iso --utc"

$J --list-boots                                # boot IDs with first/last times
$J -b -1                                       # previous boot only
$J --since "2026-09-20" --until "2026-09-22 06:00"
$J -u ssh -u sshd                              # a unit (or several)
$J _COMM=sudo                                  # by process name
$J _UID=1001                                   # by user
$J -p warning..emerg                           # priority range (0 emerg … 7 debug)
$J -k                                          # kernel messages (dmesg equivalent, all boots)
$J _SYSTEMD_UNIT=cron.service                  # cron executions
$J -g 'Accepted|Failed password|session opened|COMMAND='   # regex grep across everything
$J -o verbose _PID=4242                        # every field of matching entries
$J -o json | jq -r 'select(.MESSAGE|test("useradd|usermod|passwd")) | [.__REALTIME_TIMESTAMP,.MESSAGE]|@tsv'
$J --disk-usage; $J --verify                   # is the journal intact? tampered files fail verification
$J -o export > journal.export                  # lossless copy; re-import with systemd-journal-remote

[!info] Volatile vs persistent If /var/log/journal/ does not exist, the journal was volatile (/run/log/journal) and died at power-off — that is your gap. Storage= in /etc/systemd/journald.conf tells you which. Check MaxRetentionSec/SystemMaxUse for how far back you can expect to see.

auditd — when it is on, it is the best log you have

cat /etc/audit/auditd.conf /etc/audit/rules.d/*.rules           # what was being recorded
auditctl -l                                                     # live rules

A="ausearch -if /mnt/img/var/log/audit/audit.log -i"            # -i = interpret UIDs/syscalls; -if = file
$A -m USER_LOGIN,USER_AUTH,USER_START                            # logins
$A -m USER_CMD; $A -m EXECVE                                     # sudo commands / every execve (if rule -a always,exit -S execve)
$A -ua 1001 -ts 09/20/2026 00:00:00 -te 09/22/2026 23:59:59     # by audit UID (survives su/sudo) in a window
$A -k persistence                                               # by rule key
$A -f /etc/passwd; $A -f /etc/shadow                            # who touched a file
$A -sc connect; $A -sc ptrace                                   # by syscall
$A -x /usr/bin/curl                                             # by executable
$A --session 42                                                 # everything in one login session
$A -m AVC                                                       # SELinux denials — malware tripping policy

aureport -if audit.log --summary                                # counts by type
aureport -if audit.log -au -i                                   # authentication report
aureport -if audit.log -x -i --summary                          # executables by frequency
aureport -if audit.log -f -i | grep -E '/tmp|/dev/shm|/var/tmp' # files touched in world-writable dirs

auid (audit UID) is the login user even after sudo -i; uid/euid show the effective identity. ses= links every record of a session. Records with key= came from a named rule — many hardening baselines (CIS, Neo23x0’s auditd rules) tag T1xxx MITRE keys straight into the log.

Shell histories and user activity

for h in /mnt/img/root /mnt/img/home/*; do
  for f in .bash_history .zsh_history .python_history .mysql_history .psql_history .lesshst .viminfo .wget-hsts .rediscli_history .sqlite_history .node_repl_history .local/share/fish/fish_history .local/share/nano/search_history .config/nvim/shada; do
    [ -s "$h/$f" ] && { echo "=== $h/$f ($(stat -c %y "$h/$f"))"; cat "$h/$f"; }
  done
done 2>/dev/null

# zsh extended history has epoch timestamps: ": 1758866400:0;whoami"
awk -F'[:;]' '/^: [0-9]+/ {cmd=$0; sub(/^: [0-9]+:[0-9]+;/,"",cmd); print strftime("%F %T",$2), cmd}' .zsh_history
# bash with HISTTIMEFORMAT set writes "#1758866400" lines above each command
awk '/^#[0-9]+$/ {ts=strftime("%F %T",substr($0,2)); next} {print ts, $0}' .bash_history

# Signs of history tampering
ls -la ~/.bash_history                 # symlink to /dev/null? size 0 with old mtime?
grep -E 'HISTSIZE=0|HISTFILE=|unset HIST|history -c|set \+o history' /mnt/img/etc/profile /mnt/img/etc/bash.bashrc /mnt/img/home/*/.bashrc /mnt/img/home/*/.profile 2>/dev/null
# Other traces: .viminfo (files edited, searches), .lesshst, .recently-used.xbel (GTK file dialogs), .cache/, Trash
cat /mnt/img/home/*/.local/share/recently-used.xbel | grep -o 'href="[^"]*"' | sort -u
ls -la /mnt/img/home/*/.local/share/Trash/{files,info}/

[!tip] History is written at shell exit A live attacker’s session has no history file yet. Read it from memory instead: cat /proc/<bash pid>/mem regions via gdb/pymem, or Volatility’s linux.bash plugin on the RAM image. ~/.bash_history also has no timestamps unless HISTTIMEFORMAT was set — order is all you get.

Persistence hunt

Check every one of these; attackers use the boring ones.

R=/mnt/img
# One-shot sweep of the file-based locations (adjust R for live: R=/)
grep -rE 'ExecStart|ExecStartPre|ExecStop' $R/etc/systemd/system $R/usr/lib/systemd/system $R/home/*/.config/systemd 2>/dev/null | grep -Ev 'ExecStart=/(usr/)?(s?bin|lib)/' | sort -u
cat $R/etc/crontab $R/etc/cron.d/* $R/var/spool/cron/crontabs/* $R/var/spool/cron/* 2>/dev/null | grep -Ev '^\s*(#|$)'
cat $R/etc/ld.so.preload 2>/dev/null; ls -la $R/etc/ld.so.preload 2>/dev/null
find $R/etc/systemd $R/etc/cron* $R/etc/profile.d $R/etc/init.d $R/etc/rc.local $R/etc/ssh $R/etc/pam.d $R/etc/sudoers.d $R/etc/udev/rules.d -type f -printf '%TY-%Tm-%Td %TH:%TM %p\n' 2>/dev/null | sort | tail -40
find $R -xdev -type f \( -path '*/tmp/*' -o -path '*/dev/shm/*' -o -path '*/var/tmp/*' \) -perm -u+x -ls 2>/dev/null
find $R -xdev -type f -name '.*' -perm -u+x -not -path '*/.git/*' -ls 2>/dev/null | head -50
find $R -xdev -type f -perm -4000 -printf '%TY-%Tm-%Td %p\n' 2>/dev/null | sort | tail -20
# Anything in a bin dir not owned by a package (Debian / RHEL)
for f in $R/usr/bin/* $R/usr/sbin/* $R/bin/* $R/sbin/*; do dpkg -S "${f#$R}" >/dev/null 2>&1 || echo "unowned: $f"; done 2>/dev/null | head

Package integrity

# Debian/Ubuntu — compare installed files against package MD5s (from the image's own dpkg DB)
debsums -c --root /mnt/img 2>/dev/null           # changed files only
debsums -ac --root /mnt/img                       # include config files
# RHEL/Fedora — S=size M=mode 5=md5 D=device L=link U=user G=group T=mtime P=caps
rpm -Va --root /mnt/img 2>/dev/null | grep -Ev '^\.{9}\s+c '   # ignore untouched configs
rpm -Va --root /mnt/img | awk '$1 ~ /5/'                       # content changed
rpm -qf /mnt/img/usr/bin/ls; rpm -V coreutils
# Arch
pacman -Qkk --root /mnt/img 2>/dev/null | grep -v '0 altered'
# Package DB itself tampered? Verify against the repo copy:
apt-get download coreutils && dpkg-deb -x coreutils_*.deb /tmp/clean && diff <(sha256sum /tmp/clean/usr/bin/ls) <(sha256sum /mnt/img/usr/bin/ls)

Rootkit scanners are a cheap second opinion, not a verdict: chkrootkit -r /mnt/img, rkhunter --check --rootdir /mnt/img. Better: debsums/rpm -Va plus comparing ls, ps, netstat, ss, lsof, find outputs against a static busybox on a live box (busybox ps vs ps).

Hidden processes and /proc

# PIDs in /proc that ps does not show (LD_PRELOAD rootkits hook readdir in libc, not the kernel)
diff <(ls /proc | grep -E '^[0-9]+$' | sort -n) <(ps -eo pid= | tr -d ' ' | sort -n)
# Brute-force /proc — bypasses readdir hooks
for p in $(seq 1 65535); do [ -d /proc/$p ] && [ ! -e /proc/$p/status.x ] && echo -n "$p "; done 2>/dev/null; echo
# Sockets with no owning process in ss (hidden by rootkit) vs /proc/net/tcp
cat /proc/net/tcp /proc/net/tcp6 | awk 'NR>1{print $2, $4, $10}'    # local addr (hex), state (0A=LISTEN), inode
# Process running from a deleted binary
ls -l /proc/*/exe 2>/dev/null | grep '(deleted)'
# Executable memory mappings not backed by a file (injected code, packers)
grep -E 'rwx|r-xp 00000000 00:00 0' /proc/*/maps 2>/dev/null | head
# Namespaces: container escapes / hidden via unshare
ls -la /proc/*/ns/pid 2>/dev/null | awk '{print $NF}' | sort | uniq -c
# Mount namespace tricks: bind-mount over /proc/<pid> to hide it
cat /proc/mounts | grep -E ' /proc/[0-9]+'
# Kernel modules: /proc/modules vs sysfs
diff <(awk '{print $1}' /proc/modules | sort) <(ls /sys/module | sort) | grep '^>'

Timestamps and timestomping

Every inode has atime, mtime, ctime, and on ext4/xfs/btrfs also crtime (birth). touch -d and utimensat change atime/mtime freely; ctime cannot be set from userspace (it moves to now on any change), and crtime never changes. A file whose mtime is 2019 but ctime is last Tuesday was modified last Tuesday.

stat file                                   # Access / Modify / Change / Birth
stat -c '%n %w %x %y %z' /mnt/img/usr/bin/*  | awk '$3 > $4 || $4 > $5'    # mtime after ctime = impossible without tampering
# crtime on ext4 when stat shows "-" for Birth (older coreutils):
debugfs -R "stat /usr/bin/ls" /dev/loop0p2 | grep -E 'crtime|ctime|mtime|atime'
# Files changed in a window, sorted by ctime (attacker cannot fake ctime)
find /mnt/img -xdev -type f -newerct '2026-09-20' ! -newerct '2026-09-22' -printf '%CY-%Cm-%Cd %CH:%CM %p\n' 2>/dev/null | sort
# Fractional-second timestamps: tools that set a whole-second time (touch -r, tar defaults) leave .000000000 — real writes almost never do
find /mnt/img/usr/bin -type f -printf '%TS %p\n' | grep -E '^00\.000000000' | head
# Mismatch between package mtime and actual
dpkg-query -W -f='${Package}\n' | head -1 >/dev/null; ls -l --time-style=full-iso /mnt/img/usr/bin/ps /mnt/img/usr/bin/pgrep   # siblings from one package share mtimes

[!info] relatime and noatime Most distros mount with relatime: atime updates only if it is older than mtime/ctime or >24 h old. So “atime = last read” is only sometimes true. Check /etc/fstab and /proc/mounts before concluding anything from access times.

Dead-disk analysis on SIFT Workstation

SIFT (SANS Investigative Forensic Toolkit) is the free Ubuntu-based analysis VM that FOR508 and most Linux DFIR modules run on: Sleuth Kit, plaso, Volatility, ewf-tools, libvshadow, RegRipper, Autopsy, bulk_extractor, YARA, hindsight, and the EZ CLI tools under .NET are preinstalled. Get it as an OVA from sans.org/tools/sift-workstation or bolt it onto any Ubuntu with sift install (Cast: github.com/teamdfir/sift-cli). Alternatives with the same toolset: Tsurugi Linux, CAINE, REMnux (malware focus), and Kali’s forensics metapackage.

Mount the image (read-only)

mmls sdb.E01                                       # partition table with start sectors (TSK reads E01 natively)
fsstat -o 2048 sdb.E01 | head -30                  # filesystem type, block size, last mount, label
# Mount via ewfmount → loop → mount (see acquisition sheet for LVM / LUKS)
sudo ewfmount sdb.E01 /mnt/ewf && sudo losetup -rfP --show /mnt/ewf/ewf1
sudo mount -o ro,noexec,noload /dev/loop0p2 /mnt/img            # ext4: noload = don't replay journal
sudo mount -o ro,noexec,norecovery /dev/loop0p3 /mnt/img/var    # xfs uses norecovery

Sleuth Kit essentials

O=2048                                           # partition offset in sectors from mmls
fls -o $O -r -p sdb.E01 > fls-all.txt            # recursive, full paths; deleted entries are marked "* " and "(realloc)"
fls -o $O -r -p -d sdb.E01                       # deleted only
fls -o $O -r -p -m / sdb.E01 > body.txt          # bodyfile (mactime format) with mount point /
istat -o $O sdb.E01 131074                       # timestamps, size, block list for inode 131074
icat -o $O sdb.E01 131074 > recovered.bin        # dump a (deleted) file by inode
ifind -o $O -n /etc/passwd sdb.E01               # inode for a path
ffind -o $O sdb.E01 131074                       # path(s) for an inode
tsk_recover -o $O -e sdb.E01 ./recovered/        # -e = everything incl. allocated; default = unallocated (deleted) only
blkls -o $O sdb.E01 > unalloc.raw                # unallocated blocks → feed to bulk_extractor / foremost
jls -o $O sdb.E01 | head; jcat -o $O sdb.E01 8 1234 | xxd | head    # journal block 1234 (old content)
tsk_loaddb -o $O -d case.db sdb.E01              # SQLite of the whole fs → query with sqlite3 / Autopsy

Timeline: mactime (fast) and plaso (complete)

# 1. Filesystem-only timeline from the bodyfile — seconds
mactime -b body.txt -d -z UTC 2026-09-15..2026-09-27 > fs-timeline.csv
# columns: Date,Size,Type(m a c b),Mode,UID,GID,Meta,File Name

# 2. Super-timeline with plaso: filesystem + logs + journal + histories + browser + apt/dpkg + cron + ... in one file
log2timeline.py --storage-file case.plaso --parsers linux /mnt/img                 # against the mounted tree (preset "linux")
log2timeline.py --storage-file case.plaso --partitions all sdb.E01                # or the image directly (TSK), all partitions
log2timeline.py --storage-file case.plaso --parsers 'linux,!filestat' --hashers sha256 /mnt/img   # tune parsers, add hashes
pinfo.py case.plaso                                                                # what got parsed, counts, errors

# 3. Output: filter a window and sort
psort.py -o l2tcsv -w timeline.csv case.plaso "date > '2026-09-20 00:00:00' AND date < '2026-09-23 00:00:00'"
psort.py -o dynamic --fields datetime,timestamp_desc,source,source_long,message,parser,display_name -w tl.csv case.plaso
psort.py -o l2tcsv case.plaso "message contains 'authorized_keys' OR message contains 'crontab'" | head
psort.py --analysis tagging --tagging-file /usr/share/plaso/tag_linux.txt case.plaso   # tag logins, sudo, package installs ...
psort.py -o opensearch ... / -o timesketch  → Timesketch for collaborative review

Open timeline.csv in Timeline Explorer (Windows VM) or visidata timeline.csv on SIFT; filter around the first suspicious login and read outward. timestamp_desc tells you which timestamp (Content Modification, Last Access, Metadata Modification, Creation); source tells you the artefact (FILE, LOG, WEBHIST, BASH, UTMP, JOURNAL).

[!tip] Pick the tool for the question fls/mactime: “what files changed between 02:00 and 03:00” — a minute of work. plaso: “everything that happened, from every source, in one sorted list” — an hour of parsing, but it is the artefact of record for the report.

Carving, strings and unknowns

Unallocated space (blkls output), swap, and unknown binaries go to strings & file triage: bulk_extractor, foremost/scalpel, strings, binwalk, YARA. Memory → Volatility 3 (linux.pslist, linux.bash, linux.lsof, linux.malfind, linux.check_syscall, linux.check_modules need a symbol table built with dwarf2json from the exact kernel’s debug symbols).

Quick answers

Related

• Acquisition — AVML/LiME memory, dc3dd/ewfacquire, mounting images
• Volatility 3 — the memory side
• strings & file triage — unknown binaries, carving unallocated space
• Digital Forensics reference — the cross-platform overview
• Linux credential & flag hunting — the same file locations from the other side of the table