When a Linux server is compromised, rapid and precise forensic evidence collection is critical. Every passing second, volatile data can be lost, rendering incident analysis incomplete or even misleading. The primary goal is to capture a snapshot of the system’s state at the time of compromise, while minimizing evidence alteration. This script, developed and tested in real-world incident response scenarios, automates the collection of essential information, from active processes to network connections, system logs to recently modified files, providing a solid foundation for subsequent forensic investigation.
Tested on: Ubuntu 22.04 LTS · Debian 11 · CentOS 7 · [July 2024]
Prerequisites / Test Environment
To run the script, you need root access or a user with sudo access on the compromised system. Ideally, the script should be run from non-persistent media (e.g., a live USB boot) or copied and executed from a temporary directory, to reduce disk impact and prevent evidence overwriting. Ensure that standard commands like tar, gzip, sha256sum, ps, netstat, ls, cat, find, mount are available on the system, as they are typically present in all Linux distributions. The script is designed to be as distribution-agnostic as possible.
1. Understanding Volatile Data Priority
In a forensic investigation, data is classified by its volatility. The most volatile data changes rapidly or is lost upon system reboot, such as processes in memory and active network connections. Collecting this data must be the first priority to preserve an accurate image of the system’s state at the time of the incident. The script starts precisely here, capturing the most ephemeral data before it can be altered or lost.
2. The Forensic Evidence Collection Script
This Bash script is designed to run on a potentially compromised Linux system. It collects a series of critical information and packages it into a timestamped tar.gz archive for easier subsequent analysis. Read also: How to automate report generation with Ansible
#!/bin/bash
# Forensic evidence collection script for Linux systems
# Author: Rosario Giordano
# Version: 1.0
# Date: 2024-07-22
# Check for root privileges
if [[ $EUID -ne 0 ]]; then
echo "This script must be run as root or with sudo."
exit 1
fi
OUTPUT_DIR="/tmp/forensic_collection_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$OUTPUT_DIR"
echo "Starting forensic evidence collection in $OUTPUT_DIR..."
# 1. Volatile Data
# Running processes
ps auxww > "$OUTPUT_DIR/ps_auxww.txt"
ps -ef > "$OUTPUT_DIR/ps_ef.txt"
ls -l /proc/*/exe > "$OUTPUT_DIR/proc_exe_links.txt" 2>/dev/null
ls -l /proc/*/fd > "$OUTPUT_DIR/proc_fd_links.txt" 2>/dev/null
# Network connections
netstat -tulnp > "$OUTPUT_DIR/netstat_tulnp.txt"
netstat -anp > "$OUTPUT_DIR/netstat_anp.txt"
ss -tulnp > "$OUTPUT_DIR/ss_tulnp.txt" # Alternative to netstat on modern systems
ss -anp > "$OUTPUT_DIR/ss_anp.txt"
# ARP tables
arp -an > "$OUTPUT_DIR/arp_an.txt"
# DNS cache
cat /etc/resolv.conf > "$OUTPUT_DIR/resolv.conf"
# For specific service DNS caches (e.g., systemd-resolved, dnsmasq), specific integration might be needed
# Environment variables (for specific processes, if relevant)
# Example: cat /proc/PID/environ
# 2. System and Configuration Information
# Kernel info and OS version
uname -a > "$OUTPUT_DIR/uname_a.txt"
cat /etc/os-release > "$OUTPUT_DIR/os_release.txt"
# Users and Groups
cat /etc/passwd > "$OUTPUT_DIR/passwd.txt"
cat /etc/shadow > "$OUTPUT_DIR/shadow.txt"
cat /etc/group > "$OUTPUT_DIR/group.txt"
cat /etc/sudoers > "$OUTPUT_DIR/sudoers.txt" 2>/dev/null
# Network configurations
ifconfig -a > "$OUTPUT_DIR/ifconfig_a.txt"
ip a > "$OUTPUT_DIR/ip_a.txt"
ip r > "$OUTPUT_DIR/ip_r.txt"
cat /etc/network/interfaces > "$OUTPUT_DIR/network_interfaces.txt" 2>/dev/null
# Running services
systemctl list-units --type=service > "$OUTPUT_DIR/systemctl_services.txt"
# Cron jobs
crontab -l > "$OUTPUT_DIR/crontab_root.txt" 2>/dev/null
# For other users: for user in $(cut -f1 -d: /etc/passwd); do crontab -l -u $user 2>/dev/null >> "$OUTPUT_DIR/crontab_${user}.txt"; done
# 3. System Logs
# Recent logs (last 24h or more, depending on rotation)
journalctl --no-pager --since "1 day ago" > "$OUTPUT_DIR/journalctl_last_day.log"
find /var/log -maxdepth 1 -type f -name "*.log" -o -name "messages" -o -name "syslog" -o -name "auth.log" -exec cp --parents {} "$OUTPUT_DIR/logs/" \; 2>/dev/null
# 4. Suspicious Files and Directories
# Files modified in the last X days (e.g., 7 days)
find / -type f -mtime -7 -print0 2>/dev/null | xargs -0 tar -rf "$OUTPUT_DIR/recently_modified_files.tar" 2>/dev/null
# Files with SUID/SGID permissions
find / -type f -perm /6000 -exec ls -la {} \; > "$OUTPUT_DIR/suid_sgid_files.txt" 2>/dev/null
# Hidden files in home directory (e.g., .bashrc, .profile)
find /home -maxdepth 2 -type f -name ".*" -print0 2>/dev/null | xargs -0 tar -rf "$OUTPUT_DIR/hidden_home_files.tar" 2>/dev/null
# 5. Evidence Integrity
# Generate SHA256 hashes for all collected files
find "$OUTPUT_DIR" -type f -exec sha256sum {} \; > "$OUTPUT_DIR/sha256sums.txt"
# 6. Compression and Cleanup
# Compress the entire directory into a tar.gz archive
ARCHIVE_NAME="forensic_evidence_$(date +%Y%m%d_%H%M%S).tar.gz"
tar -czvf "/tmp/$ARCHIVE_NAME" -C "$(dirname "$OUTPUT_DIR")" "$(basename "$OUTPUT_DIR")"
echo "Collection complete. Archive created at /tmp/$ARCHIVE_NAME"
echo "Deleting temporary directory $OUTPUT_DIR..."
rm -rf "$OUTPUT_DIR"
echo "Script completed."3. Script Section Analysis
The script is structured to cover the most critical areas in case of an incident. Read also: Wazuh vs Security Onion: 2-Week SIEM/EDR Test
Volatile Data
This section is the most important. ps auxww and ps -ef capture all running processes with their full arguments, revealing any malicious or anomalous processes. netstat -tulnp and ss -tulnp show active network connections and listening programs, crucial for identifying C2 (Command and Control) communications or unexpected services. Links in /proc//exe and /proc//fd can reveal processes that have deleted their binaries from disk (fileless malware) or open files to unusual destinations.
System and Configuration Information
Capturing uname -a and /etc/os-release provides essential context about the operating system version. Files like /etc/passwd, /etc/shadow, /etc/group are vital for identifying compromised or illicitly added user accounts. /etc/sudoers reveals who has elevated privileges. Network configurations (ifconfig, ip a, /etc/network/interfaces) can show changes to IP addresses, routes, or interfaces. System services (systemctl) and cron jobs (crontab -l) are key points where attackers often establish persistence.
System Logs
Logs are the narrative of events. journalctl (for modern systems with systemd) and files in /var/log (auth.log, syslog, messages) contain records of access, errors, failed login attempts, and suspicious activity. Their analysis is fundamental for reconstructing the attack timeline.
Suspicious Files and Directories
find is a powerful tool to identify recently modified files (-mtime), files with unusual SUID/SGID permissions (often used for privilege escalation), or hidden files that might contain malicious configurations. Including these files in a separate archive facilitates targeted analysis. Read also: Honeypot August: 7 Days, 1,200 SSH Attacks
4. Common Errors and Troubleshooting
- Execution on a compromised system: Running the script on the compromised system can alter evidence. Ideally, you should mount the compromised system’s disk on a separate forensic machine and collect data from there. If not possible, run the script in
/tmpand immediately transfer the archive. - Permissions: Ensure the script has execute permissions (
chmod +x script.sh) and is run with root privileges (sudo ./script.sh). - Disk space: Collecting logs and modified files can generate a large archive. Check available space in
/tmpor at the archive’s destination. - Missing commands: If a command like
ssorjournalctlis not present (on very old or minimal distributions), the script will continue, but data for that section will be missing. This is an acceptable compromise for a generic script. - Time Stomping: Attackers can modify file timestamps. The SHA256 hashes and the script’s collection timestamps are crucial for demonstrating evidence integrity. Read also: NIS2 Healthcare: Technical Processing Register
FAQ — Frequently Asked Questions
Do I need to modify the script for every incident?
No, the script is designed to be generic. However, you might want to add specific sections for particular software (e.g., database configurations, web application logs) if you know they are installed on the target system. The important thing is to maintain its non-invasive nature and the collection of critical data.
Where should I save the evidence archive?
The archive is created in /tmp. It is crucial to transfer it as soon as possible to a secure, offline storage system, such as a centralized log server or a dedicated USB device for forensic analysis, to ensure its integrity and availability for future investigations.
Is this script sufficient for a complete forensic analysis?
This script provides an excellent foundation for initial evidence collection. It does not replace in-depth forensic analysis, which might include full disk imaging, RAM memory analysis, and the use of specialized forensic tools. It is a quick and robust first step to “stop time” and capture the most critical data.
How can I verify the integrity of the archive after transfer?
The script generates a sha256sums.txt file inside the archive. After transferring the archive, you can extract this file and re-run sha256sum -c sha256sums.txt within the extracted directory to verify that no files have been altered during transfer or storage.
Conclusions with Operational Takeaways
Managing an incident on a Linux system requires promptness and methodology. This script provides an operational framework for rapid and systematic forensic evidence collection. The main takeaway is that automation at this stage not only accelerates the process but significantly reduces the risk of human error and the loss of crucial data. Having such a tool at hand, tested and understood, is an invaluable asset for any security team or sysadmin. Always remember to transfer collected evidence to a secure system and avoid analyzing directly on the compromised system to preserve the chain of custody. Read also: PSN Cloud Migration: The Reality Behind the 2026 Mandate
Updated: July 2024
