analyzing-linux-audit-logs-for-intrusion

Analyzing Linux Audit Logs for Intrusion

When to Use

• Investigating suspected unauthorized access or privilege escalation on Linux hosts
• Hunting for evidence of exploitation, backdoor installation, or persistence mechanisms
• Auditing compliance with security baselines (CIS, STIG, PCI-DSS) that require system call monitoring
• Reconstructing a timeline of attacker actions during incident response
• Detecting file tampering on critical system files such as /etc/passwd, /etc/shadow, or SSH keys

Do not use for network-level intrusion detection; use Suricata or Zeek for network traffic analysis. Auditd operates at the kernel level on individual hosts.

Prerequisites

• Linux system with auditd package installed and the audit daemon running (systemctl status auditd)
• Root or sudo access to configure audit rules and query logs
• Audit rules deployed via /etc/audit/rules.d/*.rules or loaded with auditctl
• Recommended: Neo23x0/auditd ruleset from GitHub for comprehensive baseline coverage
• Familiarity with Linux syscalls (execve, open, connect, ptrace, etc.)
• Log storage with sufficient retention (default location: /var/log/audit/audit.log)

Workflow

Step 1: Verify Audit Daemon Status and Configuration

Confirm the audit system is running and check the current rule set:

# Check auditd service status
systemctl status auditd

# Show current audit rules loaded in the kernel
auditctl -l

# Show audit daemon configuration
cat /etc/audit/auditd.conf | grep -E "log_file|max_log_file|num_logs|space_left_action"

# Check if the audit backlog is being exceeded (dropped events)
auditctl -s

If the backlog limit is being reached, increase it:

auditctl -b 8192

Step 2: Deploy Intrusion-Focused Audit Rules

Add rules that target common intrusion indicators. Place these in /etc/audit/rules.d/intrusion.rules:

# Monitor credential files for unauthorized reads or modifications
-w /etc/passwd -p wa -k credential_access
-w /etc/shadow -p rwa -k credential_access
-w /etc/gshadow -p rwa -k credential_access
-w /etc/sudoers -p wa -k privilege_escalation
-w /etc/sudoers.d/ -p wa -k privilege_escalation

# Monitor SSH configuration and authorized keys
-w /etc/ssh/sshd_config -p wa -k sshd_config_change
-w /root/.ssh/authorized_keys -p wa -k ssh_key_tampering

# Monitor user and group management commands
-w /usr/sbin/useradd -p x -k user_management
-w /usr/sbin/usermod -p x -k user_management
-w /usr/sbin/groupadd -p x -k user_management

# Detect process injection via ptrace
-a always,exit -F arch=b64 -S ptrace -F a0=0x4 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x5 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x6 -k process_injection

# Monitor execution of programs from unusual directories
-a always,exit -F arch=b64 -S execve -F exe=/tmp -k exec_from_tmp
-a always,exit -F arch=b64 -S execve -F exe=/dev/shm -k exec_from_shm

# Detect kernel module loading (rootkit installation)
-a always,exit -F arch=b64 -S init_module -S finit_module -k kernel_module_load
-a always,exit -F arch=b64 -S delete_module -k kernel_module_remove
-w /sbin/insmod -p x -k kernel_module_tool
-w /sbin/modprobe -p x -k kernel_module_tool

# Monitor network socket creation for reverse shells
-a always,exit -F arch=b64 -S socket -F a0=2 -k network_socket_created
-a always,exit -F arch=b64 -S connect -F a0=2 -k network_connection

# Detect cron job modifications (persistence)
-w /etc/crontab -p wa -k cron_persistence
-w /etc/cron.d/ -p wa -k cron_persistence
-w /var/spool/cron/ -p wa -k cron_persistence

# Monitor log deletion or tampering
-w /var/log/ -p wa -k log_tampering

Reload rules after editing:

augenrules --load
auditctl -l | wc -l   # Confirm rule count

Step 3: Search for Intrusion Indicators with ausearch

Use ausearch to query the audit log for specific events:

# Search for all failed login attempts in the last 24 hours
ausearch -m USER_LOGIN --success no -ts recent

# Search for commands executed by a specific user
ausearch -ua 1001 -m EXECVE -ts today

# Search for all file access events on /etc/shadow
ausearch -f /etc/shadow -ts this-week

# Search for privilege escalation via sudo
ausearch -m USER_CMD -ts today

# Search for kernel module loading events
ausearch -k kernel_module_load -ts this-month

# Search for processes executed from /tmp (common attack staging)
ausearch -k exec_from_tmp -ts this-week

# Search for SSH key modifications
ausearch -k ssh_key_tampering -ts this-month

# Search for a specific event by audit event ID
ausearch -a 12345

# Search events in a specific time range
ausearch -ts 03/15/2026 08:00:00 -te 03/15/2026 18:00:00

# Interpret syscall numbers and format output readably
ausearch -k credential_access -i -ts today

Step 4: Generate Summary Reports with aureport

Use aureport to produce aggregate summaries for triage:

# Summary of all authentication events
aureport -au -ts this-week --summary

# Report of all failed events (login, access, etc.)
aureport --failed --summary -ts today

# Report of executable runs
aureport -x --summary -ts today

# Report of all anomaly events (segfaults, promiscuous mode, etc.)
aureport --anomaly -ts this-week

# Report of file access events
aureport -f --summary -ts today

# Report of all events by key (maps to your custom rule keys)
aureport -k --summary -ts this-month

# Report of all system calls
aureport -s --summary -ts today

# Report of events grouped by user
aureport -u --summary -ts this-week

# Detailed time-based event report for timeline building
aureport -ts 03/15/2026 08:00:00 -te 03/15/2026 18:00:00 --summary

Step 5: Reconstruct the Attack Timeline

Combine ausearch queries to build a chronological narrative:

# Step 5a: Identify the initial access timestamp
ausearch -m USER_LOGIN -ua 0 --success yes -ts this-week -i | head -50

# Step 5b: Trace what the attacker did after gaining access
# Get all events from the compromised account within the incident window
ausearch -ua <UID> -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i \
  | aureport -f -i

# Step 5c: Extract all commands executed during the incident window
ausearch -m EXECVE -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i

# Step 5d: Check for persistence mechanisms installed
ausearch -k cron_persistence -ts "03/15/2026 14:00:00" -i
ausearch -k ssh_key_tampering -ts "03/15/2026 14:00:00" -i

# Step 5e: Check for lateral movement (outbound connections)
ausearch -k network_connection -ts "03/15/2026 14:00:00" -i

Step 6: Forward Audit Logs to SIEM

Configure audisp-remote or auditbeat to ship logs to a central SIEM for correlation:

# Option A: Using audisp-remote plugin
# Edit /etc/audit/plugins.d/au-remote.conf
active = yes
direction = out
path = /sbin/audisp-remote
type = always

# Configure remote target in /etc/audit/audisp-remote.conf
remote_server = siem.internal.corp
port = 6514
transport = tcp

# Option B: Using Elastic Auditbeat
# Install auditbeat and configure /etc/auditbeat/auditbeat.yml
# Auditbeat reads directly from the kernel audit framework

Key Concepts

Term	Definition
auditd	The Linux Audit daemon that receives audit events from the kernel and writes them to /var/log/audit/audit.log
auditctl	Command-line utility to control the audit system: add/remove rules, check status, set backlog size
ausearch	Query tool that searches audit logs by message type, user, file, key, time range, or event ID
aureport	Reporting tool that generates aggregate summaries of audit events for triage and compliance
audit rule key (-k)	A user-defined label attached to an audit rule, enabling fast filtering of related events with ausearch and aureport
syscall auditing	Kernel-level monitoring of system calls (execve, open, connect, ptrace) that captures process and file activity
augenrules	Utility that merges all files in /etc/audit/rules.d/ into /etc/audit/audit.rules and loads them into the kernel

Verification

• [ ] auditd is running and rules are loaded (auditctl -l returns expected rule count)
• [ ] No audit backlog overflow (auditctl -s shows backlog: 0 or low value, lost: 0)
• [ ] ausearch returns events for each custom key (ausearch -k <key> -ts today returns results)
• [ ] aureport generates non-empty summaries for authentication, executable, and file events
• [ ] Timeline reconstruction produces a coherent chronological sequence of attacker actions
• [ ] Critical file watches trigger alerts on test modifications (touch /etc/shadow generates an event)
• [ ] Logs are forwarding to central SIEM (verify with a test event and confirm receipt)
• [ ] Audit rules persist across reboot (rules in /etc/audit/rules.d/, not only via auditctl)