Anyone who has seen the Linux Audit logs will quickly disregard it’s usefulness on first glance. It is a deluge of nonsensical hexadecimals with almost no value. With a little Perl and Splunk, I have made it a powerful forensics tool. I can view command history, egress connections, file changes, and socket binds.
I wrote a plugin to audisp that is available on SourceForge at https://github.com/mtkirby/audisp-simplify/. The audisp daemon is an event multiplexer that allows for plugins to process audit data. My audisp plugin simply reads the data from stdin, consolidates relevent information from the multiple lines per event, and outputs to a logfile in a key=value format that is easily consumed by Splunk. My plugin will also translate uid to login names and translate saddr fields from bind/connect syscalls.
For example, here is the difference between the output from the Linux ausearch command compared to the output of my program. This is from a single command, uname -a
type=PATH msg=audit(1380483704.387:146400): item=1 name=(null) inode=2233758 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0
type=PATH msg=audit(1380483704.387:146400): item=0 name=”/bin/uname” inode=2228517 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0
type=CWD msg=audit(1380483704.387:146400): cwd=”/root”
type=EXECVE msg=audit(1380483704.387:146400): argc=2 a0=”uname” a1=”-a”
type=SYSCALL msg=audit(1380483704.387:146400): arch=c000003e syscall=59 success=yes exit=0 a0=1967b30 a1=195b3e0 a2=18a1270 a3=7fffaeeda150 items=2 ppid=2437 pid=9596 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=3 tty=pts1 comm=”uname” exe=”/usr/bin/uname” subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=”EXECVE”
And here is the log from my program:
type=EXECVE key=EXECVE auditid=146400 time=”2013-09-29 14:41:44-0500″ host=”goemon.lunari.net” tty=pts1 ppid=2437 pid=9596 origuser=mkirby user=root cwd=”/root” command=”uname -a”
Once it is in Splunk, I can easily search for command history from different users and piece together an entire shell session through the tty and ppid. My favorite key is the auid, which is the original login id. Almost all system changes are performed as the root user. If you need to trace who made a change as root, the auid will show that. My plugin will rename uid to “user” and auid to “origuser”.
Here is the Splunk query that I run: source=”/var/log/audisp-simplify” |table _time key tty ppid origuser user cwd exe name saddr command |sort _time desc
Here is the content from my wiki for further explanation and instructions:
README
This program is an audisp plugin that will make audit events from the Linux Audit daemon human readable.
It will look for execve, socketcall, bind, and connect system calls as well as filesystem changes (if you setup audit rules) and consolidate the events into a simple log format.
The log is in key=value format for easy consumption from Splunk or other log analytic software.
Here is a Splunk query that I run: source=”/var/log/audisp-simplify” |table _time key tty ppid origuser user cwd exe name saddr command |sort _time desc
The logs include the auid, which is the original login id. This is beneficial for seeing who made the syscalls after su’ing to another user such as root.
Here is an execve syscall example of a shell command, uname -a, executed as myself after su’ing to root:
type=EXECVE key=EXECVE auditid=143600 time=”2013-09-29 13:33:08-0500″ host=”goemon.lunari.net” tty=pts1 ppid=2437 pid=8189 origuser=mkirby user=root cwd=”/var/log” command=”uname -a”
Here is an example of /etc/hosts being edited:
type= key=FILE auditid=143645 time=”2013-09-29 13:36:19-0500″ host=”goemon.lunari.net” tty=pts1 ppid=2437 pid=8208 origuser=mkirby user=root cwd=”/var/log” exe=”/usr/bin/vi” name=”/etc/hosts”
Here is a connect syscall example of an egress connection to mkirby.org:
type=SOCKADDR key=CONNECT auditid=143682 time=”2013-09-29 13:38:49-0500″ host=”goemon.lunari.net” tty=pts1 ppid=2437 pid=8229 origuser=mkirby user=root saddr=”184.82.178.105 port 443″ exe=”/usr/bin/telnet”
Here is a bind syscall example of httpd binding to port 443:
type=SOCKADDR key=BIND auditid=143745 time=”2013-09-29 13:39:31-0500″ host=”goemon.lunari.net” tty=(none) ppid=1 pid=8252 user=root saddr=”0.0.0.0 port 443″ exe=”/usr/sbin/httpd”
INSTALLATION for RedHat, Centos, Scientific Linux, and Fedora
1. Place the script in /bin/ and chmod it to 750
2. Install the audit and audispd-plugins packages
3. Enable auditd on bootup and start the service
4. Create a new file, /etc/audisp/plugins.d/simplify.conf and add the following:
active = yes
direction = out
path = /bin/audisp-simplify
type = always
format = string
5) Increase queue in /etc/audisp/audispd.conf and set overflow_action to ignore
q_depth = 65536
overflow_action = ignore
5. Replace /etc/audit/audit.rules with the following: (you may want to add/del to dir monitoring)
# delete all existing rules
-D
# disable auditing during load
-e 0
# fail silently
-f 0
# 65k buffer
-b 65536
# no rate
-r 0
# continue loading if bad rule and report
-c
#
# Add any other dirs you want monitored for file writes
# These can be noisy during patching. Enable at your own risk
#-w /etc/ -p w -k FILE
#-w /root/ -p w -k FILE
#-w /var/spool/at/ -p w -k FILE
#-w /var/spool/cron/ -p w -k FILE
#-w /usr/lib/ -p w -k FILE
#-w /usr/lib64/ -p w -k FILE
#-w /usr/libexec/ -p w -k FILE
#-w /usr/bin/ -p w -k FILE
#-w /usr/sbin/ -p w -k FILE
#-w /usr/local/ -p w -k FILE
#-w /boot/ -p w -k FILE
#
# Monitor commands
-a exit,always -F arch=b32 -S execve -k EXECVE
-a exit,always -F arch=b64 -S execve -k EXECVE
#
# Monitor network connections.
# These are VERY noisy. Enable at your own risk
#-a exit,always -F arch=b32 -S socketcall -k SOCKETCALL -F exit!=-2
#-a exit,always -F arch=b64 -S bind -k BIND -F exit!=-2
#-a exit,always -F arch=b64 -S connect -k CONNECT -F exit!=-2
#
# activate auditing
-e 1
6. Modify /etc/audisp/audispd.conf and change the q_depth to 32768.
7. Setup log rotation by creating /etc/logrotate.d/audisp-simplify
Add the following:
/var/log/audisp-simplify
{
rotate 30
daily
create
compress
dateext
dateyesterday
shred
sharedscripts
create 0600 root root
postrotate
/sbin/service auditd restart >/dev/null 2>&1 || true
/bin/systemctl reload auditd.service >/dev/null 2>&1 || true
endscript
}
8. [OPTIONAL] Setup an ignores file for strings that you don’t want logged.
Create a new file /etc/audisp/simplify.ignores and use key=value pairs to specify what you don’t want logged.
The string values can be in Perl regex format.
Here is an example of my file:
saddr=netlink.*
saddr=public
saddr=private
saddr=/dev/log
saddr=.*port 53
saddr=.*:53
saddr=::::::: port
name=.*swx”$
name=.*swp”$
name=.*swpx”$
exe=”/var/ossec/bin/ossec-syscheckd”
exe=”/opt/splunk/bin/splunkd”
exe=”/opt/splunkforwarder/bin/splunkd”
9. Restart the auditd service
10. Done. Now you can watch the simple audit logs in /var/log/audisp-simplify
11. Periodically look at the output of auditctl -s. If it says that logs are lost, increase the q_depth in audispd.conf and the buffer, -b, in audit.rules
INSTALLATION ADDENDUM FOR SELINUX
If you are running SELinux, you may want to add the following to your local policy.
Add these lines to /etc/selinux/targeted/modules/active/src/local.te
module local 1.0;
require {
class dir { open getattr search write read remove_name add_name };
class file { create open read write execute execute_no_trans getattr };
type audisp_t;
type auditd_t;
type auditd_etc_t;
type auditd_log_t;
type var_log_t;
}
allow audisp_t var_log_t:file { create open read write execute execute_no_trans getattr };
allow audisp_t var_log_t:dir { write add_name };
allow audisp_t auditd_etc_t:dir { read search open };
allow audisp_t auditd_etc_t:file { read open getattr };
allow audisp_t auditd_log_t:dir { read search open };
allow audisp_t auditd_log_t:file { read open getattr };
and then run
cd /etc/selinux/targeted/modules/active/src/
checkmodule -M -m -o local.mod local.te
semodule_package -o local.pp -m local.mod
semodule -i local.pp