OSSEC on mkirby.org | mkirby.org

I run OSSEC HIDS/FIM at mkirby.org.
If you don’t know what that is, goto http://ossec.net.
The goal of this post isn’t to tell you why I run it, or how awesome it is. I’m going to tell you how my environment is configured.

The mkirby.org webserver runs the OSSEC agent and is monitored/configured by the OSSEC server, which is on a separate network filtered by a firewall. I use the client/server configuration because it is more secure than the local-only configuration. With local-only, a badguy who gains root access can tamper with the configuration/alerting/database to hide their tracks. With client/server, everything is handled and protected on the server.
The OSSEC server pushes the configuration to the agent and records the alerts/logs, from the agent, in Splunk. I wrote a script that reads /var/ossec/logs/alerts/alerts.log and reformats into key=value pairs for easy consumption in Splunk. The script is called ossec2splunk.pl is hosted at SourceForge at https://github.com/mtkirby/ossec2splunk. I also wrote a script to email me the FIM alerts. It is also hosted at SourceForge in the ossec2splunk project. The script is called osseccompactor.pl. It is written in a similar style as the LogCompactor scripts I wrote and hosted at SplunkBase. The osseccompactor script formats the FIM alerts into an html table and emails it to me. The script was written for an environment with thousands of servers, such as a cloud. The email shows the names of the modified files, the permissions/ownership (if they changed), and the contextual difference. It does not show the checksums, file sizes, and hostnames. That information just bloats the email alert with useless data. One of my biggest gripes with FIMs is that they show the checksum of the file, which doesn’t tell me anything other than “something changed”. With my osseccompactor email alert, I can see what in the file changed, what lines were removed/added. The osseccompactor email also contains links that go to Splunk so that I can easily dig deeper into an alert. If you want to run ossec2splunk, you will need to setup a Splunk Search to run a wrapper shell script for osseccompactor. If you setup LogCompactor, then you’ll be half-way there.

In Splunk, I setup a search macro called ‘ossectable’. I can run a search query and pipe it through `ossectable` to format the output as a table. Here is the search macro: table Integrity_checksum_changed_for hostname New_sha1sum_is oldsize newsize oldowner newowner oldgroup newgroup oldpermissions newpermissions deleted diffrm diffadd |sort Integrity_checksum_changed_for ASC

To enable remote commands from the manager, I set logcollector.remote_commands=1 in etc/internal_options.conf. This allows the scripts that block IPs to run. When the logcollector on the manager finds suspicious activity in the system logs, it will tell the agent to block the offending IP.

Below is my agent.conf.
A few things to note that many admins don’t think about are:

The “report_changes” on etc, cron, and ssh authorized_keys files. This will show the contextual change that was made to the files.
The alert_new_files is enabled. By default, it doesn’t alert on new files.
I don’t do real-time checking on file changes because it isn’t capable of reporting new files.
I changed ‘netstat -an’ to ‘netstat -peanut’. The peut flags add the process name/id, userid, and other info to the output. I always write it as ‘-peanut’ because it’s easy to remember. These flags are only available in Linux. For BSD/other, I would use ‘lsof -Pni’.
The file change alerts do not tell you who made the change. For that, see my posting about the Linux audit system and how I set it up to log shell/file activity to Splunk

<agent_config>
<global>
    <email_notification>no</email_notification>
    <email_to>ossecwatch@mkirby.org</email_to>
    <smtp_server>127.0.0.1</smtp_server>
    <email_from>ossecm@mkirby.org</email_from>
</global>

<rules>
    <include>rules_config.xml</include>
    <include>pam_rules.xml</include>
    <include>sshd_rules.xml</include>
    <include>syslog_rules.xml</include>
    <include>named_rules.xml</include>
    <include>smbd_rules.xml</include>
    <include>proftpd_rules.xml</include>
    <include>hordeimp_rules.xml</include>
    <include>wordpress_rules.xml</include>
    <include>web_rules.xml</include>
    <include>web_appsec_rules.xml</include>
    <include>apache_rules.xml</include>
    <include>mysql_rules.xml</include>
    <include>firewall_rules.xml</include>
    <include>postfix_rules.xml</include>
    <include>dovecot_rules.xml</include>
    <include>spamd_rules.xml</include>
    <include>ossec_rules.xml</include>
    <include>local_rules.xml</include>
</rules>

<syscheck>
    <frequency>7200</frequency>
    <disabled>no</disabled>
    <alert_new_files>yes</alert_new_files>
    <scan_on_start>yes</scan_on_start>
    <auto_ignore>no</auto_ignore>

    <!– Directories to check (perform all possible verifications) –>
    <directories realtime=”no” report_changes=”yes” check_all=”yes”>/etc,/var/spool/cron,/var/spool/at,/var/named/chroot/etc,/var/ossec/etc,/root/.ssh/authorized_keys,/var/ossec/.ssh/authorized_keys,/home/*/.ssh/authorized_keys</directories>
    <directories realtime=”no” check_all=”yes”>/opt,/var/www,/boot,/lib,/lib64,/usr,/var/ossec/bin</directories>
    <directories realtime=”no” check_owner=”yes” check_group=”yes” check_perm=”yes”>/dev</directories>

    <!– Files/directories to ignore –>
    <ignore>/etc/mtab</ignore>
    <ignore>/etc/mnttab</ignore>
    <ignore>/etc/hosts.deny</ignore>
    <ignore>/etc/mail/statistics</ignore>
    <ignore>/etc/random-seed</ignore>
    <ignore>/etc/adjtime</ignore>
    <ignore>/etc/httpd/logs</ignore>
    <ignore>/etc/utmpx</ignore>
    <ignore>/etc/wtmpx</ignore>
    <ignore>/etc/cups/certs</ignore>
    <ignore>/etc/dumpdates</ignore>
    <ignore>/etc/svc/volatile</ignore>
    <ignore>/etc/selinux/targeted</ignore>
    <ignore>/etc/ld.so.cache</ignore>
    <ignore>/etc/webmin/system-status</ignore>
    <ignore>/opt/splunk/var</ignore>
</syscheck>

<rootcheck>
    <frequency>7200</frequency>
    <disabled>no</disabled>
    <rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
    <rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>
    <system_audit>/var/ossec/etc/shared/system_audit_rcl.txt</system_audit>
</rootcheck>

<global>
    <white_list>127.0.0.1</white_list>
    <white_list>^localhost.localdomain$</white_list>
    <white_list>192.168.0.15</white_list>
    <white_list>8.8.8.8</white_list>
    <white_list>192.168.1.50</white_list>
    <white_list>192.168.1.10</white_list>
    <white_list>192.168.1.1</white_list>
    <white_list>192.168.1.4</white_list>
    <white_list>192.168.1.5</white_list>
</global>

<command>
    <name>host-deny</name>
    <executable>host-deny.sh</executable>
    <expect>srcip</expect>
    <timeout_allowed>yes</timeout_allowed>
</command>

<command>
    <name>firewall-drop</name>
    <executable>firewall-drop.sh</executable>
    <expect>srcip</expect>
    <timeout_allowed>yes</timeout_allowed>
</command>

<command>
    <name>restart-ossec</name>
    <executable>restart-ossec.sh</executable>
    <expect></expect>
</command>

<command>
    <name>route-null</name>
    <executable>route-null.sh</executable>
    <expect>srcip</expect>
    <timeout_allowed>yes</timeout_allowed>
</command>

<!– Active Response Config –>
<active-response>
    <!– This response is going to execute the host-deny
       – command for every event that fires a rule with
       – level (severity) >= 6.
       – The IP is going to be blocked for 600 seconds.
      –>
    <command>host-deny</command>
    <location>local</location>
    <level>6</level>
    <timeout>600</timeout>
</active-response>

<active-response>
    <!– Firewall Drop response. Block the IP for
       – 600 seconds on the firewall (iptables,
       – ipfilter, etc).
      –>
    <command>firewall-drop</command>
    <location>local</location>
    <level>6</level>
    <timeout>600</timeout>
</active-response>

<!– Files to monitor (localfiles) –>

<localfile>
<log_format>syslog</log_format>
<location>/var/log/messages</location>
</localfile>

<localfile>
<log_format>syslog</log_format>
<location>/var/log/secure</location>
</localfile>

<localfile>
<log_format>syslog</log_format>
<location>/var/log/maillog</location>
</localfile>

<localfile>
<log_format>apache</log_format>
<location>/var/log/httpd/*log</location>
</localfile>

<localfile>
<log_format>full_command</log_format>
<command>netstat -peanut |grep LISTEN |grep -v 127.0.0.1 | sort</command>
</localfile>
</agent_config>