Understanding System auditing with auditd
What is auditd
Auditing system is an important part of system administration. It allows us to audit minute details related to what exactly is happening within the system. auditd is a userspace component to the Linux auditing system. This means that system users will be able to run auditd to configure rules and alerts for auditing functionality with the Linux system.
One of the best things about auditd is that it is tightly integrated with the kernel, so it gives us the power to monitor almost everything we want, really. In order to allow users to see what is going on, auditd can record all the audit-related events to a disk and we can use various tools such as ausearch or aureport to go through the log files.
auditd is generally installed by default on all linux systems.
# rpm -qa | grep audit audit-libs-2.8.1-3.el7.x86_64 audit-libs-python-2.8.1-3.el7.x86_64 audit-2.8.1-3.el7.x86_64
In case, it is not installed, you can install it using the package manager, for e.g.:
# yum install audit
Make sure the audit service is enabled and active:
# systemctl status auditd ● auditd.service - Security Auditing Service Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2018-06-10 14:10:08 UTC; 54min ago Docs: man:auditd(8) https://github.com/linux-audit/audit-documentation Process: 658 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS) Process: 653 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS) Main PID: 654 (auditd) Tasks: 2 CGroup: /system.slice/auditd.service └─654 /sbin/auditd
Configuring the audit service
Since auditd doesn’t rely on syslog or any other external services, logging needs to be configured through the audit daemon’s configuration file /etc/audit/rules.d/auditd.conf.
By default, auditd logs to /var/log/audit/audit.log and rotates the file once it reaches 8 megabytes in size. After editing the configuration file the auditd service needs to be restarted, which can only be done through the “service” command instead of systemctl:
# service auditd restart
The reason for this limitation is that the audit daemon has to record any changes to the audit trail including starting and stopping the audit daemon itself. Systemctl uses dbus to talk to the system which hides who issued the command and looks like root did it.
Listing audit rules
By default, there are no rules configured in auditd. The command to check what rules are configured by auditd is “auditctl -l“.
# auditctl -l No rules
Auditing files/directories for changes using auditd
The Audit system operates on a set of rules that define what is to be captured in the log files. To audit access to a particular file or directory rules need to be in the following format:
-w [path] -p [permissions] -k [key]
path – is the full path of a file or directory and cannot contain wildcards, because it gets converted to an inode in the kernel. In case of a directory this rule will place a recursive watch on the directory and its whole subtree as long there’s no mount point under it.
Permissions – can be any combination of read, write, xecute or attribute change(permission/ownership of file).
key – is an arbitrary string of 31 bytes or less which can uniquely identify the audit records (logs) produced by the rule. It makes it easier to filter the logs or to group multiple rules tagged by the same key.
Example : monitor /etc/shadow for changes with Auditd
Let’s see an example of configuring an auditd rule to monitor write and attribute changes to the file /etc/shadow.
1. Edit the /etc/audit/rules.d/audit.rules and paste the line shown below:
# vi /etc/audit/rules.d/audit.rules -w /etc/shadow -p wa -k shadow
2. Restart the auditd deamon for the changes to take effect.
# service auditd restart
3. Verify the new rule:
# auditctl -l -w /etc/shadow -p wa -k shadow
Setting auditd rules temporarily
1. Temporary rules are useful to test new audit policies before setting them permanently for the system. These rules become effective immediately but then get wiped out at reboot. The rules from the above example can temporarily be added to the system using the auditctl command:
# auditctl -w /etc/shadow -p wa -k shadow
2. Active rules can be listed as:
# auditctl -l -w /etc/shadow -p wa -k shadow
Reading the audit log
Once the rules are active any changes made to the files or directories being watched will be recorded to the log file configured in auditd.conf (by default: /var/log/audit/audit.log).
For example, modifying the permissions of the /etc/shadow file with chmod produces the following entries:
type=SYSCALL msg=audit(1528729002.537:190): arch=c000003e syscall=268 success=yes exit=0 a0=ffffffffffffff9c a1=21300f0 a2=1ff a3=7fff25df27e0 items=1 ppid=1807 pid=1849 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm="chmod" exe="/usr/bin/chmod" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="shadow" type=CWD msg=audit(1528729002.537:190): cwd="/root" type=PATH msg=audit(1528729002.537:190): item=0 name="/etc/shadow" inode=638136 dev=ca:01 mode=0100000 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:shadow_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0 type=PROCTITLE msg=audit(1528729002.537:190): proctitle=63686D6F6400373737002F6574632F736861646F77
It clearly shows the type of access, the file name, audit (auid) and actual (uid) user IDs, the command (chmod) and the key identifying the audit rule that triggered the entry.
Using ausearch to read the logs
Even though reading the log file directly provides all the information we need, it’s not particularly user friendly or human readable. The audit package comes with the ausearch tool which produces a more human readable output:
# ausearch -i
The output from above command can be really long. But the results can also be filtered by different parameters such as key. For example, we can use the “shadow” key to filter out the logs.
type=PROCTITLE msg=audit(06/11/2018 14:56:42.537:190) : proctitle=chmod 777 /etc/shadow type=PATH msg=audit(06/11/2018 14:56:42.537:190) : item=0 name=/etc/shadow inode=638136 dev=ca:01 mode=file,000 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:shadow_t:s0 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 type=CWD msg=audit(06/11/2018 14:56:42.537:190) : cwd=/root type=SYSCALL msg=audit(06/11/2018 14:56:42.537:190) : arch=x86_64 syscall=fchmodat success=yes exit=0 a0=0xffffffffffffff9c a1=0x21300f0 a2=0777 a3=0x7fff25df27e0 items=1 ppid=1807 pid=1849 auid=user uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts1 ses=2 comm=chmod exe=/usr/bin/chmod subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=shadow
One of the advantages of reading the log file directly is to monitor it for changes using tail (tail -f /var/log/audit/audit.log). To achieve the same with ausearch use:
# watch -n2 'ausearch -i -k shadow | tail -20'
As an alternative to ausearch, you can use aureport to produce a series of audits in the following way:
1. To monitor unusual behavior, you can use:
# aureport --key --summary Key Summary Report =========================== total key =========================== 11 shadow
2. To build a report on user logins, you can use:
# aureport -l -i -ts yesterday -te today Login Report ============================================ # date time auid host term exe success event ============================================ 1. 06/10/2018 07:03:13 (unknown) 184.108.40.206 ssh /usr/sbin/sshd no 278 2. 06/10/2018 07:03:50 (unknown) 220.127.116.11 ssh /usr/sbin/sshd no 286 3. 06/10/2018 07:05:41 (unknown) 18.104.22.168 ssh /usr/sbin/sshd no 298 ...
3. To review access violations, you can try:
# ausearch --key access --raw | aureport --file --summary File Summary Report =========================== total file =========================== [no events of interest were found]
4. Finally, to review anomalies, you can use:
# aureport --anomaly Anomaly Report ========================================= # date time type exe term host auid event ========================================= 1. 01/07/2015 16:40:33 ANOM_ABEND systemd-tty-ask ? ? 1001 1010 2. 01/07/2015 17:47:12 ANOM_ABEND systemd-tty-ask ? ? 1001 74 3. 12/19/2015 23:04:44 ANOM_ABEND systemd-journal ? ? -1 98 4. 12/19/2015 23:04:44 ANOM_ABEND systemd-localed ? ? -1 99 ...
Excluding messages from audit log
Instead of filtering unwanted messages when reading the logs, it’s also possible to completely exclude them from the log file. The following rule excludes any Cron related messages by dropping entries with the subject type of ‘crond_t’:
# vi /etc/audit/rules.d/audit.rules -A exclude,always -F subj_type=crond_t
Exclude service start and stop messages:
# vi /etc/audit/rules.d/audit.rules -A exclude,always -F msgtype=SERVICE_START -A exclude,always -F msgtype=SERVICE_STOP