Control Your Logs

Print 0

 

Every AIX system generates log files, including those from third-party applications and admin scripts. If you don’t keep on top of log file maintenance, it can become a major headache pretty quickly. Ever tried searching for an event in a 3GB log file? It’s not fun. Keeping your system or application logs to a manageable size allows for better efficiency when doing investigative work. Truncating logs to keep the size down is not the answer because that destroys any history or audit trace on your system.

Rolling Those Logs

These logs, especially the text-based ones, need to be rolled, keeping an archive of the previous log. How many archives you keep and how often you roll a log is dependent on your own AIX audit policy, but I recommend log rolling every two days and keep four archives of the old log files. This ensures that the archived logs will get backed up to tape during the backup cycle. Older archives are then removed via a batch schedule job. This also ensures the filesystems where the logs reside don’t fill up.

When I say rolling a log, I mean to move that log to an archive and re-create the new log, if necessary. For instance, for the log, mylog, I could action:

$ cp mylogfile mylogfile`date +%d%m%y`
$ >mylogfile
$ compress mylogfile270614

We end up with a backup of mylogfile with the day-month-year appended to the file, in compressed format:

mylogfile270614.Z

Minimize Impact by Keeping Logs Out of Root Filesystem

As a general rule, I try to store all log files outside of the root filesystem. For the SMIT logs, I change the location of SMIT.log to /var/adm, using the SMIT options available, as these can get quite large very quickly when building a host.

The SMIT logs are important in two ways. First, they provide an audit on what admin tasks were implemented via SMIT. Second, they are a good learning resource when you wish to script certain tasks that have previously been done via SMIT, as this records the commands issued.

Don’t Mix and Match the Messages

Many AIX admins use sudo to allow elevated privileges for certain users, especially for batch jobs. Sudo pumps the messages of the commands executed to /var/adm/messages via syslog by default. This isn’t a good idea. You need to pump sudo messages to a separate file that only contains sudo entries. I put these to /var/adm/sudo.log. Be sure to create the file first before making changes to in the sudoers file:

$ >/var/adm/sudo.log

To redirect the sudo messages, simply add the following lines to an /etc/sudoers file:

#--- change sudo log
Defaults logfile=/var/adm/sudo.log
Defaults !syslog

Not All Logs Can be Rolled

One file that can very quickly grow is the wtmp file. This holds entries from last logins and accounting information. You can’t roll this log as it is in binary format. To quickly clear this log, use the following:

$ cp /dev/null  /var/adm/wtmp

If you wish to convert the wtmp file to text, use the following to convert the binary to text to a file called:/tmp/wtmp.txt:

$ fwtmp /tmp/wtmp.txt

As a general rule, I don’t save the contents of this file as I use the AIX audit system to log login/access entries.

Error and Audit

One logging file you don’t have to worry about is the system errpt file, as this a circular file. The file is held in /var/adm/ras/errlog. To see the current threshold of the errlog, use:

# /usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds

From this output, the log size is set to 1 MB. If you want to set the threshold higher, for example to 5 MB, you could use:

# /usr/lib/errdemon -s 5242880

Without a doubt, AIX offers a comprehensive auditing system. You can have it monitor and report just admin-related tasks or audit the whole system. For my auditing, I created a policy, which includes all users, that reports on certain files that have been modified and user account changes. However, be advised this log can become very large very quickly, so a daily maintenance is absolutely required.

I stop the audit service, roll the log and manipulate it to make it more presentable. An email is then generated that’s sent to all AIX admins and IT security personnel. The audit service is then restarted. The old logs are held on the system for one year. When using auditing, be sure to create a separate filesystem to hold your logs or your filesystem will fill quickly.

For the /var/adm/messages log, you need to stop syslogd first, roll the log, and then restart syslogd, because the file descriptors will be open if you roll it while syslogd is running.

Target Your Log Files

Housekeeping log files is generally done by some third-party utility such as logrotate or rotatelog, or by a script written by a system admin. Either is generally executed via a scheduler such as cron. No matter which method you use, be sure it targets all known logs and rolls them correctly.