Control Your Logs AIX

Control Your Logs

 

 

 

Every AIX system generates log files, including those from third-party applications and admin scripts. If you don’t keep on top of log file maintenance, it can become a major headache pretty quickly. Ever tried searching for an event in a 3GB log file? It’s not fun. Keeping your system or application logs to a manageable size allows for better efficiency when doing investigative work. Truncating logs to keep the size down is not the answer because that destroys any history or audit trace on your system.

Rolling Those Logs

These logs, especially the text-based ones, need to be rolled, keeping an archive of the previous log. How many archives you keep and how often you roll a log is dependent on your own AIX audit policy, but I recommend log rolling every two days and keep four archives of the old log files. This ensures that the archived logs will get backed up to tape during the backup cycle. Older archives are then removed via a batch schedule job. This also ensures the filesystems where the logs reside don’t fill up.

When I say rolling a log, I mean to move that log to an archive and re-create the new log, if necessary. For instance, for the log, mylog, I could action:

$ cp mylogfile mylogfile`date +%d%m%y`
$ >mylogfile
$ compress mylogfile270614

We end up with a backup of mylogfile with the day-month-year appended to the file, in compressed format:

mylogfile270614.Z

Minimize Impact by Keeping Logs Out of Root Filesystem

As a general rule, I try to store all log files outside of the root filesystem. For the SMIT logs, I change the location of SMIT.log to /var/adm, using the SMIT options available, as these can get quite large very quickly when building a host.

The SMIT logs are important in two ways. First, they provide an audit on what admin tasks were implemented via SMIT. Second, they are a good learning resource when you wish to script certain tasks that have previously been done via SMIT, as this records the commands issued.

Don’t Mix and Match the Messages

Many AIX admins use sudo to allow elevated privileges for certain users, especially for batch jobs. Sudo pumps the messages of the commands executed to /var/adm/messages via syslog by default. This isn’t a good idea. You need to pump sudo messages to a separate file that only contains sudo entries. I put these to /var/adm/sudo.log. Be sure to create the file first before making changes to in the sudoers file:

$ >/var/adm/sudo.log

To redirect the sudo messages, simply add the following lines to an /etc/sudoers file:

#--- change sudo log
Defaults logfile=/var/adm/sudo.log
Defaults !syslog

Not All Logs Can be Rolled

One file that can very quickly grow is the wtmp file. This holds entries from last logins and accounting information. You can’t roll this log as it is in binary format. To quickly clear this log, use the following:

$ cp /dev/null  /var/adm/wtmp

If you wish to convert the wtmp file to text, use the following to convert the binary to text to a file called:/tmp/wtmp.txt:

$ fwtmp /tmp/wtmp.txt

As a general rule, I don’t save the contents of this file as I use the AIX audit system to log login/access entries.

Error and Audit

One logging file you don’t have to worry about is the system errpt file, as this a circular file. The file is held in /var/adm/ras/errlog. To see the current threshold of the errlog, use:

# /usr/lib/errdemon -l
Error Log Attributes
--------------------------------------------
Log File                /var/adm/ras/errlog
Log Size                1048576 bytes
Memory Buffer Size      32768 bytes
Duplicate Removal       true
Duplicate Interval      10000 milliseconds

From this output, the log size is set to 1 MB. If you want to set the threshold higher, for example to 5 MB, you could use:

# /usr/lib/errdemon -s 5242880

Without a doubt, AIX offers a comprehensive auditing system. You can have it monitor and report just admin-related tasks or audit the whole system. For my auditing, I created a policy, which includes all users, that reports on certain files that have been modified and user account changes. However, be advised this log can become very large very quickly, so a daily maintenance is absolutely required.

I stop the audit service, roll the log and manipulate it to make it more presentable. An email is then generated that’s sent to all AIX admins and IT security personnel. The audit service is then restarted. The old logs are held on the system for one year. When using auditing, be sure to create a separate filesystem to hold your logs or your filesystem will fill quickly.

For the /var/adm/messages log, you need to stop syslogd first, roll the log, and then restart syslogd, because the file descriptors will be open if you roll it while syslogd is running.

Target Your Log Files

Housekeeping log files is generally done by some third-party utility such as logrotate or rotatelog, or by a script written by a system admin. Either is generally executed via a scheduler such as cron. No matter which method you use, be sure it targets all known logs and rolls them correctly.

0 (0)
Article Rating (No Votes)
Rate this article
Attachments
There are no attachments for this article.
Comments
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
Setting new device attributes with chdef
Viewed 2210 times since Mon, Jun 3, 2019
AIX, System Admin Configuring dsh DSH
Viewed 3972 times since Fri, Apr 19, 2019
Getting bosboot errors, don’t reboot just yet
Viewed 2860 times since Tue, Apr 16, 2019
To do a quick check on the number of path present (does not mean all are Enabled] using for loop
Viewed 3821 times since Fri, Jun 8, 2018
AIX - How to shutdown or reboot
Viewed 5587 times since Fri, Jun 8, 2018
Install and configure GNU’s screen on AIX
Viewed 8932 times since Thu, Feb 21, 2019
How to set Kernel Parameters in AIX ?
Viewed 14822 times since Tue, Jul 2, 2019
The new VIOS performance advisor tool part util
Viewed 3226 times since Tue, Jun 4, 2019
AIX check the HBA status
Viewed 16543 times since Tue, May 22, 2018
Epoch & Unix Timestamp Conversion Tools
Viewed 61955 times since Fri, Jun 22, 2018