Network problem determination: AIX tools for a system administrator

Part 2, Detailed diagnosis and troubleshooting

This article provides you with a set of commands available on IBM AIX®, many of which are also available on other flavors of UNIX®, that can help you get as much information as you can about exactly what is going on when your host has problems communicating with another. It also provides a logical step-by-step approach to diagnosing common issues.

For the purposes of this article, the target host system used in all sample commands and output is called testhost.

Tell me more

Depending on the nature of the network problem you're diagnosing, it's sometimes worth investigating whether the failing application or command has any kind of verbose, trace, or debug options. For example, both the ssh (Secure Shell) and scp (Secure Copy) commands have a verbose switch (-v) that can provide you with an extensive trace of the communication, key exchange, and authentication that takes place between client and server (see Listing 1).

Listing 1. Connecting to a remote host with a verbose ssh session

If you have login access to the problematic host (ideally the server failing to service network requests to a particular port, although sometimes errors can be reported on the requesting client as well), you should check system logs for relevant messages. These include files such as /var/adm/messages, /var/log/syslog, and /var/log/mail, depending on how the system logging daemon is configured in /etc/syslog.conf, as well as daemon-specific logs if any exist (for example, ftpd, sshd, telnetd). It's often the case that warnings, errors, or failures are logged in one or more of these logs. Therefore, they're a good place to look for information that might help identify root cause.

Some services allow for the configuration of verbose, debug, or trace-type logging so that more than the standard informational or error messages are logged. If the problem is reproducible, it's worth investigating the potential to use this type of diagnostic logging for the duration of the testing. However, it's not advisable to keep verbose logging on indefinitely, as doing so can cause disk and file system space issues.

To establish a better idea of what a process is doing, you can use truss to trace the system calls that a process makes. The truss command can either execute a specified command or attach to an existing process to produce a trace (assuming that you own the running process or have root privileges). In the case of the latter, you can stop the trace at any time by pressing Control-C.

Listing 2 shows an extract of a basic trace of the command, ssh testhost, along with a short extract. The -l switch prefixes each trace entry with the process ID, while the -d switch displays a timestamp relative to the start of the trace.

Listing 2. Basic system call trace of a command

Listing 3 shows a more verbose trace of a running process (process ID 976) to an output file along with a short extract. The following switches were used:

Table 1. Switches used for more verbose trace of a running process

Listing 3. More verbose trace of a running process to a file

You can use the system call trace to identify errors that may be the potential root cause of your problem. Look for calls marked with #Err, indicating a non-zero return code, which you can look up in /usr/include/sys/errno.h. You can also use it to identify potential performance delays by looking for long deltas between calls when using the -D switch. For example, the elapsed time between the last two events in the sample trace output in Listing 3 finished was 2.0001 seconds.

Houston, we have a problem!

Now that you have a useful toolkit of network diagnostic aids, it's time to look at a logical, step-by-step approach to troubleshooting common problems. The following section lists a number of common AIX network-related issues and provides a guide for diagnosis and what to look for in each one.

Host unknown

If a host name being used on a command or by an application isn't recognized, check the search order in which names are resolved by looking at the hosts record in /etc/irs.conf and /etc/netsvc.conf. For a host to be referenced by name, it has to be resolved through name resolution.

If local is specified in the hosts record, look for the host name in the /etc/hosts file. If you look at the example in Listing 4, you can see a simple grep of the host testhost from this file returning a successful match. Your host name must appear in any of the fields after the first field (the IP address). In the example shown, the server is also known by two aliases: testhost.testdomain.com and aixserver. This means that you can refer to this particular host by any of those three names when it comes to using commands that require a host name argument.

Listing 4. Looking for a host in /etc/hosts

If bind or dns is specified in the hosts record, use nslookup to ensure that the host name resolves through DNS. If you look at the example in Listing 5, you can see that resolution has been successful and the DNS server testdns.testdomain.com (shown with its IP address) has returned a known IP address for the host testhost of 10.217.1.206.

Listing 5. Resolving a host name via DNS

Any additional name resolution services specified in your configuration files are outside the scope of this document and won't be discussed here.

Unresponsive host

If a host is known but you find that users are complaining that the host itself or an application running on it isn't responding, use ping and look for 0% packet loss (see Listing 6). Anything else means that there could be a problem with the target host or the network.

Listing 6. Pinging a responsive host

Also, look for long response times in the time= field or a spike in the values reported. Both of these can indicate poor network or host performance, which may be causing the application that your users are complaining about to time out.

Ensure that there's a route out to the target host by using route get (see Listing 7), verifying with the network administrator that this is the correct gateway to use.

Listing 7. Getting routing table information for a host

Use ifconfig (see Listing 8) to make sure that the interface reported is configured to AIX and showing as UP and RUNNING.

Listing 8. Displaying network interface status

Use ping to check that the gateway reported by ifconfig is contactable from your host. If it isn't, there may be a problem with the physical connection from the network adapter on your host to the gateway (for example, a faulty switch port, cable, network card).

Use traceroute (see Listing 9) to trace the full route to and from the target host. A route to a host doesn't necessarily mean a route back, so check with the network administrator to ensure that both exist and the application traffic being sent or received isn't being blocked by any firewalls. If any of the hops on the route returns no data (marked by asterisks [*]), it could indicate a problem with the routing. Although, it might be that the packets that traceroute uses to trace the route may also be blocked by firewalls. The output from this command should help the network administrator determine whether there's a real routing issue.

Listing 9. Tracing a successful route to a host

Unresponsive TCP port

If a host is known and responding to ping but a particular TCP port used by an application or remote service doesn't appear to be, use telnet to try and make a connection to the specific port on the target host using the example shown in Listing 10, which attempts to connect to port 25 on host testhost.

Listing 10. Testing port 25 (SMTP) on a host (successful)

Common ports are listed in /etc/services. A successful connection should result in the message Escape character is '^]' and optionally a message from the remote service, such as the mail server, shown in Listing 10. If no such messages are received and the connection times out or is refused, then check with the network administrator that there are no firewalls en route blocking the type of traffic being sent. Also, check with the systems administrator of the target host that the application server or remote service is running and listening on the specified port and that firewalls running on the host are not blocking traffic.

Not connecting to a responsive TCP port

If a host is known, responding to ping, and responding on a particular TCP port to other hosts but not yours, use telnet to try and make a specific connection to the specific port following the logic shown in Unresponsive TCP port.

Use netstat to look for connections to the host and their state using the second example shown in Listing 11, which looks for all connections to a particular IP address and port (port 22 at 10.217.1.206). Unless the connection state is shown as ESTABLISHED, the connection is either still being made or has been terminated. For example, a status of SYN_SENT indicates that a three-way handshake has been initiated by your host, but as yet no acknowledgement has been received from the target host. This could mean that there's a route to the target but no route back for this type of traffic. In this situation, ask the network administrator whether any firewalls on the route back are blocking this type of traffic.

Listing 11. Displaying the status of connections to hosts

Use tcpdump to display packets sent to and received from the host on the specified port using the example shown in Listing 12. If only packets sent by your host are shown, this is another indication that the problem is with traffic sent back by the target and therefore the route back.

Listing 12. Display packets destined for or sent by a specific host on a specific port

Long login times

If users are complaining that response times logging in to a particular host are slow, log in to the host and use dig to perform a reverse lookup of the IP address of the user's computer using the example shown in Listing 13.

Listing 13. Reverse lookup of an IP address in DNS

During login, a host may perform a reverse lookup on the IP address of the source address on the packets it receives. Depending on the configuration of that host, it can take some time for the lookup to fail and the login to continue. This process appears to the user as a long login delay.

Look in the ANSWER SECTION of the output that dig returns—specifically, the PTR record, which will be a pointer to the host name. If none is returned, this could explain the delay.

MAC address query

If an application requires a MAC address—for example, one where an ACL is based on one, a firewall has rules based on one, or a configuration requires one (for example, KickStart or JumpStart installation services on Linux® and Sun® Solaris®)—use entstat to find out the MAC address of a local interface using the example shown in Listing 14, looking for the Hardware Address (aka MAC address).

Listing 14. Displaying Ethernet statistics for a network adapter

Use arp to the find out the MAC address of a remote host (see Listing 15), assuming that the host in question is known to your host and therefore has an entry in the cache.

Listing 15. Displaying a host entry in the arp table

Use ping to force an entry for the remote host into the arp cache if one doesn't exist.

Are packets being sent?

If a host is not responding in some way (either completely or on a particular port) and you need to verify that your host is sending out packets, reproduce the problem with the local application or command. Then, use tcpdump to display packets sent to the host using the example shown in Listing 16.

Listing 16. Display packets destined for a specific host

If no packets are seen leaving your host, then either there's a problem with the sending application or with the interface or routing (which you can diagnose using the previous diagnostic steps in this section).

Are packets being received?

If a host is not responding in some way (either completely or on a particular port) and you need to verify that packets are not being received from that host, reproduce the problem with the local application or command. Then, use tcpdump to display packets received from the host using the example shown in Listing 17.

Listing 17. Display packets sent by a specific host

If you have verified that your host is sending packets (using Are packets being sent?) and no packets are being received, this means that the host is not responding, the service on the host is not responding, or there is no route back (either one does not exist or the traffic is blocked by a firewall along one that does).

Connection made but application or command fails

If users are complaining that an application or command appears to establish a successful connection but fails afterwards, see whether the command has a debug, trace, or verbose option, and rerun to see whether any additional output produced identifies potential root cause. For example, both ssh and scp have a verbose switch (-v) that can provide details of the protocol exchange between client and server as follows:

1. The connection is established to the remote host on TCP port 22.
2. Local private key files are identified.
3. Protocol versions are exchanged and agreed upon.
4. A remote host key is identified and matched to the entry in the local known_hosts file for that host.
5. Key authentication is tried for each private key type found locally.
6. When these fail, the user is finally prompted for a password as authentication.
7. The user successfully enters it, login is successful, and a shell prompt is presented.

The verbose option here can help identify whether any of those steps fails and the probable cause.

Use truss to trace the command or the process running the remote service (see Tracing a problem application or daemon).

Tracing a problem application or daemon

If the previous steps fail to uncover the exact root cause of the problem and you need to diagnose connectivity issues further, use truss to run a verbose system call trace of the command.

Also, use truss to run a verbose system call trace of the daemon that runs the remote service where you are attempting to make a connection. You may need to ask the systems administrator of the remote host for help if you don't have access or don't have access to the user running the process or the root user.

When a system call returns with an error, the non-zero return code is shown marked with Err# followed by the error number and an error code (for example, ENOSPC). Standard error codes can be found in /usr/include/sys/errno.h and can help indicate the cause of the error. For example, a system call returning with Err#2 ENOENT (No such file or directory) would indicate that the command is expecting to find a file or directory but can't and subsequently fails. A system call returning with Err#28 ENOSPC (No space left on device) would indicate that a disk or file system is full, potentially causing the daemon to fail to respond to service requests.

The verbose trace will display data from the parameters of system calls (-x all) and the contents of the buffer for both Read (-r all) and Write (-w all) calls. The contents of these buffers can sometimes identify root cause, as well.

You can also use the -D switch to display each system call with a time delta, representing the elapsed time in seconds since the last event. You should look for long time deltas, as these could indicate delays and lead to long response times and poor performance.

Conclusion

Network problem diagnosis does not need to be the obstacle that some systems administrators feel it is. This series has shown that armed with the correct knowledge, many problems you might normally take to a network administrator can be diagnosed, analyzed, and root cause identified, thus making the network administrator's job a lot easier when fixing it. In some cases, you'll even find that it's something you can fix yourself. Happy troubleshooting!