Network problem determination: AIX tools for a system administrator

Part 1, The basics of network troubleshooting

As an IBM AIX® systems administrator, it's inevitable that at some point in your role you will encounter a problem that's linked to or directly caused by an issue within the LAN or WAN. In such instances, its good practice to make an initial diagnosis of the problem before engaging the help of a suitable network administrator to help identify root cause, or at least give the administrator a general direction in which to start his or her investigation.

Once engaged, you may be required to assist in the analysis, so it's essential that you come armed with a relevant diagnostic toolkit. This article provides you with a set of commands available on AIX, many of which are also available on other flavors of UNIX®, which can help you to troubleshoot TCP/IP network-related issues.

For the purposes of this article, the target host system used in all sample commands and output is called testhost.

Is anybody there?

The first step in diagnosing any network-related issue is to verify whether the target host is running. You can use ping to test whether a host is reachable across a network (see Listing 1). This command sends an Internet Control Message Protocol (ICMP) echo request packet to the host and waits for an echo reply.

A successful ping means that:

• Your host has an active network adapter that can be used to send out the request.
• The target host is running and has an active network adapter configured with the IP address that you used.
• Name resolution is working for that host if a name was used rather than an IP address.
• There is a route from your host to the target and back.
• No firewalls on the route between hosts or running on either host are blocking ICMP traffic.

The output from a successful ping can also be useful in helping to determine network latency, as it reports on the time taken to receive the echo reply. Long response times are likely to mean poor performance for any applications that exchange data with the target host.

Listing 1. Pinging a responsive host

If no echo reply is received, then one or more of the conditions described above hasn't been met and the ping fails (see Listing 2). A ping fails when the number of packets received is less than the number sent and packet loss is greater than 0 percent.

Listing 2. Pinging an unresponsive host

If the ping was unsuccessful, you can check whether the adapter used to send the request is up by using ifconfig.

You can use the ifconfig command to display the status of an individual adapter (for example, en1 shown in Listing 3) or all adapters using the -a switch, also shown in Listing 3. You should ensure that the adapter used to send packets out to your host is showing as UP and RUNNING. If it's not, then you need to investigate further.

Listing 3. Displaying network adapter status

You can display Ethernet statistics on an adapter with entstat (see Listing 4). The example shown uses the -d switch to display all statistics, including device-specific statistics, for the adapter en2. This command can also be useful for telling you the link status (up or down) and media speed (for example, 100Mbps Full Duplex). The media speed is useful if you need to verify the setting based on the link partner and network that the adapter is connected to, as a speed or duplex mismatch can cause problems.

Listing 4. Displaying Ethernet statistics for a network adapter

If the adapter is up, you can establish whether the route from your host to the target is correct by using route get (see Listing 5). If there's no route at all, then ping will inform you, but if there is, you will need to establish what it is to verify with the network administrator that it's correct. Based on the information in the routing table that your host uses, route get will tell you the gateway the packets will be routed to when leaving your host on the way to the target.

Listing 5. Getting routing table information for a host

If the route is correct, then you can use traceroute to determine the exact route that packets will take across the network to the target host. The output of a successful traceroute (see Listing 6) shows each router the packets travel through to reach the target host along with the minimum, average, and maximum response time taken to get to that router.

Listing 6. Tracing a successful route to a host

An unsuccessful traceroute (see Listing 7) has asterisks (*) in the time fields, as they cannot be determined because the probe to the next router timed out. The example also shows the use of the -n switch, which prints numeric host addresses, thereby avoiding name lookup and resolution and speeding up the trace.

Listing 7. Tracing an unsuccessful route to a host

Services running at the application layer of a TCP/IP network listen on one or more ports that are used to exchange data between clients and the host server as managed by the transport layer. If a valid route exists to the host, and it's responding to pings but the application service is failing to respond, then you can check connectivity to the relevant ports using telnet.

The telnet command, used in its basic form, establishes a terminal connection to a host. However, you can also use it to establish a connection to a specific port on the host (the default being 23, the telnet service). For a list of standard ports, look in /etc/services.

If the connection is successful, a message indicating the telnet escape sequence is shown (see Listing 8). You need to enter this key sequence (typically, Control-]) to escape back to a telnet> prompt and enter quit to return to a shell prompt.

Listing 8. Testing port 80 (HTTP) on a host (successful)

Depending on the type of connection you're making, the remote service you're connecting to may generate a message similar to Listing 9.

Listing 9. Testing port 25 (SMTP) on a host (successful)

If the connection fails, then either a connection timeout or a connection refused message will be displayed (see Listing 10). This message can mean that the service on the target host isn't running (and therefore nothing is listening on the port), or that a firewall running on the host (or somewhere en route) is blocking connections to the port.

Listing 10. Testing port 515 (remote printing) on a host (unsuccessful)

Do I know you?

When using a host name in an application or any of the diagnostic commands covered in this article, it's imperative that the host name can be resolved to an IP address. An IP address is what the Internet layer of a TCP/IP network uses when handling data packets.

A host name must resolve through one of the name-resolution services specified in /etc/irs.conf and /etc/netsvc.conf. The hosts record determines the order the name resolution is performed. Only local and BIND/DNS resolution is covered here; the remaining options are outside the scope of this article.

When local is specified, the /etc/hosts file is used to resolve host names. So, check to see whether there's an entry for the target host (see Listing 11).

Listing 11. Looking for a host in /etc/hosts

If you specify bind or dns, then DNS is used to resolve host names, and you can use nslookup to check whether the host name resolves (see Listing 12).

Listing 12. Resolving a host name via DNS

A more powerful DNS interrogation tool is dig. This command has a much richer set of options and arguments than nslookup. The latter has an interactive mode that provides the additional functionality. So, for more complex queries—particularly where the output will be parsed by a script—dig is preferred (see Listing 13).

Listing 13. Reverse lookup of an IP address in DNS

A host uses the Address Resolution Protocol (ARP) table or arp cache to keep track of the media access control (MAC) address of other network devices alongside their IP addresses. The link layer of a TCP/IP network uses a device's MAC address, so the ARP table is used to translate a MAC address to an IP address and back. If your host has communicated successfully with another host, it's likely that there is an entry in the ARP table. You can use arp to display the entry for a particular host if one exists (see Listing 14).

Listing 14. Displaying a host entry in the ARP table

You can also display the entire table using the -a switch (see Listing 15). The -n switch specifies that host name-to-IP address resolution shouldn't be performed.

Listing 15. Displaying the contents of the ARP table

Can you hear me?

To establish a connection, TCP uses a three-way handshake. A client initiates a connection to a host (and a specific port) by sending a SYN synchronize packet. After successfully receiving it, the host responds with a SYN-ACK acknowledgement. If the client successfully receives this acknowledgement, the client completes the handshake with an ACK acknowledgement. All of this assumes that the host server is listening on the specified port, that a route exists from the client to the host and back again, and that no firewalls are blocking this kind of traffic.

You can use netstat to display existing connections from your host to other hosts and the current state of each. Using the command with the -a switch (show the state of all sockets) and the -n switch (show addresses numerically, avoiding lookup), you can pipe the output to a suitable grep to look for connections in a particular state (for example, ESTABLISHED for post-handshake, active connections) or connections to a particular host or port.

Listing 16 shows all connections and their state to a particular IP address (two connections to 10.217.1.206, both fully established), to a particular port at a particular IP address (to 10.217.1.206 at port 22), and all fully established connections to any host respectively.

Listing 16. Displaying the status of connections to hosts

You can monitor outgoing data sent from a particular adapter using tcpdump, which displays the content of each packet as it is sent. The command takes various options to allow you to display more or less of the packet either in descriptive or raw form and allows a number of Boolean expressions to filter the type of data you want to see. For example, monitoring packets on adapter en2, you can show only data being sent to a specific host (see Listing 17).

Listing 17. Display packets destined for a specific host

You can show only packets coming from a specific host (see Listing 18).

Listing 18. Display packets sent by a specific host

You can show only packets sent to or coming from a specific port (see Listing 19).

Listing 19. Display packets destined for or sent by a specific host on a specific port

You can stop the trace by pressing Control-C. The tcpdump command is much more feature rich than the simple examples shown here, so I recommend that you familiarize yourself with its man pages.

As you can see from the output in these three examples, traffic is shown with:

• A timestamp
• Source Host.Source Port
• Destination Host.Destination Port
• Packet flags
• Other packet information

You can use the command to establish whether traffic is leaving your host destined for the target host and whether traffic is making its way back. If no inbound traffic appears, it may be that the host isn't responding or there's no valid route from your host to the target or vice versa. If a particular service (TCP port) isn't responding or a firewall is blocking packets of the type you are sending, you will typically see an R in the packet flags field, indicating that the connection has been reset. For more information on the exact layout and format of a TCP packet, refer to RFC 793: Transmission Control Protocol.

Depending on the nature of the problem, it is sometimes good practice to run a tcpdump for a period of time while capturing packet information to a file using the -w switch. Once you feel you have captured enough data, press Control-C to stop the trace. At this point, you can process the file using the -r option to read the packet data captured. You can then use the vast array of switches, options, and Boolean arguments to analyze the data. Listing 20 shows an example of this process.

Listing 20. Capture packet data to a file and analyze it

Conclusion

This article covered some of the AIX tools you can use to test connectivity to a host, extract useful network-related information about a host, and analyze data sent to and from a host. In the next article, you'll get under the covers to see what is really going on when your host has problems communicating with another. The article will conclude with a step-by-step guide to logical problem diagnosis when encountering network-related issues.