Optimizing AIX 7 network performance

Part 3, Monitoring your network packets and tuning the network

Martin Brown and Ken Milberg
Published on January 11, 2011

FacebookTwitterLinked InGoogle+E-mail this page

1

While running commands (such as netstat) can provide useful information, sometimes you need to drill down more to the packet level. This is where tracing tools come in handy, such as iptrace, ipreport, and tcpdump. You'll also learn how you can tune a network using tools such as no. The no command is similar to vmo and ioo, but no is the network flavor. This article focuses on tcp workload tuning, udp workload tuning, and some other noteworthy parameters with the no utility. The article also addresses ARP cache tuning and how to monitor and tune ARP statistics. You will also look at name resolution and how you can easily increase performance by making small adjustments to resolve hostnames.

Monitoring network packets

In this section, you'll see an overview of tools available to help you monitor your network packets. These tools allow you to troubleshoot a performance problem quickly and capture data for historical trending and analysis.

Part 1 of this series addressed some of the very basic flags, such as -in, that you typically use with netstat. Using netstat, you can also monitor more detailed information about the packets themselves. For example, the -D option shows the overall number of packets received, transmitted, and dropped in your communication subsystem. The results are sorted by device, driver, and protocol (see Listing 1).

Listing 1. netstat with the -D option

Another useful flag is the -s, which shows detailed statistics for all protocols used, including packets sent, received and dropped. If you only want to view tcp, you can also use the -p flag (see Listing 2).

Listing 2. netstat with the -p option

There are actually so many different ways of using netstat that the best place to start is by looking at the man page and go from there. Don't be afraid to run these commands, because they won't eat up disk space or affect performance. The tracing tools that are provided within AIX 7 are used to record detailed information about the packets. Use them with more caution. These tools are extremely helpful when trying to determine the root cause of network performance problems.

First, look at iptrace and ipreport. The iptrace command records all the packets received from the network interfaces. The ipreport command formats the data that is generated from iptrace into a readable trace report. Further, you can also use ipfilter to sort the output file created from ipreport. Try starting the trace and keep it going for one minute (see Listing 3).

Listing 3. Starting the trace

When you are done with the trace, you need to kill the process (see Listing 4).

Listing 4. Killing the process

Now, examine the output (see Listing 5):

Listing 5. Examining the output

Listing 5 shows the captured information about each packet, including packet size and IP address information. As you can imagine, the trace file can get very large quickly. The example file grew to 40MB in less than one minute! Be very careful when running these traces, because you will run out of disk space really fast if you don't have the disk bandwidth for these files.

You can also start the trace using the System Resource Controller (SRC). See Listing 6.

Listing 6. Starting the trace using SRC

What about tcpdump? tcpdump prints out headers of the packets, which are captured for each NIC. One important difference with tcpdump is that, unlike iptrace, it can look at only one network interface at a time. And, because iptrace examines the entire packet from the kernel space, the results can offer lots of dropped packets. With tcpdump, you can also limit the amount of data to be traced. Also, you do not need to use an ipreport type of command to format binary data, because tcpdump does the trace and the output. See Listing 7 for an example.

Listing 7. Using tcpdump

tcpdump continues to capture packets until you hit Ctrl+C. If any packets were dropped due to a lack of buffer space, it reports that, too.

Listing 8 shows what you see when you end the example trace and view the file.

Listing 8. End of the trace

The main benefit of tcpdump is that you can specify filters so that you can select only particular protocols, sources, destinations, ports and other combinations. This is useful if you want diagnose or determine the NFS traffic, for example, between two hosts.

Tuning network performance

In this section, you will learn how to use the no command to tune your network subsystem. You'll also look at other areas that can impact network performance, and you'll learn about recommended tuning methodologies where appropriate.

The most important command for tuning network parameters is the no command. First, take a look at all the parameters, using the -a flag (see Listing 9). Be warned, though, that the list is quite extensive.

Listing 9. Viewing the parameters

Alternatively, you can also use the -L flag. It provides much more detailed information, including current, default, boot, and range of settings. This can make it much easier to determine whether a given value is working at its optimum value and whether a specific item could be improved beyond it's current value. Listing 10 shows you only the first few lines.

Listing 10. Using the -L flag

There are many parameters here. The thewall defines the upper limit for network kernel buffers. Today, its size is defined at installation time, depending on the amount of RAM and the kernel type. For example, if you are running AIX 5.3 on a 64-bit kernel, the parameter is set at half the size of real memory.

Listing 11. Setting the size of the thewall parameter

Part 1 discussed mbufs, but it's worth another mention here, because it relates to thewall. Remember that mbufs are used to store data in the kernel for both incoming and outgoing traffic. This is why determining the right amount of mbufs is extremely important. The value of the maxmbuf tunable limits the amount of memory that the communication systems use. If the value is 0, thewall tunable is used, and it cannot be modified from its default. Changing this tunable is a way to lower the thewall limit. As the default, if maxmbuf is 0, this value is used regardless of what thewall uses. netstat -m is used to detect shortages of failures of network memory requests (see Listing 12).

Listing 12. netstat with the -m option

In the example, there are no shortages (failures).

Although there are many parameters you can change with the no utility, most of them are better left alone. The most important parameters are ones that refer to TCP streaming workload tuning.

• tcp_sendspace—This controls how much buffer space in the kernel is used to buffer application data. You really want to bump this up from the default, because if its limit is reached, the sending application suspends data transfer until TCP sends the data to the buffer.
• tcp_receivespace—In addition to controlling the amount of buffer space to be consumed by receive buffers, AIX 7 also uses this value to determine the size to make its transmit window.
• udp_sendspace—With UDP, you can set this to no more than 65536, because IP has an upper limit of 65536 bytes per packet.
• udp_resvspace—This value should be greater than udp_sendpsace, because it needs to handle as many simultaneous UDP packets per socket as it can. This parameter can easily be set to 10 times the value of udp_sendspace.

Now, let's make some changes. First, increase the size of udp_sendspace (see Listing 13).

Listing 13. Increasing the size of udp_sendspace

Next, change udp_recsvspace to the recommended configuration of 10 times udp_sendspace). See Listing 14.

Listing 14. Changing udp_recsvspace

Note that the -p flag keeps the entries, even after a reboot. It appends the /etc/tunables/nextboot stanza file, as shown in Listing 15.

Listing 15. Looking at the /etc/tunables/nextbook file

Regarding the tcp parameters for higher speed adapters, there is no problem setting tcp_sendspace to twice the value of tcp_recvspace. For example, you can use the settings in Listing 16.

Listing 16. Examples settings for tcp_sendspace

Other important workload parameters include rfc1323 and sb_max.

The rfc1323 tunable enables the TCP window scaling option, which allows TCP to use a larger window size. Turning it on enables the best TCP performance. The sb_max tunable sets an upper limit on the number of socket buffers queued to an individual socket, which controls the amount of buffer space consumed by buffers (queued to either a sender or received socket). This amount should usually be less than the wall and approximately 4 times the size of the largest value of the tcp or udp send and receive settings. For example, if your udp_recvspace is 655360, you can't go wrong if you double it to 1310720.

Now look at tcp_nodelayack. This tunable prompts TCP to send an immediate acknowledgement, rather than a delayed acknowledgement. While this can add more overhead in some environments, it can greatly improve network performance in others. If you change this parameter, but it does not improve performance, you can quickly change it back.

Next look at ipqmalen. This tunable controls the length of the IP input queue. If you see an overflow counter (through the use of netstat -s), setting a maximum length of this queue can help fix the overflow.

What about ARP? When many clients are connected to the system, you might want to tune the ARP cache. You can look at the statistics using netstat (see Listing 17).

Listing 17. Using netstat with -p arp

If you see a high purge count, increase the size of the ARP table. For the example table, this isn't needed.

Here are the no parameters that relate to ARP (see Listing 18).

Listing 18. Using the no parameters

You can view the specific interface settings using either ifconfig or lsattr. In the example in Listing 19, look at the settings using ifconfig (look at the last line which references some of the tunables mentioned earlier).

Listing 19. Viewing specific interface settings using ifconfig

You can change these options (by interface) by using SMIT, chdev, or ifconfig. Note that ifconfig will not update the Object Data Manager (ODM). Therefore, on a reboot, it will revert to the previous value. Because of that, you should use SMIT: the fastpath of smit tcpip>further configuration>Network interfaces>Change/Show characteristics of an interface (see Figure 1).

Figure 1. Using SMIT to change interface settings

You might wonder why the no parameters don't apply to some interfaces. Name resolution is another area that can impact performance. If you know how you want to resolve (using DNS or the hosts file), make sure name resolution is set up correctly in the /etc/netsvc.conf file. Look at a piece of the file in Listing 20.

Listing 20. Piece of /etc/netsvc.conf file

If you're using DNS, take out the local if you are not using a host's file at all, or you can leave it in if you are using it as a backup to DNS (but make it the second entry). Alternatively, take out the bind if you're not using DNS at all, because it will only slow down your performance by first attempting (if it is the first entry in the record) to resolve using a Name Server that doesn't exist.

Summary

This article discussed how to monitor network packets on the network. You used netstat and drilled down to the packet level using tracing tools, such as iptrace and tcpdump. Further, you learned how to tune your network using the no utility. Using this utility, you explored tcp and udp workload tuning while also learning some other noteworthy parameters. You made tuning changes and read about how you might want to tune certain settings. You also examined ARP cache tuning and saw how you could monitor and tune ARP statistics. You looked at ISNO and learned how you could tune specific no tunables by interface. You also looked at name resolution and how you could easily increase performance by making small adjustments in how to resolve hostnames.

Related topics

• AIX Wiki: Get the nmon manual or downloads here.
• Improving database performance with AIX concurrent I/O: Read this white paper for more information on how to improve database performance.
• Power Architecture: High-Performance Architecture with a History: Read this white paper.
• nmon analyser -- A free tool to produce AIX performance reports (Steven Atkins, developerWorks, April 2006): You can download nmon analyser from here.
• Operating System and Device Management from IBM provides users and system administrators with complete information that can affect your selection of options when performing such tasks as backing up and restoring the system, managing physical and logical storage, and sizing appropriate paging space.
• The AIX 7.1 Information Center is your source for technical information about the AIX operating system.
• The IBM AIX Version 6.1 Differences Guide can be a useful resource for understanding changes in AIX 6.1.
• AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
• IBM trial software: Build your next development project with software for download directly from developerWorks.
• Future Tech: Visit Future Tech's site to learn more about their latest offerings.