Part 1, Network overview - Monitoring the hardware

Martin Brown and Ken Milberg
Published on November 23, 2010

FacebookTwitterLinked InGoogle+E-mail this page

0

About this series

Part 1 of this three-part series on AIX networking provides a networking overview and discusses the tools that help you monitor your hardware. Part 2 covers tuning the Network File System (NFS) with monitoring utilities, such as nfsstat and nmon, and it also goes over how to tune with nfso. Part 3 shows you how to monitor network packets and how to use netstat for this purpose. You'll learn how to tune your network subsystem using the no utility. This series also expounds on various best practices of network I/O performance tuning.

The first thing that usually comes to mind when a system administrator hears that there might be some network contention issues is to run netstat. netstat, the equivalent of using vmstat or iostat for your memory reports, is a quick way to get an overview of how your network is configured. Unlike vmstat or iostat, the defaults usually do not give you as much information as you probably would like. You need to understand the correct usage of netstat and how best to utilize it when monitoring your system.

netstat is really not a monitoring tool in the sense of vmstat and iostat. You can use other tools more suitable (discussed later in the article) to help monitor your network subsystem. At the same time, you can't really start to monitor unless you have a thorough understanding of the various components related to network performance. These components include your network adapters, your switches and routers, and how you are using virtualization on your host logical partitions. If you determine you are indeed having a network bottleneck, fixing the problem might actually lay outside of your immediate host machine. There is little you can do if the network switch is improperly configured on the other end. Of course, you might be able to point the network team in the right direction. You should also spend time gathering overall information about your network. How are you going to be able to understand how to troubleshoot your network devices unless you really understand your network? In this article, you'll look at specific AIX network tracing tools, such as netpmon, and how they can help you isolate your bottlenecks.

Finally, no matter which subsystem you are looking to tune, you must think of systems tuning as an ongoing process. As stated before, the best time to start monitoring your systems is at the beginning, before you have any problems and users aren't screaming. You must have a baseline of network performance so that you know what the system looks like when it is behaving normally. Finally, when making changes, be careful to make changes only one at a time so that you can really assess the impact of your change.

Network I/O overview

This section provides an overview of the network as it relates to AIX 7 and covers the physical aspects of the network (device drives and adapters), the AIX networking stack, and how to make some changes to your adapter.

Understanding the network subsystem, as it relates to AIX, is not an easy undertaking. When examining the CPU and memory bottlenecks, there are far fewer areas that you need to examine from a hardware and software aspect. Disk I/O tuning is more complex, as there are many more issues that impact performance, particularly during the architectural and build-out of your systems. In this respect, tuning your network is probably most like tuning your disk I/O, which is actually not too surprising, as they both relate to I/O. Let's start.

Figure 1 illustrates the AIX Transmission Control Protocol/Internet Protocol (TCP/IP) layers.

Figure 1. The AIX TCP/IP layers

Figure 1 illustrates that there is more to network monitoring than running netstat and looking for collisions. From the application layer through the media layer, there are areas that need to be configured, monitored, and tuned. At this point, you should notice some similarities between this illustration and the Open Systems Interconnection Basic Reference Model (OSI Model). The OSI Model has seven layers (bottom to top):

• Physical
• Data-link
• Network
• Transport
• Session
• Presentation
• Application

Perhaps the most important concept to understand is that on the host machine each layer communicates with its corresponding layer on the remote machine. The actual application programs transmit data using either User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) transport layer protocols. They receive the data from whatever application you are using and divide them into packets. The packets themselves differ, depending on whether it is a UDP or TCP packet. Generally speaking, UDP is faster, while TCP is more secure. There are many tunable parameters to look at—you'll get to these parameters during subsequent phases of the series. You might want to start to familiarize yourself with the no command, which is the utility designed to make the majority of your network changes. From a hardware perspective, it is critical that you understand the components that need to be configured appropriately to optimize performance. Though you might work together with the network teams that manage your switches and routers, it is unlikely that you will be configuring them (unless you are a small shop or a one-person IT department). The most important component you will be working with is your network adapter. Gigabit Ethernet is now the standard network interface, although 10Gbit network cards and infrastructure is becoming more accessible.

Maximum Transfer Unit

Maximum Transfer Unit (MTU) is defined as the largest packet that can be sent over a network. The size depends on the type of network. For example, 16-bit token ring has a default MTU size of 17914, while Fiber Distributed Data Interface (FDDI) has a default size of 4352. Ethernet has a default size of 1500 (9000 with jumbo frames enabled). Larger packets require less packet transfers, which result in higher bandwidth utilization on your system. In particular, using jumbo frames allows for an entire 8KB NFS block to be exchanged in a single packet, which can significantly improve performance. An exception to this is if your application prefers smaller packets, and this includes web applications on the Internet, since most Internet connections do not support jumbo frames. If you are using a Gigabit Ethernet, you can use a jumbo frames option. To support the use of jumbo frames, it's important to note that your switch must also be configured, accordingly.

To change to jumbo frames, use this fastpath: # smit devices.

Then go to Communication>Ethernet>Adapter>Change/show characteristics of an Ethernet adapter. Try to change the transmit jumbo frames option from "No" to "Yes" (see Listing 1).

Listing 1. Characteristics of an Ethernet adapter screen

Where is the jumbo frames option? In this case, you cannot make the change. The reason for this is because you are only using the Virtual I/O Ethernet on this system—this topic is discussed in more detail later.

On a system where you have direct access to the network hardware, check the system using Listing 2.

Listing 2. Checking the system

You have now changed the field to support jumbo frames.

Media speed

Most modern network switches and environments can take advantage of auto-negotiation to provide the best speed, especially as full-duplex network switches have become the standard. However, you force a particular configuration if the auto-negotiation fails to configure a speed that you know your network switch is capable of supporting.

The lsattr command gives you the information that you need. The en prefix displays your driver parameters, while the ent prefix displays your hardware parameters. Let's display your hardware parameters (see Listing 3).

Listing 3. Displaying the hardware parameters

In this case, your interface is set as auto-negotiate.

You should also check your firmware levels to make sure they are up to date. We've seen many network problems fixed when updating to the latest levels of firmware. The lscfg command gives you the firmware information (see Listing 4).

Listing 4. Using the lscfg command for firmware information

See the Related topics section at the end of the article for a link to the most current release information for your adapter.

Though the series focuses on tuning in subsequent parts, you might want to start to familiarize yourself with the memory management facility of network subsystems. What you need to know at this point is that it relates to data structures called mbufs. These are used to store kernel data for incoming and outbound traffic. The buffer sizes themselves can range from 32 to 16384 bytes. They are created by making allocation requests to the Virtual Memory Manager (VMM). In an SMP box, each memory pool is split evenly for every processor. The monitoring section below shows you how to view mbufs. An important concept to note is that processors cannot borrow from the memory pool outside of its own processor.

Two other concepts you should be familiar with are virtual Ethernet and shared Ethernet.

• Virtual Ethernet: Virtual Ethernet allows for inter-partition- and IP-based communications between logical partitions on the same frame. This is done by the use of a virtual I/O switch. The Ethernet adapters themselves are created and configured using the HMC. If you recall, you tried to change an adapter earlier that was configured with virtual Ethernet.
• Shared Ethernet: Shared Ethernet allows for the use of Virtual I/O servers (VIOs), where several host machines can actually share one physical network adapter. Typically, this is used in environments that do not require substantial network bandwidth.

While the scope of this series is not on virtualization, you should understand that if you are using virtualization, there might be other reasons for your bottleneck outside of what you are doing on your host machine. While virtualization is a wonderful thing, be careful not to share too many adapters from your VIO server, or you might pay a large network I/O penalty. Keep in mind as well that with workload partitions (WPAR), the network configuration of the host environment is used and shared by each partition; you cannot configure WPAR network performance individually. Using appropriate monitoring tools should inform you if you have a problem.

In addition, application performance can be affected fundamental services, such as domain name service (DNS) used to resolve hostnames and Internet addresses.

Monitoring

This section provides an overview of general network monitoring commands and specific AIX 7 tools available to you. Some of the tools allow you to troubleshoot a performance problem quickly while others capture data for historical trending and analysis.

Let's get back to the old standby, netstat, which displays overall network statistics. Probably one of the most common commands you type in is netstat -in (see Listing 5).

Listing 5. Using netstat with the -in option

Here is what it means:

• Name: Interface name.
• MTU: Interface Maximum Transfer Unit size.
• Network: The actual network address that the interface connects to.
• Address: Mac and IP address.
• Ipkts: The total amount of packets received by the interface.
• Ierrs: The amount of errors reported back from the interface.
• Opkts: The amount of packets transmitted from the interface.
• Oerrs: The amount of error packets transmitted from the interface.
• Coll: The amount of collisions on the adapter. If you are using Ethernet, you won't see anything here.

Another handy netstat flag is the -m option. This flag allows you to view the Kernel malloc statistics; the mbuf memory requests, including the size of the buffers, the amount in use and the failures by CPU (see Listing 6).

Listing 6. netstat with -m option

The -m option is particularly useful because it shows the network performance statistics in relation to individual CPUs. When monitoring and managing the performance with LPAR and WPAR environments, the ability to correlate your CPU resources and the network resources can give you valuable information about the correct allocation and distribution of network resources.

For Ethernet, you can use the entstat command to display device-driver statistics. This provides a plethora of information (see Listing 7).

Listing 7. Using the entstat command to display device driver statistics

Collisions are largely a thing of the past with modern network switches, but look for transmit errors and make sure they are not increasing too fast. You need to learn to troubleshoot collision and error problems before you even begin to think about tuning. Alternatively, you can use netstat -v, which provides similar information.

Let's look at netpmon. netpmon provides information on CPU usage as it relates to the network, and it also includes data about the network device driver I/O, Internet socket calls, and other various statistics. Similar to its other trace brethren, tprof and filemon, it starts a trace and runs in the background until you stop it with the trcstop command. We like netpmon because it really gives you a detailed overview of network activity and also captures data for trending and analysis (though it is not as useful as nmon for this purpose). Here you'll use a trace buffer size of two million bytes (see Listing 8).

Listing 8. netpmom with -T option

Now you'll stop it (see Listing 9).

Listing 9. Stopping netpmom

Let's look at the data. Here is just a small sampling of the output (see Listing 10).

Listing 10. Sample output

As you can see, there is little overall network I/O activity going on during this time. The top section is most important, as it really helps you get an understanding of what processes are consuming network I/O time, and you can use the more detailed per-process output to obtain more specific information. lsattr (used earlier to view the hardware parameters) is another command you will be using frequently to display statistics on your interfaces. The attributes that you see here are configured using either the chdev or no commands. Let's display your driver parameters (see Listing 11).

Listing 11. Displaying the driver parameters using lsattr

Finally, let's look at Figure 2.

Figure 2. nmon statistics

If you've been following the other series on AIX 7 (see Related topics), you know we love nmon and you should also, once you start using it. With nmon (type in n after startup), you have a quick snapshot of everything going on in your network, including adapter details, MTU, error counters and collisions, and megabit rating.

Further, you also have the ability to capture data with nmon. Using the nmon analyzer, you can print out graphical reports directly from Microsoft® Excel spreadsheets. See Related topics for a link to an IBM Wiki for the nmon manual or for downloads.

Summary

This article covered the relative importance of the network I/O subsystem, and defined the AIX 7 network I/O layers and how it relates to the OSI Model. You learned some best practices for network configuration, changed your Ethernet settings to support jumbo frame, and viewed interface hardware and driver data. You even examined the monitoring tools available to you and captured data using netpmon and nmon. In the next part of the series, you'll tune NFS, find out more about monitoring utilities, such as nfsstat and nmon, and discover how to tune with nfso.