Ethernet tunning for VIOS and AIX for 10 Gib/sec environtement.

I had the pleaseure to setup from scratch an IBM pureflex p270 system, including chassis, ethernet switches, fiber channel switches, FSM, dual VIOS and LPARs. Although, all connections inside pureflex node is at 10 Gbits/sec, this chassis is connected to a 1Gb cisco backbone. Once all components were on production, I found there was an additional setup on ethernets devices to approach maximum speed inside the p270 node. All internal ports of ethernet switches EN4093r were configured to 10Gbits/sec.

Here you can find what tunning I did on VIOS and LPARs.

Let’s tart with VIO servers.

Both vios have two physical interface 10Gb/s. See:

root@vios # lsdev|grep ent|grep 10
ent0        Available 00-00       10GbE 4-port Mezzanine Adapter (a2191007df1033e7)
ent1        Available 00-01       10GbE 4-port Mezzanine Adapter (a2191007df1033e7)

Also both vio servers has on etherchannel adapter ent2 configured on LACP (aggregation).

root@vios # lsdev|grep ent|grep EtherChannel
ent2        Available             EtherChannel / IEEE 802.3ad Link Aggregation

Here’s etherChannel configuration. It was created with two physical adapters ent0 and ent1 and 8023ad mode:

root@vios # lsattr -EHl ent2
attribute       value          description                                     user_settable

adapter_names   ent0,ent1      EtherChannel Adapters                           True
alt_addr        0x000000000000 Alternate EtherChannel Address                  True
auto_recovery   yes            Enable automatic recovery after failover        True
backup_adapter  NONE           Adapter used when whole channel fails           True
hash_mode       src_dst_port   Determines how outgoing adapter is chosen       True
interval        long           Determines interval value for IEEE 802.3ad mode True
mode            8023ad         EtherChannel mode of operation                  True
netaddr         0              Address to ping                                 True
noloss_failover yes            Enable lossless failover after ping failure     True
num_retries     3              Times to retry ping before failing              True
retry_time      1              Wait time (in seconds) between pings            True
use_alt_addr    no             Enable Alternate EtherChannel Address           True
use_jumbo_frame no             Enable Gigabit Ethernet Jumbo Frames            True

SEA adapter ent9 was created with Etherchannel en2 and 4 Virtual I/O Ethernet Adapters (ent5,ent6,ent7 and ent8) using ha_mode on sharing, see:

root@vios # lsdev|grep ent|grep Shar
ent9        Available             Shared Ethernet Adapter

root@vios # lsattr -EHl ent9
attribute     value               description                                                        user_settable

accounting    enabled             Enable per-client accounting of network statistics                 True
adapter_reset yes                 Reset real adapter on HA takeover                                  True
ctl_chan      ent3                Control Channel adapter for SEA failover                           True
gvrp          no                  Enable GARP VLAN Registration Protocol (GVRP)                      True
ha_mode       sharing             High Availability Mode                                             True
hash_algo     0                   Hash algorithm used to select a SEA thread                         True
jumbo_frames  no                  Enable Gigabit Ethernet Jumbo Frames                               True
large_receive yes                 Enable receive TCP segment aggregation                             True
largesend     1                   Enable Hardware Transmit TCP Resegmentation                        True
lldpsvc       no                  Enable IEEE 802.1qbg services                                      True
netaddr       0                   Address to ping                                                    True
nthreads      7                   Number of SEA threads in Thread mode                               True
pvid          4000                PVID to use for the SEA device                                     True
pvid_adapter  ent5                Default virtual adapter to use for non-VLAN-tagged packets         True
qos_mode      disabled            N/A                                                                True
queue_size    8192                Queue size for a SEA thread                                        True
real_adapter  ent2                Physical adapter associated with the SEA                           True
send_RARP     yes                 Transmit Reverse ARP after HA takeover                             True
thread        1                   Thread mode enabled (1) or disabled (0)                            True
virt_adapters ent5,ent6,ent7,ent8 List of virtual adapters associated with the SEA (comma separated) True

Virtual adapter ent4 was created to management for vios. VIOS’s IP is setup here.

root@vios # lsdev|grep ent4
ent4        Available             Virtual I/O Ethernet Adapter (l-lan)

Virtual adapter ent3 is the control channel between both vios. See attribute ctl_chan on SEA output above.

root@vios # lsdev|grep ent3
ent3        Available             Virtual I/O Ethernet Adapter (l-lan)

Tunning on VIO server, is done on physical adapters, SEA adapter and on trunk adapters.

For SEA adapter ent9 run this command: (It’s recommended to add these parameters when you create the SEA adapter)

chdev -l ent9 -a large_receive=yes -a largesend=1

Although it’s mnot always necessary, you can increase Min Buffers. (reboot required) ( Please check this with command entstat -d entXX)
For all trunk adapters ent5,ent6,ent7 and ent8 increase Min Buffers

root@vios # chdev -l entXX -a max_buf_tiny=3072 -a min_buf_tiny=2048 -P
root@vios # chdev -l entXX -a max_buf_small=4096 -a min_buf_small=4096 -P
root@vios # chdev -l entXX -a max_buf_medium=2048 -a min_buf_medium=2048 -P
root@vios # chdev -l entXX -a max_buf_large=256 -a min_buf_large=256 -P
root@vios # chdev -l entXX -a max_buf_huge=128 -a min_buf_huge=128 -P

Fpr physical adapters ent0 and ent1, enable largesend, largereceive and flow control (reboot required)

chdev -l ent0 -a flow_control=yes -a large_receive=yes -a large_send=yes -P
chdev -l ent1 -a flow_control=yes -a large_receive=yes -a large_send=yes -P

For LPARs enable largesend, RFC 1323 Window Scaling, Socket Buffer Space for Receiving/Sending on all Virtual Ethernet adapters

chdev -l en0 -a mtu_bypass=on -a rfc1323=1 -a tcp_recvspace=262144 -a tcp_sendspace=262144

Check if all this was enabled with ifconfig -a command.

root@aix # ifconfig -a
en4: flags=1e084863,5c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
        inet 192.168.1.1 netmask 0xfffffff0 broadcast 192.168.1.250
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1


IMPORTANT NOTE! If your external switch / firewall has ethernet speed below 1 Gbits/sec, you have to disable RFC 1323 Window Scaling 
                on virtual adapters on all AIX (for example,  ifconfig en0 rfc1323 0)
That's it!
Thanks if it was helpful. Thanks.