Ethernet tunning for VIOS and AIX for 10 Gib/sec environtement.

I had the pleaseure to setup from scratch an IBM pureflex p270 system, including chassis, ethernet switches, fiber channel switches, FSM, dual VIOS and LPARs. Although, all connections inside pureflex node is at 10 Gbits/sec, this chassis is connected to a 1Gb cisco backbone. Once all components were on production, I found there was an additional setup on ethernets devices to approach maximum speed inside the p270 node. All internal ports of ethernet switches EN4093r were configured to 10Gbits/sec.

Here you can find what tunning I did on VIOS and LPARs.

Let’s tart with VIO servers.

Both vios have two physical interface 10Gb/s. See:

root@vios # lsdev|grep ent|grep 10
ent0        Available 00-00       10GbE 4-port Mezzanine Adapter (a2191007df1033e7)
ent1        Available 00-01       10GbE 4-port Mezzanine Adapter (a2191007df1033e7)

Also both vio servers has on etherchannel adapter ent2 configured on LACP (aggregation).

root@vios # lsdev|grep ent|grep EtherChannel
ent2        Available             EtherChannel / IEEE 802.3ad Link Aggregation

Here’s etherChannel configuration. It was created with two physical adapters ent0 and ent1 and 8023ad mode:

root@vios # lsattr -EHl ent2
attribute       value          description                                     user_settable

adapter_names   ent0,ent1      EtherChannel Adapters                           True
alt_addr        0x000000000000 Alternate EtherChannel Address                  True
auto_recovery   yes            Enable automatic recovery after failover        True
backup_adapter  NONE           Adapter used when whole channel fails           True
hash_mode       src_dst_port   Determines how outgoing adapter is chosen       True
interval        long           Determines interval value for IEEE 802.3ad mode True
mode            8023ad         EtherChannel mode of operation                  True
netaddr         0              Address to ping                                 True
noloss_failover yes            Enable lossless failover after ping failure     True
num_retries     3              Times to retry ping before failing              True
retry_time      1              Wait time (in seconds) between pings            True
use_alt_addr    no             Enable Alternate EtherChannel Address           True
use_jumbo_frame no             Enable Gigabit Ethernet Jumbo Frames            True

SEA adapter ent9 was created with Etherchannel en2 and 4 Virtual I/O Ethernet Adapters (ent5,ent6,ent7 and ent8) using ha_mode on sharing, see:

root@vios # lsdev|grep ent|grep Shar
ent9        Available             Shared Ethernet Adapter
root@vios # lsattr -EHl ent9
attribute     value               description                                                        user_settable

accounting    enabled             Enable per-client accounting of network statistics                 True
adapter_reset yes                 Reset real adapter on HA takeover                                  True
ctl_chan      ent3                Control Channel adapter for SEA failover                           True
gvrp          no                  Enable GARP VLAN Registration Protocol (GVRP)                      True
ha_mode       sharing             High Availability Mode                                             True
hash_algo     0                   Hash algorithm used to select a SEA thread                         True
jumbo_frames  no                  Enable Gigabit Ethernet Jumbo Frames                               True
large_receive yes                 Enable receive TCP segment aggregation                             True
largesend     1                   Enable Hardware Transmit TCP Resegmentation                        True
lldpsvc       no                  Enable IEEE 802.1qbg services                                      True
netaddr       0                   Address to ping                                                    True
nthreads      7                   Number of SEA threads in Thread mode                               True
pvid          4000                PVID to use for the SEA device                                     True
pvid_adapter  ent5                Default virtual adapter to use for non-VLAN-tagged packets         True
qos_mode      disabled            N/A                                                                True
queue_size    8192                Queue size for a SEA thread                                        True
real_adapter  ent2                Physical adapter associated with the SEA                           True
send_RARP     yes                 Transmit Reverse ARP after HA takeover                             True
thread        1                   Thread mode enabled (1) or disabled (0)                            True
virt_adapters ent5,ent6,ent7,ent8 List of virtual adapters associated with the SEA (comma separated) True

Virtual adapter ent4 was created to management for vios. VIOS’s IP is setup here.

root@vios # lsdev|grep ent4
ent4        Available             Virtual I/O Ethernet Adapter (l-lan)

Virtual adapter ent3 is the control channel between both vios. See attribute ctl_chan on SEA output above.

root@vios # lsdev|grep ent3
ent3        Available             Virtual I/O Ethernet Adapter (l-lan)

Tunning on VIO server, is done on physical adapters, SEA adapter and on trunk adapters.

For SEA adapter ent9 run this command: (It’s recommended to add these parameters when you create the SEA adapter)

chdev -l ent9 -a large_receive=yes -a largesend=1

Although it’s mnot always necessary, you can increase Min Buffers. (reboot required) ( Please check this with command entstat -d entXX)
For all trunk adapters ent5,ent6,ent7 and ent8 increase Min Buffers

root@vios # chdev -l entXX -a max_buf_tiny=3072 -a min_buf_tiny=2048 -P
root@vios # chdev -l entXX -a max_buf_small=4096 -a min_buf_small=4096 -P
root@vios # chdev -l entXX -a max_buf_medium=2048 -a min_buf_medium=2048 -P
root@vios # chdev -l entXX -a max_buf_large=256 -a min_buf_large=256 -P
root@vios # chdev -l entXX -a max_buf_huge=128 -a min_buf_huge=128 -P

Fpr physical adapters ent0 and ent1, enable largesend, largereceive and flow control (reboot required)

chdev -l ent0 -a flow_control=yes -a large_receive=yes -a large_send=yes -P
chdev -l ent1 -a flow_control=yes -a large_receive=yes -a large_send=yes -P

For LPARs enable largesend, RFC 1323 Window Scaling, Socket Buffer Space for Receiving/Sending on all Virtual Ethernet adapters

chdev -l en0 -a mtu_bypass=on -a rfc1323=1 -a tcp_recvspace=262144 -a tcp_sendspace=262144

Check if all this was enabled with ifconfig -a command.

root@aix # ifconfig -a
        inet netmask 0xfffffff0 broadcast
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1

IMPORTANT NOTE! If your external switch / firewall has ethernet speed below 1 Gbits/sec, you have to disable RFC 1323 Window Scaling 
                on virtual adapters on all AIX (for example,  ifconfig en0 rfc1323 0)
That's it!
Thanks if it was helpful. Thanks.
0 (0)
Article Rating (No Votes)
Rate this article
There are no attachments for this article.
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
AIX, Storage, System Admin Allocating shared storage to VIOS clients
Viewed 1879 times since Fri, Apr 19, 2019
Setting new device attributes with chdef
Viewed 1864 times since Mon, Jun 3, 2019
Installation of the Virtual I/O Server directly from the HMC
Viewed 17970 times since Tue, Jun 4, 2019
Script to make mksysb backup from NIM to LPAR
Viewed 3141 times since Thu, Feb 21, 2019
O’Reilly’s CD bookshelf
Viewed 10997 times since Wed, Jun 27, 2018
Install and configure yum on AIX
Viewed 4063 times since Thu, Feb 21, 2019
AIX Net How to check VLAN ID number on AIX?
Viewed 14430 times since Thu, Nov 29, 2018
AIX lspath Missing path
Viewed 8407 times since Fri, Oct 5, 2018
AIX Full memory dump configure
Viewed 2988 times since Mon, Jul 16, 2018
AIX disk queue depth tuning for performance
Viewed 14269 times since Thu, Jan 16, 2020