I had the pleaseure to setup from scratch an IBM pureflex p270 system, including chassis, ethernet switches, fiber channel switches, FSM, dual VIOS and LPARs. Although, all connections inside pureflex node is at 10 Gbits/sec, this chassis is connected to a 1Gb cisco backbone. Once all components were on production, I found there was an additional setup on ethernets devices to approach maximum speed inside the p270 node. All internal ports of ethernet switches EN4093r were configured to 10Gbits/sec.
Here you can find what tunning I did on VIOS and LPARs.
Let’s tart with VIO servers.
Both vios have two physical interface 10Gb/s. See:
root@vios # lsdev|grep ent|grep 10 ent0 Available 00-00 10GbE 4-port Mezzanine Adapter (a2191007df1033e7) ent1 Available 00-01 10GbE 4-port Mezzanine Adapter (a2191007df1033e7)
Also both vio servers has on etherchannel adapter ent2 configured on LACP (aggregation).
root@vios # lsdev|grep ent|grep EtherChannel ent2 Available EtherChannel / IEEE 802.3ad Link Aggregation
Here’s etherChannel configuration. It was created with two physical adapters ent0 and ent1 and 8023ad mode:
root@vios # lsattr -EHl ent2 attribute value description user_settable adapter_names ent0,ent1 EtherChannel Adapters True alt_addr 0x000000000000 Alternate EtherChannel Address True auto_recovery yes Enable automatic recovery after failover True backup_adapter NONE Adapter used when whole channel fails True hash_mode src_dst_port Determines how outgoing adapter is chosen True interval long Determines interval value for IEEE 802.3ad mode True mode 8023ad EtherChannel mode of operation True netaddr 0 Address to ping True noloss_failover yes Enable lossless failover after ping failure True num_retries 3 Times to retry ping before failing True retry_time 1 Wait time (in seconds) between pings True use_alt_addr no Enable Alternate EtherChannel Address True use_jumbo_frame no Enable Gigabit Ethernet Jumbo Frames True
SEA adapter ent9 was created with Etherchannel en2 and 4 Virtual I/O Ethernet Adapters (ent5,ent6,ent7 and ent8) using ha_mode on sharing, see:
root@vios # lsdev|grep ent|grep Shar ent9 Available Shared Ethernet Adapter
root@vios # lsattr -EHl ent9 attribute value description user_settable accounting enabled Enable per-client accounting of network statistics True adapter_reset yes Reset real adapter on HA takeover True ctl_chan ent3 Control Channel adapter for SEA failover True gvrp no Enable GARP VLAN Registration Protocol (GVRP) True ha_mode sharing High Availability Mode True hash_algo 0 Hash algorithm used to select a SEA thread True jumbo_frames no Enable Gigabit Ethernet Jumbo Frames True large_receive yes Enable receive TCP segment aggregation True largesend 1 Enable Hardware Transmit TCP Resegmentation True lldpsvc no Enable IEEE 802.1qbg services True netaddr 0 Address to ping True nthreads 7 Number of SEA threads in Thread mode True pvid 4000 PVID to use for the SEA device True pvid_adapter ent5 Default virtual adapter to use for non-VLAN-tagged packets True qos_mode disabled N/A True queue_size 8192 Queue size for a SEA thread True real_adapter ent2 Physical adapter associated with the SEA True send_RARP yes Transmit Reverse ARP after HA takeover True thread 1 Thread mode enabled (1) or disabled (0) True virt_adapters ent5,ent6,ent7,ent8 List of virtual adapters associated with the SEA (comma separated) True
Virtual adapter ent4 was created to management for vios. VIOS’s IP is setup here.
root@vios # lsdev|grep ent4 ent4 Available Virtual I/O Ethernet Adapter (l-lan)
Virtual adapter ent3 is the control channel between both vios. See attribute ctl_chan on SEA output above.
root@vios # lsdev|grep ent3 ent3 Available Virtual I/O Ethernet Adapter (l-lan)
Tunning on VIO server, is done on physical adapters, SEA adapter and on trunk adapters.
For SEA adapter ent9 run this command: (It’s recommended to add these parameters when you create the SEA adapter)
chdev -l ent9 -a large_receive=yes -a largesend=1
Although it’s mnot always necessary, you can increase Min Buffers. (reboot required) ( Please check this with command entstat -d entXX)
For all trunk adapters ent5,ent6,ent7 and ent8 increase Min Buffers
root@vios # chdev -l entXX -a max_buf_tiny=3072 -a min_buf_tiny=2048 -P root@vios # chdev -l entXX -a max_buf_small=4096 -a min_buf_small=4096 -P root@vios # chdev -l entXX -a max_buf_medium=2048 -a min_buf_medium=2048 -P root@vios # chdev -l entXX -a max_buf_large=256 -a min_buf_large=256 -P root@vios # chdev -l entXX -a max_buf_huge=128 -a min_buf_huge=128 -P
Fpr physical adapters ent0 and ent1, enable largesend, largereceive and flow control (reboot required)
chdev -l ent0 -a flow_control=yes -a large_receive=yes -a large_send=yes -P chdev -l ent1 -a flow_control=yes -a large_receive=yes -a large_send=yes -P
For LPARs enable largesend, RFC 1323 Window Scaling, Socket Buffer Space for Receiving/Sending on all Virtual Ethernet adapters
chdev -l en0 -a mtu_bypass=on -a rfc1323=1 -a tcp_recvspace=262144 -a tcp_sendspace=262144
Check if all this was enabled with ifconfig -a command.
root@aix # ifconfig -a
en4: flags=1e084863,5c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>
inet 192.168.1.1 netmask 0xfffffff0 broadcast 192.168.1.250
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
IMPORTANT NOTE! If your external switch / firewall has ethernet speed below 1 Gbits/sec, you have to disable RFC 1323 Window Scaling
on virtual adapters on all AIX (for example, ifconfig en0 rfc1323 0)
That's it!
Thanks if it was helpful. Thanks.
Article Number: 503
Posted: Thu, Feb 21, 2019 5:48 PM
Last Updated: Thu, Feb 21, 2019 5:48 PM
Online URL: http://kb.ictbanking.net/article.php?id=503