IVM and VLAN Tagging -

IVM and VLAN Tagging

cggibbo |Nov 8 2011| Comments (9) | Visits (39247)

I was working with a customer recently on a Power Blade that was running the Integrated Virtualisation Manager (IVM). They’d installed a VIO partition onto the Blade and had hoped to install a couple of AIX LPARs on the system. However they didn’t get very far.

As soon as they attempted to NIM install the LPARs, they would get stuck at trying to ping the NIM master from the client. Basically, the Shared Ethernet Adapter (SEA) was not working properly and none of the LPARs could communicate with the external network. So they asked for some assistance.

The Blade server name was Server-8406-71Y-SN06BF99Z. The SEA was configured as ent7.

# lsdev -Cc adapter

ent0 Available Logical Host Ethernet Port (lp-hea)

ent1 Available Logical Host Ethernet Port (lp-hea)

ent2 Available Virtual I/O Ethernet Adapter (l-lan)

ent3 Available Virtual I/O Ethernet Adapter (l-lan)

ent4 Available Virtual I/O Ethernet Adapter (l-lan)

ent5 Available Virtual I/O Ethernet Adapter (l-lan)

ent6 Available EtherChannel / IEEE 802.3ad Link Aggregation

ent7 Available Shared Ethernet Adapter

ent9 Available Virtual I/O Ethernet Adapter (l-lan)

fcs0 Available 02-00 8Gb PCIe FC Blade Expansion Card (7710322577107601)

fcs1 Available 02-01 8Gb PCIe FC Blade Expansion Card (7710322577107601)

ibmvmc0 Available Virtual Management Channel

lhea0 Available Logical Host Ethernet Adapter (l-hea)

sissas0 Available 01-08 PCI-X266 Planar 3Gb SAS Adapter

usbhc0 Available 00-08 USB Host Controller (33103500)

usbhc1 Available 00-09 USB Host Controller (33103500)

usbhc2 Available 00-0a USB Enhanced Host Controller (3310e000)

vhost0 Available Virtual SCSI Server Adapter

vsa0 Available LPAR Virtual Serial Adapter

vts0 Available Virtual TTY Server Adapter

The SEA was configured with Port-VLAN ID (PVID) of 68 without any VLAN tags. This was the root cause of the problem.

# lsattr -El ent7

accounting enabled Enable per-client accounting of network statistics True

ctl_chan Control Channel adapter for SEA failover True

gvrp no Enable GARP VLAN Registration Protocol (GVRP) True

ha_mode disabled High Availability Mode True

jumbo_frames no Enable Gigabit Ethernet Jumbo Frames True

large_receive no Enable receive TCP segment aggregation True

largesend 0 Enable Hardware Transmit TCP Resegmentation True

netaddr 0 Address to ping True

pvid 68 PVID to use for the SEA device True

pvid_adapter ent9 Default virtual adapter to use for non-VLAN-tagged packets True

qos_mode disabled N/A True

real_adapter ent6 Physical adapter associated with the SEA True

thread 1 Thread mode enabled (1) or disabled (0) True

virt_adapters ent9 List of virtual adapters associated with the SEA (comma separated) True

$ entstat -all ent8 | grep -i vlan

VLAN Ids :

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID: 68

VLAN Tag IDs: None

On the network switch port, the native VLAN (PVID), was configured as 11, with VLAN tag 68 added as an allowed VLAN. If the client LPARs tried to access the network using a PVID of 68, instead of a VLAN TAG of 68, they would get stuck at the switch port i.e. the un-tagged packets for 10.1.68.X via PVID 11 would fail. The packets for 10.1.68.X needed to be tagged with VLAN id 68 in order for the switch to pass the traffic.

So the question was, how do we add VLAN tags in the IVM environment? If we’d been using a HMC, then this would be simple to fix. Just add the VLAN tags into the Virtual Ethernet Adapter used by the SEA and we’d be done.

We had to use the lshwres and chhwres commands to resolve this one. First we listed the virtual adapters known to the VIO server (IVM). At slot 12, we found our SEA adapter with port_vlan_id set to 68 and addl_vlan_ids set to none.

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

We needed to change port_vlan_id to 11 and addl_vlan_ids to 68. We also required the ieee_virtual_eth value set to 1.

First we removed the existing SEA adapter, as we would not be able to make changes to it while it was “active”. We then removed the adapter from slot 12 and then re-added it, again at slot 12, with port_vlan_id and addl_vlan_ids set to the desired values.

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o r

$ chhwres -m Server-8406-71Y-SN06BF99Z -p 06-BF99Z -r virtualio --rsubtype eth -s 12 -o a -a "ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1" -d 5

$ lshwres -r virtualio --rsubtype eth --level lpar

lpar_name=06-BF99Z,lpar_id=1,slot_num=3,state=1,ieee_virtual_eth=0,port_vlan_id=1,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62803

lpar_name=06-BF99Z,lpar_id=1,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=2,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62804

lpar_name=06-BF99Z,lpar_id=1,slot_num=5,state=1,ieee_virtual_eth=0,port_vlan_id=3,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62805

lpar_name=06-BF99Z,lpar_id=1,slot_num=6,state=1,ieee_virtual_eth=0,port_vlan_id=4,addl_vlan_ids=none,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA62806

lpar_name=06-BF99Z,lpar_id=1,slot_num=12,state=1,ieee_virtual_eth=1,port_vlan_id=11,addl_vlan_ids=68,is_trunk=1,trunk_priority=1,is_required=0,mac_addr=F67D5DA6280C

lpar_name=aixlpar1,lpar_id=2,slot_num=4,state=1,ieee_virtual_eth=0,port_vlan_id=68,addl_vlan_ids=none,is_trunk=0,trunk_priority=0,is_required=0,mac_addr=F67D5345AD04

Using the mkvdev command we created the SEA again. Then using the entstat command we found that the PVID and VLAN tags had been configured correctly.

$ mkvdev -sea ent6 -vadapter ent2 -default ent2 -defaultid 11

ent7 Available

en7

et7

$ entstat -all ent7 | grep -i vlan

VLAN Ids :

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

VLAN Extract: False

VLAN tagged filtering mode: Filter according to VLAN permit array

Max number of VLAN IDs per HEA port: 20

Invalid VLAN ID Packets: 5388

Port VLAN ID: 11

VLAN Tag IDs: 68

Once this was done, the client LPARs were able to ping the NIM master. The customer happily started installing AIX onto each of the blades Partitions.