AIX HA / HACMP, System Admin↑ Mountguard

This article describes how to add a new volume group to an existing resource group of an active PowerHA cluster.

The first step is to add the storage to both of the nodes of the PowerHA cluster. In the case of SAN storage, please ensure that your storage administrator adds the storage to both nodes of the cluster. Then discover the newly added storage by running the cfgmgr command on one of the nodes:

# cfgmgr

Set a PVID on all the new disks that have been discovered. For example, for disk hdisk77, run:

# chdev -l hdisk77 -a pv=yes

Repeat this command for any of the new disks.

Next, log in to the other node, and run the cfgmgr command on that node as well, so the disks will be discovered on the other node as well. And when you run "lspv" on the second node after running cfgmgr, you'll notice that the PVID is already set for all the discovered disks (it was set on the first node).

On both nodes, make sure that the disk attributes are set correctly. Now, this may differ for the type of storage used (so please make sure to check your storage vendor's recommendations on this topic), but a good starting point is (for example, for disk hdisk4):

# chdev -l hdisk4 -a max_transfer=0x100000 -a queue_depth=32 -a reserve_policy=no_reserve -a algorithm=round_robin

For clustered nodes, it's important that the reserve_policy is set to no_reserve.

A note about the max_transfer attribute: This value should be set to the same value as the max_xfer_size attribute of the fiber adapter. By default attribute max_transfer is set to 0x40000, which is usually lower than the max_xfer_size attribute on the fiber adapter, which results in a smaller buffer size memory being used. To check the max_xfer_size attribute on the fiber adapter, run (for example for adapter fcs0):

# lsattr -El fcs0 -a max_xfer_size

Also please make sure to set the disk attributes for all the new disks on both nodes. These attributes are stored in the ODM of the AIX system locally, and don't automatically transfer over to other nodes of the cluster, so you'll have to set these attributes for all new disks on all cluster nodes.

The next step is to create the new volume group(s). The first thing that we'll need to know is a common majar number that is available on both nodes of a cluster. For that purpose, run the following command on all cluster nodes, which lists the available major numbers:

# lvlstmajor

Choose a major number available on all cluster nodes. For the purpose of this article, let's assume major number 57 is available.

For PowerHA, a new volume group should be configured as concurrent-capable (configured by the -C option of the mkvg command). Also, The quorum should be disabled (by using -Qn), and the auto-varyon should be disabled as well (by using the -n option) as PowerHA will varyon the volume group for us. Finally, the major number should be set (in the example below: -V 57).

As such, run the following mkvg command to create the volume group (Note: please adjust the volume group name, the major number and disk names according to your situation and preference):

# mkvg -v 57 -S -n -Qn -y fs01vg hdisk38 hdisk42 hdisk45

Note: run this command on only one of the cluster nodes, and continue working on this cluster node for now for the next steps.

Next, create a logical volume (adjust per your situation and preference):

# mklv -y fs01lv -t jfs2 -u 1 -x 1278 hdisk38

This command will create logical volume fs01lv for the purpose of using it for a JFS2 type file system, with an upper bound of 1 (just use 1 disk), and allocate 1278 partitions on disk hdisk38.

Then create a file system on the previously defined logical volume:

# crfs -v jfs2 -d fs01lv -a logname=INLINE -a options=noatime -m /fs01 -A no

This command will create file system /fs01 on top of logical volume fs01lv, and will use an inline log (recommended for optimal performance), will not record access times (options=noatime) to avoid unneccessary writes to the file system, and will tell AIX not to automatically mount the file system at system start (-A no), as PowerHA will mount the file system instead.

At this point, the volume group, logical volume and file system have been created. You can create additional volume groups, logical volumes and file systems as it pertains to your situation. Once done, ensure that the volume group is varied off, for example for volume group fs01vg:

# varyoffvg fs01vg

Should the varyoff command fail at this point, then please check if the file system(s) is/are still mounted, and un-mount them before running the varyoffvg command.

Now run "lspv" on both nodes of the cluster, and look at the PVIDs listed. Pick one of the disks on which you configured the volume group on the first node, and take that disk's PVID, for example 00fac78651b28b53. On the other node, run the importvg command to import the volume group (which also imports the information about any logical volumes and file systems configured in that volume group), using the PVID of one of the disks, and the major number previously defined, for example:

# importvg -n -V 57 -y fs01vg 00fac78651b28b53

Repeat this for any additional volume groups in your situation. Do make sure to select the correct PVID of one of the disks in the volume group on the first node when importing this volume group on the second node, as you'll want to make sure both nodes of a PowerHA cluster use the same volume group names.

At this time, the storage is properly configured on both nodes, and it is time to add the volume group(s) to the PowerHA resource group.

First, verify that the cluster is in a good state by running:

# smitty hacmp
  problem determination tools
    powerha verification
      verify powerha configuration

Once confirmed, allow PowerHA to discover the new disks in the cluster:

# smitty cm_discover_nw_interfaces_and_disks

Now add the volume group(s) to the resource group of the cluster:

# smitty hacmp
  cluster applications and resources
    resource groups
      change/show resources and attributes for a resource group

Select your resource group, and add the volume group(s) to the "Volume Groups" entry.

Next, sync the cluster, which should bring the volume group(s) online:

# smitty hacmp
  cluster applications and resources
    verify and synchronize cluster configuration

Once this is complete, you should be able to see the new volume group(s) online, by running:

# lsvg -o

And you should be able to see that any new file systems have been mounted by PowerHA:

# df

Not required, but good practice, is to now schedule a failover test for your PowerHA cluster, to ensure everything is still working as it should, in case of a fail-over scenario.

 

 

Support matrix / life cycle for IBM PowerHA (with a typical 3 year lifecycle):

	AIX 5.1	AIX 5.2	AIX 5.3	AIX 6.1	AIX 7.1	AIX 7.2	Release Date	End Of Support
HACMP 5.1	Yes	Yes	Yes	No	No	No	7/11/2003	9/1/2006
HACMP 5.2	Yes	Yes	Yes	No	No	No	7/16/2004	9/30/2007
HACMP 5.3	No	ML4+	ML2+	Yes	No	No	8/12/2005	9/30/2009
HACMP 5.4.0	No	TL8+	TL4+	No	No	No	7/28/2006	9/30/2011
HACMP 5.4.1	No	TL8+	TL4+	Yes	Yes	No	9/11/2007	9/30/2011
PowerHA 5.5	No	No	TL7+	tl2 sp1+	Yes	No	11/14/2008	4/30/2012
PowerHA 6.1	No	No	TL9+	tl2 sp1+	Yes	No	10/20/2009	4/30/2015
PowerHA 7.1.0	No	No	No	tl6+	Yes	No	9/10/2010	9/30/2014
PowerHA 7.1.1	No	No	No	tl7 sp2+	tl1 sp2+	No	9/10/2010	4/30/2015
PowerHA 7.1.2	No	No	No	tl8 sp1+	tl2 sp1+	No	10/3/2012	4/30/2016
PowerHA 7.1.3	No	No	No	tl9 sp1+	tl3 sp1+	No	10/7/2013	4/30/2018
PowerHA 7.2.0	No	No	No	tl9 sp5+	tl3 sp5+ tl4 sp1+	tl0 sp1+	12/4/2015	4/30/2019
PowerHA 7.2.1	No	No	No	No	tl3+	tl0 sp1+	12/16/2016	4/30/2020
PowerHA 7.2.2	No	No	No	No	tl4+	tl0 sp1+	12/15/2017	tbd

Source: PowerHA for AIX Version Compatibility Matrix

 

PowerHA version 7 (starting with version 7.1.3) includes a new feature that allows you to generate an HTML report that includes a lot of very useful information of your PowerHA cluster. There are no external requirements, and it is included is the base PowerHA product.

The command needed to generate the report is clmgr. For example:

# clmgr view report cluster TYPE=html FILE=/tmp/powerha.report

You may also include a company logo and name, for example by running:

# clmgr view report cluster TYPE=html FILE=/tmp/powerha.report COMPANY_LOGO="mylogo.jpg" COMPANY_NAME="MY COMPANY"

Tip: You may schedule this command through cron to get a regular up to date version.

This HTML report is officially only supported on Internet Explorer and Firefox, although other browsers might work just fine.

 

In PowerHA version 7.2.2, you can use a graphical user interface (GUI) to monitor your cluster environment.

The PowerHA GUI provides the following advantages over the PowerHA command line: Monitor the status for all clusters, sites, nodes, and resource groups in your environment. Scan event summaries and read a detailed description for each event. If the event occurred because of an error or issue in your environment, you can read suggested solutions to fix the problem. Search and compare log files. Also, the format of the log file is easy to read and identify important information. View properties for a cluster such as the PowerHA SystemMirror version, name of sites and nodes, and repository disk information.

Check out a video that provides an overview for the PowerHA GUI at https://www.youtube.com/watch?v=d_QVvh2dcCM.

Information on how to install and start using it can be found on the IBM website.

IBM has implemented a new feature implemented for JFS2 filesystems to prevent simultaneous mounting within PowerHA clusters.

While PowerHA can give concurrent access of volume groups to multiple systems, mounting a JFS2 filesystem on multiple nodes simultaneously will cause filesystem corruption. These simultaneous mount events can also cause a system crash, when the system detects a conflict between data or metadata in the filesystem and the in-memory state of the filesystem. The only exception to this is mounting the filesystem read-only, where files or directories can't be changed.

In AIX 7100-01 and 6100-07 a new feature called "Mount Guard" has been added to prevent simultaneous or concurrent mounts. If a filesystem appears to be mounted on another server, and the feature is enabled, AIX will prevent mounting on any other server. Mount Guard is not enabled by default, but is configurable by the system administrator. The option is not allowed to be set on base OS filesystems such as /, /usr, /var etc.

To turn on Mount Guard on a filesystem you can permanently enable it via /usr/sbin/chfs:

# chfs -a mountguard=yes /mountpoint
/mountpoint is now guarded against concurrent mounts.

The same option is used with crfs when creating a filesystem.

To turn off mount guard:

# chfs -a mountguard=no /mountpoint
/mountpoint is no longer guarded against concurrent mounts.

To determine the mount guard state of a filesystem:

# lsfs -q /mountpoint
Name      Nodename Mount Pt    VFS  Size    Options Auto Accounting
/dev/fslv --       /mountpoint jfs2 4194304 rw      no   no
  (lv size: 4194304, fs size: 4194304, block size: 4096, sparse files: yes,
  inline log: no, inline log size: 0, EAformat: v1, Quota: no, DMAPI: 
  no, VIX: yes, EFS: no, ISNAPSHOT: no, MAXEXT: 0, MountGuard: yes)

The /usr/sbin/mount command will not show the mount guard state.

When a filesystem is protected against concurrent mounting, and a second mount attempt is made you will see this error:

# mount /mountpoint
mount: /dev/fslv on /mountpoint:
Cannot mount guarded filesystem.
The filesystem is potentially mounted on another node

After a system crash the filesystem may still have mount flags enabled and refuse to be mounted. In this case the guard state can be temporarily overridden by the "noguard" option to the mount command:

# mount -o noguard /mountpoint
mount: /dev/fslv on /mountpoint:
Mount guard override for filesystem.
The filesystem is potentially mounted on another node.

Reference: http://www-01.ibm.com/support/docview.wss?uid=isg3T1018853