After login in, I had to verify that it is indeed hdisk0 that died,

Let’s try to “wake” this disk. Maybe it will come back on line?

I want use this opportunity to re-size the dump volumes of this host – I deactivate both as I will remove them both later.

For the very last time – can I wake it up?

Apparently, hdisk0 is really dead. No joke. Let’s check if any swap needs to be de-activated (the one on hdisk0) as well.

Today, I am lucky. No swap has been defined on hdisk0. Otherwise, we would have to execute the next command (where the swap_lv is the name of the swap volume to be removed).

The previously de-activated volume has to be removed executing:

The calling home host provided IBM with all the information about the missing disk so I have not been asked to provide ant FRU or Z? info (lscfg -vl hdisk0) would provide all the answers). To satisfy my own curiosity, I execute the next command and proceed with the remaining tasks.

The bootlist has to be modified as hdisk0 is useless and it cannot be used as a boot device.

hdisk1 will be the only device the host knows to boot from.

The dead disk has to be removed from its volume group which is a two step process. First, the mirrors have to be broken removing the mirror residing on hdisk0.

Next, step is the actual removal of the disk from its volume group.

The command fails not because of an error. Well, we could say that it is an error and that the error is mine. OK. I de-activated the dump volume residing on hdisk0 but as far as AIX is concerned this volume is still in use – it is still there on the disk. So I have to remove it (regardless the disk is dead or not). As you can see next, AIX still can read the disk.

The next in line is the disk removal from the group.

If I executed the next command and removed the disks, I would not be able to see it executing the diag hot plug-gable tasks…

I would have to execute the configmangler aka the cfgmgr command to get it back so diag could present it for me. Execute diag, hit the ENTER key, slide down to the Task Selection (Diagnostics, Advanced Diagnostics, Service Aids, etc.), hit CTRL-V once, and slide to the Hot Plug Task. From the next screen select SCSI and SCSI RAID Hot Plug Manager, slide down and select Replace/Remove a Device Attached to an SCSI Hot Swap Enclosure Device. From the next and the final screen select the desired disk (hdisk0 in our case). Hit the ENTER key to define your selection and proceed with the disk replacement.
Hit the same key again to declare the swap of the disk completed and after you leave the diag and are back at the command prompt execute cfgmgr (for a good measure).

If you are a puritan, you may execute the next two steps. Otherwise proceed directly to the task at hand and add the new hdisk0 to the host rootvg.

Before we get lost in the activities ahead, let’s stop for a moment and regroup. What do we need to complete this process. We need (not necessarly in the order shown) dump volumes (one per disk), re-mirror rootvg and the bootlist needs to be modified to again include both disks. OK.
To re-create and re-sync the mirrors in rootvg, the following command has to be executed.

After my return, I check the state of my volume group. I noticed that some logical volume are still stale, apparently I returned sooner than expected.