Recovery from LED 552, 554, or 556 in AIX

Technote (FAQ)

Question

Recovery from LED 552, 554, or 556 system hang conditions in AIX.

Answer

This document discusses the known causes of LED 552, 554, and 556. Included is a procedure for recovery from these errors. This document was originally written for AIX Versions 4 and 5 but applies to all versions.

Causes of an LED 552, 554, or 556
Recovery procedure
Recommended fixes

Causes of an LED 552, 554, or 556

An LED code of 552, 554, or 556 during a standard disk based boot indicates a failure occurred during the varyon of the rootvg volume group.

Some known causes of an LED 552, 554, or 556 are:

a corrupted file system
a corrupted Journaled File System (JFS) log device
a bad IPL-device record or bad IPL-device magic number; the magic number indicates the device type
a corrupted copy of the Object Data Manager (ODM) database on the boot logical volume
a fixed disk (hard disk) in the inactive state in the root volume group

Recovery procedure

To diagnose and fix the problem, boot to a Service mode shell and run the fsck command (file system check) on each file system. If the file system check fails, you may need to perform other steps.

WARNING: Do not use this document if the system is a /usr client, diskless client, or dataless client.

Boot your system into a limited function maintenance shell (Service or Maintenance mode) from bootable AIX media to use this recovery procedure.
Refer to your system user's or installation and service guide for specific IPL procedures related to your type and model of hardware. You can also refer to the document titled "Booting in Service Mode", available at http://techsupport.services.ibm.com/server/aix.srchBroker for more information.
With bootable media of the same version and level as the system, boot the system into Service mode. The bootable media can be any ONE of the following:
- Bootable CD-ROM
- mksysb
- Bootable Install Tape
Follow the screen prompts to the Welcome to Base OS menu.
Choose Start Maintenance Mode for System Recovery (Option 3). The next screen displays prompts for the Maintenance menu.
- Choose Access a Root Volume Group (Option 1).
  The next screen displays a warning that indicates you will not be able to return to the Base OS menu without rebooting.
- Choose 0 continue.
  The next screen displays information about all volume groups on the system.
- Select the root volume group by number. The logical volumes in rootvg will be displayed with two options below.
- Choose Access this volume group and start a shell before mounting the file systems (Option 2).
If you receive errors from the preceding option, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the following:
- local branch office
- your point of sale
- your AIX support center
Run the following commands to check and repair file systems.
```
   fsck -p /dev/hd4 
   fsck -p /dev/hd2 
   fsck -p /dev/hd9var 
   fsck -p /dev/hd3
   fsck -p /dev/hd1 
```
NOTE: The -y option gives the fsck command permission to repair file system corruption when necessary. This flag can be used to avoid having to manually answer multiple confirmation prompts, however, use of this flag can cause permanent data loss in some situations.

If any of the following conditions occur, proceed accordingly.
- If fsck indicates that block 8 could not be read, the file system is probably unrecoverable. See step 5 for information on unrecoverable file systems.
- If fsck indicates that a file system has an unknown log record type, or if fsck fails in the logredo process, then go to step 6.
- If the file system checks were successful, skip to step 8.
The easiest way to fix an unrecoverable file system is to recreate it. This involves deleting it from the system and restoring it from a very current system backup. Note that hd9var and hd3 can be recreated, but hd4 and hd2 cannot be recreated. If hd4 and/or hd2 is unrecoverable, AIX must be reinstalled or restored from system backup. For assistance with unrecoverable file systems, contact your local branch office, point of sale, or AIX support center. Do not follow the rest of the steps in this document.
A corruption of the JFS log logical volume has been detected. Use the logform command to reformat it.
```
   /usr/sbin/logform /dev/hd8
```
Answer yes when asked if you want to destroy the log.
Repeat step 4 for all file systems that did not successfully complete fsck the first time. If step 4 fails a second time, the file system is almost always unrecoverable. See step 5 for an explanation of the options at this point. In most cases, step 4 will be successful. If step 4 is successful, continue to step 8.
With the key in Normal position (for microchannel machines), run the following commands to reboot the system:
```
   exit
   sync;sync;sync;reboot
```
As you reboot in Normal mode, notice how many times LED 551 appears. If LED 551 appears twice, fsck is probably failing because of a bad fshelper file. If this is the case and you are running AFS, see step 11.

The majority of instances of LED 552, 554, and 556 will be resolved at this point. If you still have an LED 552, 554, or 556, you may try the following steps.

ATTENTION: The following steps will overwrite your Object Data Manager (ODM) database files with a very primitive, minimal ODM database. Due to the potential loss of user configuration data caused by this procedure, it should only be used as a last resort effort to gain access to your system to attempt to back up any data that you can. It is NOT recommended to use the following procedure in lieu of restoring from a system backup.
Repeat step 1 through step 3.
Run the following commands, which remove much of the system's configuration and save it to a backup directory.
```
   mount /dev/hd4 /mnt
   mount /dev/hd2 /mnt/usr
   mkdir /mnt/etc/objrepos/bak
   cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
   cp /etc/objrepos/Cu* /mnt/etc/objrepos
   umount /dev/hd2
   umount /dev/hd4
   exit
```
Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output.
```
   lslv -m hd5
```
Save the clean ODM database to the boot logical volume. (# is the number of the fixed disk, determined with the previous command.)
```
   savebase -d /dev/hdisk# 
```
If you are running AFS, go to step 11; otherwise, go to step 12.
If you are running the Andrew File System (AFS), use the following commands to find out whether you have more than one version of the v3fshelper file.
```
   cd /sbin/helpers
   ls -l v3fshelper*
```
If you have only one version of the v3fshelper file (for example, v3fshelper), proceed to step 12.

If there is a version of v3fshelper marked as original (for example, v3fshelper.orig), run the following commands:
```
   cp v3fshelper v3fshelper.afs
   cp v3fshelper.orig v3fshelper
```
WARNING: Do not proceed further if the system is a /usr client, diskless client, or dataless client.
Make sure that hd5 is on the edge of the drive and if it is more than 1 partition that the partitions are contiguous. For systems of 5.1 and above, make sure that hd5 is greater than 12 MB:
```
     lslv hd5 (Check to see what the PP Size: is equal to)
     lslv -m hd5
```
```
  LP    PP1  PV1           PP2   PV2                    PP3   PV3
0001   0001 hdisk2
0002   0002 hdisk2
```
Recreate the boot image (hdisk# is the fixed disk determined in step 11):
```
   bosboot -a -d /dev/hdisk# 
```
Make sure the bootlist is set correctly:
```
     bootlist -m normal -o
```
Make changes, if necessary:
```
     bootlist -m normal hdiskX cdX
```
(This can be edited to whatever you wish it to be.)
NOTE: If you suspect an inactive or damaged disk device is causing the boot problem and the boot logical device, hd5, is mirrored onto another device, you may wish to list this other boot device first in the bootlist.

Make sure that the disk drive that you have chosen as your bootable device has a yes next to it:
```
     ipl_varyon -i
```
Example:
```
PVNAME                     BOOT DEVICE       
PVID                                 VOLUME GROUP ID
hdisk1                     NO                
0007b53cbfd04a9000000000000000000007b53c00004c00
hdisk4                     NO                
0007b53c1244625d00000000000000000007b53c00004c00
hdisk2                     YES               
0007b53c8ffd631200000000000000000007b53c00004c00
```
From the above example, hdisk2 would be a bootable disk drive while hdisk1 and hdisk4 would not be.
If you copied files in step 11, copy the AFS file system helper back to v3fshelper:
```
   cp v3fshelper.afs v3fshelper
```
Turn the key to the Normal position if dealing with microchannel machine and run
```
   sync;sync;sync;reboot
```

If you followed all of the preceding steps and the system still stops at an LED 552, 554, or 556 during a reboot in Normal mode, you may want to consider reinstalling your system from a recent backup. Isolating the cause of the hang could be excessively time-consuming and may not be cost-effective in your operating environment. To isolate the possible cause of the hang, would require a debug boot of the system. Instructions for doing this are included in the document "Capturing Boot Debug", available at http://techsupport.services. ibm.com/server/aix.srchBroker . It is still possible, in the end, that isolation of the problem may indicate a restore or reinstall of AIX is necessary to correct it.

If you wish to pursue further system recovery, you may be able to obtain assistence from one of the following: