RHEL: Crash kernel dumps configuration and analysis on RHEL 5 -

RHEL: Crash kernel dumps configuration and analysis on RHEL 5

Apart from this recipe, more information about the configuration can be found in
the file “/usr/share/doc/kexec-tools-*/kexec-kdump-howto.txt“

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/ch-kdump.html

Kernel dumps may provide invaluable insights when debugging serious issues.

1.- Install following RPMs:

         kexec-tools
         crash

    To be able to analyze crash dumps, following packages should be also installed:

         kernel-debuginfo-common
         kernel-debuginfo

2.- Append "crashkernel=128M@16M" to the kernel parameters in /boot/grub/grub.conf
      (see recommended size at the end of this doc.):

         kernel /vmlinuz-2.6.18-238.el5 ro root=/dev/rvg/rootlv nodmraid rhgb quiet crashkernel=128M@16M

3.- Reboot

4.- Check that the crash kernel has been loaded:

         # cat /proc/iomem | grep Crash\ kernel
         01000000-08ffffff : Crash kernel

5.- Configure kdump to dump to:

   - Either locally; add following lines to /etc/kdump.conf:

         path /var/crash
         core_collector makedumpfile -d 31 -c

----------------------------------------------------------------------------

   This config can be done by running

   # system-config-kdump

   Please check the option box "Enable kdump" at the top of the Dialog.

   Next, you have to define the memory to reserve for Kdump In the dialog you
   see the memory information for your system and the usable memory for Kdump.
   On most systems a value of "128MB" Kdump memory should be enough.

   Finally, you need to define a location where to store the dump file. You
   have the choice between 'file', 'nfs', 'ssh', 'raw', 'ext2', and 'ext3'.
   This setup is straight forward, please configure the kdump as it fit’s best
   into your environment. The simplest configuration for the location is
   "file:///var/crash".

----------------------------------------------------------------------------

   - Or to a remote server; add following lines to /etc/kdump.conf:

         net root@<kdump_remote_server>
         core_collector makedumpfile -d 31 -c

(Opt.) 6.- Propagate SSH keys so that the vmcore could be sent via scp without the need to enter any password:

         # service kdump propagate

7.- Configure kdump service to start automatically and start it

         # chkconfig kdump off
         # chkconfig --level 2345 kdump on
         # service kdump start

8.- Sync all filesystems:

         # sync

9.- Provoke a kernel panic with:

         # echo "1" > /proc/sys/kernel/sysrq
            Note that the value of /proc/sys/kernel/sysrq influences only the invocation
            via a keyboard. Invocation of any operation via /proc/sysrq-trigger is always
            allowed (by a user with admin privileges).

         # echo "c" > /proc/sysrq-trigger

10.- Now the crash kernel should get booted and a vmcore should get created under /var/crash:

         # tree /var/crash
         /var/crash
         |-- 192.168.12.227-2010-01-21-20:16:16
         `-- vmcore.flat

11.- The vmcore.flat needs to be processed in order to analyze the core dump via the crash utility:

         # cat "vmcore.flat" | makedumpfile -R "/tmp/vmcore"
         The dumpfile is saved to /tmp/vmcore.
         makedumpfile Completed.

12.- Now you may analyze the vmcore with the crash utility:

         # crash /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux /tmp/vmcore

         crash 4.0-8.9.1.el5
         Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.
         Copyright (C) 2004, 2005, 2006 IBM Corporation
         Copyright (C) 1999-2006 Hewlett-Packard Co
         Copyright (C) 2005, 2006 Fujitsu Limited
         Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
         Copyright (C) 2005 NEC Corporation
         Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
         Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
         This program is free software, covered by the GNU General Public License,
         and you are welcome to change it and/or distribute copies of it under
         certain conditions. Enter "help copying" to see the conditions.
         This program has absolutely no warranty. Enter "help warranty" for details.

         GNU gdb 6.1
         Copyright 2004 Free Software Foundation, Inc.
         GDB is free software, covered by the GNU General Public License, and you are
         welcome to change it and/or distribute copies of it under certain conditions.
         Type "show copying" to see the conditions.
         There is absolutely no warranty for GDB. Type "show warranty" for details.
         This GDB was configured as "x86_64-unknown-linux-gnu"...

          KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux
          DUMPFILE: /tmp/vmcore [PARTIAL DUMP]
          CPUS: 1
          DATE: Thu Jan 21 20:21:20 2010
          UPTIME: 00:03:10
          LOAD AVERAGE: 1.09, 0.46, 0.17
          TASKS: 445
          NODENAME: vrhel03
          RELEASE: 2.6.18-128.1.10.el5
          VERSION: #1 SMP Wed Apr 29 13:53:08 EDT 2009
          MACHINE: x86_64 (2666 Mhz)
          MEMORY: 1 GB
          PANIC: "SysRq : Trigger a crashdump"
          PID: 7835
          COMMAND: "bash"
          TASK: ffff81040699d0c0 [THREAD_INFO: ffff8103fed24000]
          CPU: 1
          STATE: TASK_RUNNING (SYSRQ)

         crash>

*** You need to take care that you have enough disk space on the configured location.

*** Unless the system has enough memory, the Kernel Dump Configuration utility will
not start and you will be presented with an error message. For information on minimum
memory requirements, refer to the Red Hat Enterprise Linux comparison chart. When the
kdump crash recovery is enabled, the minimum memory requirements increase by the amount
of memory reserved for it. This value is determined by the user and on x86, AMD64,
and Intel 64 architectures, it defaults to 128 MB plus 64 MB for each TB of physical
memory (that is, a total of 192 MB for a system with 1 TB of physical memory).

*** Chrashkernel parameters to be used:

+---------------------------------------+
| RAM       | crashkernel | crashkernel |
| size      | parameter   | factor      |
|-----------+-------------+-------------|
| 0 - 2G   | 128M        | 15          |
| 2G - 6G   | 256M        | 23          |
| 6G - 8G   | 512M        | 15          |
| 8G - 24G | 768M        | 31          |
+---------------------------------------+

For RAM size greater than 24G:
   Try crashkernel parameter 768M and RAM/crashkernel factor of 32
   If you get an Out-Of-Memory error message, then try with increasing the crashkernel parameter to 896M

Additional Notes

   - The offset for the kdump memory reservation (crashkernel=X@Y) must be specified in RHEL5.
   Not specifying offset (crashkernel=X) is not a valid configuration under RHEL5, although it
   is valid under RHEL6.

   - kdump fails to initialise with crashkernel=1024M@16M on RHEL5 kernels earlier than 2.6.18-274.el5