RHEL: Crash kernel dumps configuration and analysis on RHEL 5

RHEL: Crash kernel dumps configuration and analysis on RHEL 5

Apart from this recipe, more information about the configuration can be found in
the file “/usr/share/doc/kexec-tools-*/kexec-kdump-howto.txt“


Kernel dumps may provide invaluable insights when debugging serious issues.

1.- Install following RPMs:


    To be able to analyze crash dumps, following packages should be also installed:


2.- Append "crashkernel=128M@16M" to the kernel parameters in /boot/grub/grub.conf
      (see recommended size at the end of this doc.):

         kernel /vmlinuz-2.6.18-238.el5 ro root=/dev/rvg/rootlv nodmraid rhgb quiet crashkernel=128M@16M

3.- Reboot

4.- Check that the crash kernel has been loaded:

         # cat /proc/iomem | grep Crash\ kernel
         01000000-08ffffff : Crash kernel

5.- Configure kdump to dump to:

   - Either locally; add following lines to /etc/kdump.conf:

         path /var/crash
         core_collector makedumpfile -d 31 -c


   This config can be done by running

   # system-config-kdump

   Please check the option box "Enable kdump" at the top of the Dialog.

   Next, you have to define the memory to reserve for Kdump In the dialog you
   see the memory information for your system and the usable memory for Kdump.
   On most systems a value of "128MB" Kdump memory should be enough.

   Finally, you need to define a location where to store the dump file. You
   have the choice between 'file', 'nfs', 'ssh', 'raw', 'ext2', and 'ext3'.
   This setup is straight forward, please configure the kdump as it fit’s best
   into your environment. The simplest configuration for the location is


   - Or to a remote server
; add following lines to /etc/kdump.conf:

         net root@<kdump_remote_server>
         core_collector makedumpfile -d 31 -c

(Opt.) 6.- Propagate SSH keys so that the vmcore could be sent via scp without the need to enter any password:

         # service kdump propagate

7.- Configure kdump service to start automatically and start it

         # chkconfig kdump off
         # chkconfig --level 2345 kdump on
         # service kdump start

8.- Sync all filesystems:

         # sync

9.- Provoke a kernel panic with:

         # echo "1" > /proc/sys/kernel/sysrq
            Note that the value of /proc/sys/kernel/sysrq influences only the invocation
            via a keyboard. Invocation of any operation via /proc/sysrq-trigger is always
            allowed (by a user with admin privileges).

         # echo "c" > /proc/sysrq-trigger

10.- Now the crash kernel should get booted and a vmcore should get created under /var/crash:

         # tree /var/crash
         `-- vmcore.flat

11.- The vmcore.flat needs to be processed in order to analyze the core dump via the crash utility:

         # cat "vmcore.flat" | makedumpfile -R "/tmp/vmcore"
         The dumpfile is saved to /tmp/vmcore.
         makedumpfile Completed.

12.- Now you may analyze the vmcore with the crash utility:

         # crash /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux /tmp/vmcore

         crash 4.0-8.9.1.el5
         Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009  Red Hat, Inc.
         Copyright (C) 2004, 2005, 2006  IBM Corporation
         Copyright (C) 1999-2006  Hewlett-Packard Co
         Copyright (C) 2005, 2006  Fujitsu Limited
         Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
         Copyright (C) 2005  NEC Corporation
         Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
         Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
         This program is free software, covered by the GNU General Public License,
         and you are welcome to change it and/or distribute copies of it under
         certain conditions.  Enter "help copying" to see the conditions.
         This program has absolutely no warranty.  Enter "help warranty" for details.
         GNU gdb 6.1
         Copyright 2004 Free Software Foundation, Inc.
         GDB is free software, covered by the GNU General Public License, and you are
         welcome to change it and/or distribute copies of it under certain conditions.
         Type "show copying" to see the conditions.
         There is absolutely no warranty for GDB.  Type "show warranty" for details.
         This GDB was configured as "x86_64-unknown-linux-gnu"...

          KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux
          DUMPFILE: /tmp/vmcore  [PARTIAL DUMP]
          CPUS: 1
          DATE: Thu Jan 21 20:21:20 2010
          UPTIME: 00:03:10
          LOAD AVERAGE: 1.09, 0.46, 0.17
          TASKS: 445
          NODENAME: vrhel03
          RELEASE: 2.6.18-128.1.10.el5
          VERSION: #1 SMP Wed Apr 29 13:53:08 EDT 2009
          MACHINE: x86_64  (2666 Mhz)
          MEMORY: 1 GB
          PANIC: "SysRq : Trigger a crashdump"
          PID: 7835
          COMMAND: "bash"
          TASK: ffff81040699d0c0  [THREAD_INFO: ffff8103fed24000]
          CPU: 1


*** You need to take care that you have enough disk space on the configured location.

*** Unless the system has enough memory, the Kernel Dump Configuration utility will
not start and you will be presented with an error message. For information on minimum
memory requirements, refer to the Red Hat Enterprise Linux comparison chart. When the
kdump crash recovery is enabled, the minimum memory requirements increase by the amount
of memory reserved for it. This value is determined by the user and on x86, AMD64,
and Intel 64 architectures, it defaults to 128 MB plus 64 MB for each TB of physical
memory (that is, a total of 192 MB for a system with 1 TB of physical memory).

*** Chrashkernel parameters to be used:

| RAM       | crashkernel | crashkernel |
| size      | parameter   | factor      |
|  0 - 2G   | 128M        | 15          |
| 2G - 6G   | 256M        | 23          |
| 6G - 8G   | 512M        | 15          |
| 8G - 24G  | 768M        | 31          |

For RAM size greater than 24G:
   Try crashkernel parameter 768M and RAM/crashkernel factor of 32
   If you get an Out-Of-Memory error message, then try with increasing the crashkernel parameter to 896M

Additional Notes

   - The offset for the kdump memory reservation (crashkernel=X@Y) must be specified in RHEL5.
   Not specifying offset (crashkernel=X) is not a valid configuration under RHEL5, although it
   is valid under RHEL6.

   - kdump fails to initialise with crashkernel=1024M@16M on RHEL5 kernels earlier than 2.6.18-274.el5
0 (0)
Article Rating (No Votes)
Rate this article
There are no attachments for this article.
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
RHEL: Reserved space on a ext2/ext3/ext4 filesystem
Viewed 2669 times since Sun, May 27, 2018
RHEL: Displaying/setting kernel parameters - ’sysctl’
Viewed 1276 times since Sat, Jun 2, 2018
Linux - How to get network speed and statistic of ethernet adapter in Linux
Viewed 721 times since Fri, Jun 8, 2018
LVM: Mount LVM Partition(s) in Rescue Mode
Viewed 1337 times since Sat, Jun 2, 2018
tcpdump usage examples
Viewed 672 times since Fri, Jul 27, 2018
ZFS: Remove an existing zfs filesystem
Viewed 786 times since Sun, Jun 3, 2018
Top 20 OpenSSH Server Best Security Practices ssh linux aix
Viewed 340 times since Fri, May 15, 2020
How to clear swap memory in Linux
Viewed 190 times since Mon, Nov 23, 2020
LVM: Reduce an existing Logical Volume / Filesystem
Viewed 849 times since Sat, Jun 2, 2018
Tunnel SSH Connections Over SSL Using ‘Stunnel’ On Debian 7 / Ubuntu 13.10
Viewed 929 times since Fri, Sep 28, 2018