RHEL: Crash kernel dumps configuration and analysis on RHEL 5

RHEL: Crash kernel dumps configuration and analysis on RHEL 5

Apart from this recipe, more information about the configuration can be found in
the file “/usr/share/doc/kexec-tools-*/kexec-kdump-howto.txt“


Kernel dumps may provide invaluable insights when debugging serious issues.

1.- Install following RPMs:


    To be able to analyze crash dumps, following packages should be also installed:


2.- Append "crashkernel=128M@16M" to the kernel parameters in /boot/grub/grub.conf
      (see recommended size at the end of this doc.):

         kernel /vmlinuz-2.6.18-238.el5 ro root=/dev/rvg/rootlv nodmraid rhgb quiet crashkernel=128M@16M

3.- Reboot

4.- Check that the crash kernel has been loaded:

         # cat /proc/iomem | grep Crash\ kernel
         01000000-08ffffff : Crash kernel

5.- Configure kdump to dump to:

   - Either locally; add following lines to /etc/kdump.conf:

         path /var/crash
         core_collector makedumpfile -d 31 -c


   This config can be done by running

   # system-config-kdump

   Please check the option box "Enable kdump" at the top of the Dialog.

   Next, you have to define the memory to reserve for Kdump In the dialog you
   see the memory information for your system and the usable memory for Kdump.
   On most systems a value of "128MB" Kdump memory should be enough.

   Finally, you need to define a location where to store the dump file. You
   have the choice between 'file', 'nfs', 'ssh', 'raw', 'ext2', and 'ext3'.
   This setup is straight forward, please configure the kdump as it fit’s best
   into your environment. The simplest configuration for the location is


   - Or to a remote server
; add following lines to /etc/kdump.conf:

         net root@<kdump_remote_server>
         core_collector makedumpfile -d 31 -c

(Opt.) 6.- Propagate SSH keys so that the vmcore could be sent via scp without the need to enter any password:

         # service kdump propagate

7.- Configure kdump service to start automatically and start it

         # chkconfig kdump off
         # chkconfig --level 2345 kdump on
         # service kdump start

8.- Sync all filesystems:

         # sync

9.- Provoke a kernel panic with:

         # echo "1" > /proc/sys/kernel/sysrq
            Note that the value of /proc/sys/kernel/sysrq influences only the invocation
            via a keyboard. Invocation of any operation via /proc/sysrq-trigger is always
            allowed (by a user with admin privileges).

         # echo "c" > /proc/sysrq-trigger

10.- Now the crash kernel should get booted and a vmcore should get created under /var/crash:

         # tree /var/crash
         `-- vmcore.flat

11.- The vmcore.flat needs to be processed in order to analyze the core dump via the crash utility:

         # cat "vmcore.flat" | makedumpfile -R "/tmp/vmcore"
         The dumpfile is saved to /tmp/vmcore.
         makedumpfile Completed.

12.- Now you may analyze the vmcore with the crash utility:

         # crash /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux /tmp/vmcore

         crash 4.0-8.9.1.el5
         Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009  Red Hat, Inc.
         Copyright (C) 2004, 2005, 2006  IBM Corporation
         Copyright (C) 1999-2006  Hewlett-Packard Co
         Copyright (C) 2005, 2006  Fujitsu Limited
         Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
         Copyright (C) 2005  NEC Corporation
         Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
         Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
         This program is free software, covered by the GNU General Public License,
         and you are welcome to change it and/or distribute copies of it under
         certain conditions.  Enter "help copying" to see the conditions.
         This program has absolutely no warranty.  Enter "help warranty" for details.
         GNU gdb 6.1
         Copyright 2004 Free Software Foundation, Inc.
         GDB is free software, covered by the GNU General Public License, and you are
         welcome to change it and/or distribute copies of it under certain conditions.
         Type "show copying" to see the conditions.
         There is absolutely no warranty for GDB.  Type "show warranty" for details.
         This GDB was configured as "x86_64-unknown-linux-gnu"...

          KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux
          DUMPFILE: /tmp/vmcore  [PARTIAL DUMP]
          CPUS: 1
          DATE: Thu Jan 21 20:21:20 2010
          UPTIME: 00:03:10
          LOAD AVERAGE: 1.09, 0.46, 0.17
          TASKS: 445
          NODENAME: vrhel03
          RELEASE: 2.6.18-128.1.10.el5
          VERSION: #1 SMP Wed Apr 29 13:53:08 EDT 2009
          MACHINE: x86_64  (2666 Mhz)
          MEMORY: 1 GB
          PANIC: "SysRq : Trigger a crashdump"
          PID: 7835
          COMMAND: "bash"
          TASK: ffff81040699d0c0  [THREAD_INFO: ffff8103fed24000]
          CPU: 1


*** You need to take care that you have enough disk space on the configured location.

*** Unless the system has enough memory, the Kernel Dump Configuration utility will
not start and you will be presented with an error message. For information on minimum
memory requirements, refer to the Red Hat Enterprise Linux comparison chart. When the
kdump crash recovery is enabled, the minimum memory requirements increase by the amount
of memory reserved for it. This value is determined by the user and on x86, AMD64,
and Intel 64 architectures, it defaults to 128 MB plus 64 MB for each TB of physical
memory (that is, a total of 192 MB for a system with 1 TB of physical memory).

*** Chrashkernel parameters to be used:

| RAM       | crashkernel | crashkernel |
| size      | parameter   | factor      |
|  0 - 2G   | 128M        | 15          |
| 2G - 6G   | 256M        | 23          |
| 6G - 8G   | 512M        | 15          |
| 8G - 24G  | 768M        | 31          |

For RAM size greater than 24G:
   Try crashkernel parameter 768M and RAM/crashkernel factor of 32
   If you get an Out-Of-Memory error message, then try with increasing the crashkernel parameter to 896M

Additional Notes

   - The offset for the kdump memory reservation (crashkernel=X@Y) must be specified in RHEL5.
   Not specifying offset (crashkernel=X) is not a valid configuration under RHEL5, although it
   is valid under RHEL6.

   - kdump fails to initialise with crashkernel=1024M@16M on RHEL5 kernels earlier than 2.6.18-274.el5
0 (0)
Article Rating (No Votes)
Rate this article
There are no attachments for this article.
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
Używanie rsync poprzez Secure Shell
Viewed 38127 times since Thu, May 24, 2018
Linux - How to get IP and MAC address of ethernet adapter in Linux
Viewed 1167 times since Fri, Jun 8, 2018
How to Analyze or Read OS Watcher Output in three easy steps -- With Example ?
Viewed 13327 times since Thu, Jun 21, 2018
Df command in Linux not updating actual diskspace, wrong data
Viewed 1387 times since Wed, May 30, 2018
RHEL: Displaying system info (firmware, serial numbers... )
Viewed 9240 times since Sun, May 27, 2018
Logowanie za pomocą kluczy Secure Shell
Viewed 1717 times since Thu, May 24, 2018
How to configure an SSH proxy server with Squid
Viewed 739 times since Sun, Dec 6, 2020
Viewed 15318 times since Sat, Jun 2, 2018
debian How to Upgrade Debian 8 Jessie to Debian 9 Stretch
Viewed 1247 times since Sun, Sep 23, 2018
RHEL: Back-up/Replicate a partition table
Viewed 1793 times since Sun, May 27, 2018