RHEL: Crash kernel dumps configuration and analysis on RHEL 5

RHEL: Crash kernel dumps configuration and analysis on RHEL 5

Apart from this recipe, more information about the configuration can be found in
the file “/usr/share/doc/kexec-tools-*/kexec-kdump-howto.txt“

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/ch-kdump.html



Kernel dumps may provide invaluable insights when debugging serious issues.


1.- Install following RPMs:

         kexec-tools
         crash

    To be able to analyze crash dumps, following packages should be also installed:

         kernel-debuginfo-common
         kernel-debuginfo


2.- Append "crashkernel=128M@16M" to the kernel parameters in /boot/grub/grub.conf
      (see recommended size at the end of this doc.):

         kernel /vmlinuz-2.6.18-238.el5 ro root=/dev/rvg/rootlv nodmraid rhgb quiet crashkernel=128M@16M


3.- Reboot


4.- Check that the crash kernel has been loaded:

         # cat /proc/iomem | grep Crash\ kernel
         01000000-08ffffff : Crash kernel


5.- Configure kdump to dump to:

   - Either locally; add following lines to /etc/kdump.conf:

         path /var/crash
         core_collector makedumpfile -d 31 -c

----------------------------------------------------------------------------

   This config can be done by running

   # system-config-kdump

   Please check the option box "Enable kdump" at the top of the Dialog.

   Next, you have to define the memory to reserve for Kdump In the dialog you
   see the memory information for your system and the usable memory for Kdump.
   On most systems a value of "128MB" Kdump memory should be enough.

   Finally, you need to define a location where to store the dump file. You
   have the choice between 'file', 'nfs', 'ssh', 'raw', 'ext2', and 'ext3'.
   This setup is straight forward, please configure the kdump as it fit’s best
   into your environment. The simplest configuration for the location is
   "file:///var/crash".

----------------------------------------------------------------------------


   - Or to a remote server
; add following lines to /etc/kdump.conf:

         net root@<kdump_remote_server>
         core_collector makedumpfile -d 31 -c



(Opt.) 6.- Propagate SSH keys so that the vmcore could be sent via scp without the need to enter any password:

         # service kdump propagate


7.- Configure kdump service to start automatically and start it

         # chkconfig kdump off
         # chkconfig --level 2345 kdump on
         # service kdump start


8.- Sync all filesystems:

         # sync


9.- Provoke a kernel panic with:

         # echo "1" > /proc/sys/kernel/sysrq
            Note that the value of /proc/sys/kernel/sysrq influences only the invocation
            via a keyboard. Invocation of any operation via /proc/sysrq-trigger is always
            allowed (by a user with admin privileges).

         # echo "c" > /proc/sysrq-trigger


10.- Now the crash kernel should get booted and a vmcore should get created under /var/crash:

         # tree /var/crash
         /var/crash
         |-- 192.168.12.227-2010-01-21-20:16:16
         `-- vmcore.flat


11.- The vmcore.flat needs to be processed in order to analyze the core dump via the crash utility:

         # cat "vmcore.flat" | makedumpfile -R "/tmp/vmcore"
         The dumpfile is saved to /tmp/vmcore.
         makedumpfile Completed.


12.- Now you may analyze the vmcore with the crash utility:

         # crash /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux /tmp/vmcore

         crash 4.0-8.9.1.el5
         Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009  Red Hat, Inc.
         Copyright (C) 2004, 2005, 2006  IBM Corporation
         Copyright (C) 1999-2006  Hewlett-Packard Co
         Copyright (C) 2005, 2006  Fujitsu Limited
         Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
         Copyright (C) 2005  NEC Corporation
         Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
         Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
         This program is free software, covered by the GNU General Public License,
         and you are welcome to change it and/or distribute copies of it under
         certain conditions.  Enter "help copying" to see the conditions.
         This program has absolutely no warranty.  Enter "help warranty" for details.
    
         GNU gdb 6.1
         Copyright 2004 Free Software Foundation, Inc.
         GDB is free software, covered by the GNU General Public License, and you are
         welcome to change it and/or distribute copies of it under certain conditions.
         Type "show copying" to see the conditions.
         There is absolutely no warranty for GDB.  Type "show warranty" for details.
         This GDB was configured as "x86_64-unknown-linux-gnu"...

          KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux
          DUMPFILE: /tmp/vmcore  [PARTIAL DUMP]
          CPUS: 1
          DATE: Thu Jan 21 20:21:20 2010
          UPTIME: 00:03:10
          LOAD AVERAGE: 1.09, 0.46, 0.17
          TASKS: 445
          NODENAME: vrhel03
          RELEASE: 2.6.18-128.1.10.el5
          VERSION: #1 SMP Wed Apr 29 13:53:08 EDT 2009
          MACHINE: x86_64  (2666 Mhz)
          MEMORY: 1 GB
          PANIC: "SysRq : Trigger a crashdump"
          PID: 7835
          COMMAND: "bash"
          TASK: ffff81040699d0c0  [THREAD_INFO: ffff8103fed24000]
          CPU: 1
          STATE: TASK_RUNNING (SYSRQ)

         crash>





*** You need to take care that you have enough disk space on the configured location.

*** Unless the system has enough memory, the Kernel Dump Configuration utility will
not start and you will be presented with an error message. For information on minimum
memory requirements, refer to the Red Hat Enterprise Linux comparison chart. When the
kdump crash recovery is enabled, the minimum memory requirements increase by the amount
of memory reserved for it. This value is determined by the user and on x86, AMD64,
and Intel 64 architectures, it defaults to 128 MB plus 64 MB for each TB of physical
memory (that is, a total of 192 MB for a system with 1 TB of physical memory).

*** Chrashkernel parameters to be used:

+---------------------------------------+
| RAM       | crashkernel | crashkernel |
| size      | parameter   | factor      |
|-----------+-------------+-------------|
|  0 - 2G   | 128M        | 15          |
| 2G - 6G   | 256M        | 23          |
| 6G - 8G   | 512M        | 15          |
| 8G - 24G  | 768M        | 31          |
+---------------------------------------+

For RAM size greater than 24G:
   Try crashkernel parameter 768M and RAM/crashkernel factor of 32
   If you get an Out-Of-Memory error message, then try with increasing the crashkernel parameter to 896M


Additional Notes

   - The offset for the kdump memory reservation (crashkernel=X@Y) must be specified in RHEL5.
   Not specifying offset (crashkernel=X) is not a valid configuration under RHEL5, although it
   is valid under RHEL6.

   - kdump fails to initialise with crashkernel=1024M@16M on RHEL5 kernels earlier than 2.6.18-274.el5
0 (0)
Article Rating (No Votes)
Rate this article
Attachments
There are no attachments for this article.
Comments
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
LVM: Create a new Volume Group
Viewed 632 times since Sat, Jun 2, 2018
Sample logrotate configuration and troubleshooting part 2
Viewed 463 times since Fri, Nov 30, 2018
A Quick and Practical Reference for tcpdump
Viewed 1641 times since Fri, Jul 27, 2018
Using stunnel to Encrypt Database Connections
Viewed 994 times since Fri, Sep 28, 2018
Linux File Systems (mkfs, mount, fstab) ext4
Viewed 803 times since Sat, Jun 2, 2018
Understanding logrotate utility part 1
Viewed 460 times since Fri, Nov 30, 2018
Open SSL HowTo: Decode SSL Certificate
Viewed 477 times since Mon, Feb 18, 2019
linux unix aix banner /etc/issue
Viewed 510 times since Fri, Aug 3, 2018
OCFS2 Cluster File System Setup Guide in Linux
Viewed 1387 times since Sat, Jun 2, 2018
Linux An introduction to swap space on Linux systems
Viewed 318 times since Thu, Jan 23, 2020