RHEL: Crash kernel dumps configuration and analysis on RHEL 7

RHEL: Crash kernel dumps configuration and analysis on RHEL 7

# The memory reserved for the kdump kernel is always reserved during system boot. That
# means that the amount of memory is specified in the system's boot loader configuration.

# To enable the dump of the vmcore file, edit /etc/default/grub configuration file and
# set the "crashkernel=auto" option, in GRUB_CMDLINE_LINUX variable according to the
# amount of memory you want to reserve. For example:

GRUB_CMDLINE_LINUX="crashkernel=256M rd.lvm.lv=rootvg/lv_root [...]"


# Minimum amount of reserved memory required for kdump
# ------------------------------------------------------------------------------------------
#
# Architecture                     Available Memory        Minimum Reserved Memory
# AMD64 and Intel 64 (x86_64)         2 GB and more        160 MB + 2 bits
#                                                          for every 4 KB of RAM.
#
# IBM POWER (ppc64)                   2 GB to   4 GB       256 MB of RAM.
#                                     4 GB to  32 GB       512 MB of RAM.
#                                    32 GB to  64 GB         1 GB of RAM.
#                                    64 GB to 128 GB         2 GB or RAM.
#                                   128 GB and more          4 GB of RAM.
#
# IBM System z (s390x)                2 GB and more        160 MB + 2 bits
#                                                          for every 4 KB of RAM.
#
# On some systems, it is possible to allocate memory for kdump automatically, either by
# using the "crashkernel=auto" parameter in the bootloader's configuration file, or by
# enabling this option in the graphical configuration utility. Nevertheless, for this to
# work a certain amount of total memory needs to be available in the system:
#
# Architecture                     Required Memory
# AMD64 and Intel 64 (x86_64)         2 GB
# IBM POWER (ppc64)                   2 GB
# IBM System z (s390x)                4 GB


# Finally, regenerate the GRUB2 configuration:

grub2-mkconfig -o /boot/grub2/grub.cfg



# When capturing a kernel crash, the core dump can be stored in a local filesystem or
# directly on a device, or sent via NFS or SSH. The default option is to store the core
# file in the /var/crash/ directory of the local file system. To change this, as root,
# modify following line in /etc/kdump.conf configuration file:

path /var/crash

# You can choose to write the core file to a different device. Following syntaxes/devices
# are accepted (among others):

# Filesystem name: ext2 /dev/vg/lv_kdump
# Filesystem label: ext3 LABEL=/crash_dump
# Filesystem UUID: ext4 UUID=03138356-5eh1-4ab3-b58e-29a07ac41x37
# Raw device: raw /dev/vg/lv_kdump
# NFS location: nfs my.server.com:/export/kdump
# SSH connection: ssh user@my.server.com line
# (if a SSH key is required, add "sshkey /root/.ssh/kdump_id_rsa" line too)



# We can configure the action to perform in case dumping to intended targer fails.
# If no default action is specified, "reboot" is assumed default.

default <reboot | halt | poweroff | shell | dump_to_rootfs>



# To reduce the size of the vmcore dump file, kdump allows to specify a program to compress
# the data, and optionally leave out all irrelevant information. Currently, the only fully
# supported core collector is "makedumpfile", by default configured like this:

core_collector makedumpfile -l --message-level 1 -d 31




# Generating a vmcore file (test purposes)
# ------------------------------------------------------------------------------------------

# Before testing, make sure that the service is running:

systemctl is-active kdump
   active

# If needed, enable and start kdump daemon:
systemctl enable kdump.service
systemctl start kdump.service


# With kdump daemon running, execute following commands:

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

# That will force de kernel to crash. * Ensure that you have enough disk space to store
# the core dump.




# Analyzing a core file
# ------------------------------------------------------------------------------------------

# First of all, install crash utility and kernel-debuginfo package which provides the
# data necessary for dump analysis:

yum install crash

rpm -ihv <kernel-debuginfo-common-x86_64-3.10.0-327.el7.x86_64.rpm>
rpm -ihv <kernel-debuginfo-3.10.0-327.el7.x86_64.rpm>


# Once necessary tools have been installed, you can analyze the core file:

crash /usr/lib/debug/lib/modules/3.10.0-327.el7.x86_64/vmlinux \
   /var/crash/127.0.0.1-2016-01-26-22\:30\:26/vmcore

   crash 7.1.2-2.el7
   Copyright (C) 2002-2014  Red Hat, Inc.
[...]

   GNU gdb (GDB) 7.6
   Copyright (C) 2013 Free Software Foundation, Inc.
   License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
   This is free software: you are free to change and redistribute it.
   There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
   and "show warranty" for details.
   This GDB was configured as "x86_64-unknown-linux-gnu"...

         KERNEL: /usr/lib/debug/lib/modules/3.10.0-327.el7.x86_64/vmlinux
       DUMPFILE: /var/crash/127.0.0.1-2016-01-26-22:30:26/vmcore  [PARTIAL DUMP]
           CPUS: 1
           DATE: Tue Jan 26 22:30:15 2016
         UPTIME: 00:09:55
   LOAD AVERAGE: 0.02, 0.10, 0.11
          TASKS: 139
       NODENAME: myserver.localdomain
        RELEASE: 3.10.0-327.el7.x86_64
        VERSION: #1 SMP Thu Nov 19 22:10:57 UTC 2015
        MACHINE: x86_64  (2009 Mhz)
         MEMORY: 2.5 GB
          PANIC: "SysRq : Trigger a crash"
            PID: 3433
        COMMAND: "bash"
           TASK: ffff8800994d2280  [THREAD_INFO: ffff88009b044000]
            CPU: 0
          STATE: TASK_RUNNING (SYSRQ)

crash>



# To display the kernel message buffer, type the "log" command at the crash prompt:

crash> log


# To show the kernel stack trace; "bt <pid>" to display the backtrace of a single process:

crash> bt


# Status of processes:

crash> ps


# Virtual memory information of the current context:

crash> vm


# Information about open files of the current context:

crash> files


crash> exit




# ------------------------------------------------------------------------------------------

# For more information about memory requirements, supported kdump targets, filtering
# levels, dump analysis, etc, refer to Red Hat official documentation.
0 (0)
Article Rating (No Votes)
Rate this article
Attachments
There are no attachments for this article.
Comments
There are no comments for this article. Be the first to post a comment.
Full Name
Email Address
Security Code Security Code
Related Articles RSS Feed
10 Linux cryptsetup Examples for LUKS Key Management (How to Add, Remove, Change, Reset LUKS encryption Key)
Viewed 4862 times since Tue, Jul 31, 2018
Zabijanie wszystkich procesów użytkownika
Viewed 2444 times since Thu, May 24, 2018
tcpdump
Viewed 8853 times since Fri, Jul 27, 2018
SPRAWDZONA KONFIGURACJA RSYSLOG I LOGROTATE, JAKO ZEWNĘTRZNEGO SERWERA SYSLOG
Viewed 3510 times since Fri, Nov 30, 2018
debian Install a newer kernel in Debian 9 (stretch) stable
Viewed 1582 times since Sun, Sep 23, 2018
HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples
Viewed 4666 times since Fri, Nov 30, 2018
RHEL: Reinstalling Boot Loader on the Master Boot Record (MBR)
Viewed 3050 times since Sun, May 27, 2018
Linux - How to monitor memory usage
Viewed 2790 times since Fri, Jun 8, 2018
15 Linux Yum Command Examples – Install, Uninstall, Update Packages
Viewed 2990 times since Thu, Oct 25, 2018
Red Hat 8 How to Set Up Automatic Updates for CentOS 8
Viewed 3379 times since Fri, Sep 25, 2020