high swap space utilization in LINUX

This morning, I got a ticket that one LINUX machine is about to consume all its swap space….. I addressed this ticket very much like I would do it if this was AIX – only the commands to resolve this situations have different names. The general idea and the flow of action is identical.

First, check swap space usage.

# free -h
             total       used       free     shared    buffers     cached
Mem:           15G        15G       349M       444K       186M        14G
-/+ buffers/cache:       752M        14G
Swap:         3.9G         3.7B      0.2G

How many swap spaces we are dealing with…..?

# swapon -s -v
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       4095996 0       -2

How it is named in /etc/fstab?

# grep swap /etc/fstab
/dev/mapper/vg_sys-lv_swap swap                    swap    defaults        0 0

Any room left in “vg-sys” for a second swap volume?

# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  vg_data   1   2   0 wz--n- 400.00g 1020.00m
  vg_sys    1   7   0 wz--n-  34.61g    8.19g

There is space, so let’s create a second swap volume with 6GB. This will be a temporary volume that will be deleted later.

#  lvcreate -n swap2 -L 6G vg_sys
  Logical volume "swap2" created.

Let’s turn it into another swap area.

# # mkswap /dev/mapper/vg_sys-swap2
mkswap: /dev/mapper/vg_sys-swap2: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 6291452 KiB
no label, UUID=eff630c2-e516-4e5e-a9fe-28cee7b46b1a

Now, let shake the contents of swap aka let’s kick them back to RAM via the new swap volume just made. Note, that the swap2 is activated before swap is deactivated…
Do otherwise and kernel may kill some very important to you processes.

# swapon /dev/mapper/vg_sys-swap2 && swapoff /dev/mapper/vg_sys-swap

Monitor, wait till swap2 stabilizes and reverse the last action.

# swapon /dev/mapper/vg_sys-swap && swapoff /dev/mapper/vg_sys-swap2

Monitor swap and when it stabilizes and it is empty or almost empty remove swap2 and it infrastructure.

# lvremove /dev/mapper/vg_sys-swap2
Do you really want to remove active logical volume swap2? [y/n]: y
  Logical volume "swap2" successfully removed

The proverbial icing on a cake is this little snippet (I found today on the net) – it lists swap usage per a running processes.

# for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
rsyslogd 3488 kB
filebeat 1364 kB
abrtd 996 kB
cupsd 940 kB
master 828 kB
qmgr 820 kB
sshd 640 kB
certmonger 576 kB
crond 520 kB
hald 484 kB
udevd 476 kB
udevd 476 kB
rpc.mountd 476 kB

Posted in LINUX.