NMON nmon

ScreenShots

nmon for Linux Screen Shots

On this webpage you can find

Screen shots of using nmon for Linux online
Sample graphs from the nmon Analyser after it generated them from nmon for Linux capture to a file
Also see the nmonchart link on the left hand menu for more graphs using that tool
Some other screen shots from a older nmon for Linux version

New for nmon for Linux 16g

New features in the latest nmon for Linux 16f

New features in the latest nmon for Linux 16e

Other large bugs were fixed like the very busy large machine crashing nmon.

New features in the latest nmon for Linux 16b

KVM host showing guest CPU use
Here are the ten Utilisation categories of a model Linux version - user "U" to show them on screen
The stats are also saved to the nmon files if you add the -U option

Note nice is user time but at a lower priority (use the nice command).
Note guest and guest_nice are for KVM host environments and show the guest OSes CPU use.
Note steal is for a gust OS to show if the CPU is being used to run other Virtual machines.
Note the irq and softirq are CPU time used for interrupts - I have seen some embedded CPU doing most of the CPU work at interrupt level.
Small improvements in the Kernel view too

nmon for Linux Online screen shots

This is the start up screen when using nmon online

There are plenty of hints on how to switch on/off (toggle) the various stats and text graphs
The only one you have to remember if "q" to QUIT

This is the Help section

If you hit "h" you get this Help section with loads of hints. Hit "h" again to remove the Help

If you want more Help then stop nmon and run "nmon -h | less"
Alternatively, read the same output with your brlower using the left-hand Menu "nmon -h" button or
Click: nmon -h

Resources section

Hit "r" for Resources to find out what nmon can discover about your processors and Linux OS version and other features
This is a bit of a weak area in Linux and vendors do this differently or not at all.
In this case we have Intel processor running Ubuntu 15.04 running on four 3.3 GHz

POWER LPAR details

This is Linux on Power running in a Logical Partition (LPAR) for this platform only we have LPAR details (hit "p")
We can see the LPAR is only consuming 1.667 Physical CPU cores out of a machine with 20 Cores with 1 cores and two virtual CPU cores in this LPAR
If you are running on x86 this information is not available and "p" is ignored (actually, not compiled in to the nmon binary).

CPU Stats - Utilisation

Hit "c" for the CPU stats.
Here we see the four CPUs
I ran a load generator (nstress) to get the graphs active
"U" is user time, "s" is System time and "W" id waiting for Disk I/O time.
Obviously blank is used for idle time.
Note the ">" marks are the peak CPU usage - if you hit "0" (zero) these are reset to the far left.

CPU Long Term Graph

Hit "l" (lowercase L) to see these. "l" for long term
On peaky workloads like a web server you can drive yourself mad watching the above graph fluctuate up and down and guessing an average
This graph lets you see the last 70 snapshots and moves to the right. Once full it will wrap around.

Note the top left says 2 seconds
Faster and Slower screen Updates - this works regardless of the stats you are watching:
- Hit "-" to halve the time nmon pauses between updates of the screen i.e. if it is set to 2 seconds now you get updates once a second
- Hit"+" to double the time nmon pauses between updates of the screen i.e. if it is set to 2 seconds now you get updates once in 4 seconds
- You can't go below 1 second - nmon would take too much CPU time for that to make sense
- Faster is good to watch for peaks
- Slower is good to even out the peaks to see a stable average
Also note the "+" on the moving vertical bar - this shows the average number for the graphed data.
Below I hit a "6" to get a 60% horizontal line drawn

CPU Wide or concise View

Hit "C" (uppercase c) to see these.
With lots of CPU cores and lots of CPU core threads (Power Simultaneous Multiple Threads or x86 Hyper-threads sometimes called logical CPUs) we have trouble showing them all on one screen.
This Concise view shows smaller graphs across the screen (rather than down the screen) and up to 192 CPU core threads at a time.

CPU cores with variable MHz

Hit "M" (uppercase m) to see these.
Some machines like the POWER S812LC and S822LC plus new in 2015 Intel PC's (so I am told - if you have one please let me know) can run different CPU cores at different MHz frequencies. Normally, all the CPU cores run as one MHz frequencies (native whole machine or at a VM level) even if they can all rise and fall the MHz together.
The stats are taken from /proc/cpuinfo and are point in time - i.e. not averages of time to there are either the lower CPU Idle frequency of the higher CPU busy and over clocking frequency and never in between.
If you hit "M" three times you cycle through three displays of this feature and the 4th one toggles this off.
1st "M" = the CPU core threads MHz
2st "M" the CPU cores MHz
3st "M" the CPU cores MHz graphs

Memory Stats

Hit "m" for memory details (lowercase M)
Note: this is a Power processor based system - there is no high memory as the Power processor has been 64 bit for a decade or more and these bizarre workaround to extend memory are not required.
We can see 3102 MB of free RAM out of 4038 MB
And the paging space has not been used, yet!

Virtual Memory Stats

Hit "V" for memory details (uppercase v)
Note: These are largely Linux Kernel details

Disk Graphs

Hit "d" for disk graphs
This is, of course, a very small machine with just the one disk.
If we had lots more disk we may not be able to see them all.
Try making the font smaller and/or the window larger to show more lines
The graph shows "W" for Writing and "R" for Reading I/O
Note: the disk like sda and the partition on the disks sda1 both report the same disk I/O = this is double counting the I/O - a kernel bug IMHO but we have a work around this using User Defined Disk Groups or now starting with the "-g auto" feature to automatically define the disks as disk groups (up to 64 disks).

If you hit "." (dot) then only the busy disks are shown - this works on Top Processes too.

Disk Stats by more numbers

Hit "D" for the Disks but this time the Numbers in more Detail.
We can see the numbers of transfers and block sizes etc.
Also we see the peak values - these can be reset to zero by hitting "0" (zero)

Disk Rain Diagram

Hit "o" (lowercase Oh!)
With just one disk this looks stupid (see the single "_" but imagine you had 640.
The panel would have 10 lines and one character per disk
As each disk gets busier more pixels are added to its character.
This gives you a quick idea of how busy your disks are, as a whole.
You also see unused disks or ranges of disks that are particularly busy.

File Systems

Hit "j" for (Journalled) File systems - OK "f" would make more sense but that was used for something else in the past!
Here we see the root filesystem is 16% full
Note the increasing trend to large numbers of fake file systems!!

Network Stats

Hit "n"
All pretty obvious (I hope)

Network Stats - without error numbers

If you wait a few seconds and there are no errors the lower panel is switched off.
It will reappear if network error happen.
On some machines with a dozen networks this saves a lot of screen space

Network File System (NFS)

Hit "N"
OK this machine was not mounted or mounting NFS so the numbers are all zero.
nmon 16a onwards just displays a warning the NFS is not running and covers NFS v2, v3 and v4
- NOTE: NEED TO GRAB AN UP TO DATE IMAGE OF NFS STATS

Kernel Stats

Hit "k"
Linux does not have so may of these but we have the important once like Run Queue, Forks, Load average are not that useful as nmon already collects the detailed CPU stats and we have Uptime here too.

Top Process Stats

Hit "t" to see the processes
Default mode is 3 and this arranges them in CPU use order.
You can see the big hitters are processes called "yes" (I also use "ncpu" from the nstress tools in some screen shots) from the "%CPU Used"
- Note: 100% means using a whole single CPU core thread. If the process has more that 100% then it is obviously multi-threaded.
"Res" columns are the Resident Set i.e. the parts of the program and its data that is currently in memory.
- "Res Set" is the total. "Res Text" is the program code. "Res Data" is the variable data. "Res Lib" is the libraries it is using but note these are probably shared between many running program.

Top Processes in mode 3 = User Commands

(Hit "t") then hit "u" for User command rather than process names - this usually shows yu the command line options used

Top Processes in mode 1 = more details

Have the top Processes on screen (Hit "t") then hit "1" for mode 1
There are different stats shown including status, nice and priority

Top Processes in mode 4 = size order

Have the top Processes on screen (Hit "t") then hit "4" for mode 4
Here the processes are in memory size order
You can see the largest process in 10432 MB and called "ssh" - Normally "Java" is the biggest!

Top Processes in mode 5 = I/O order

Have the top Processes on screen (Hit "t") then hit "5" for mode 5
Here the processes are in I/O in KB per second order
Note: You must be logged in as root for this to work due to odd /proc filesystem read/write permissions = Kernel bug IMHO

Graphical CPU Accelerator stats

Use "a" to see these
Of course, you need to have a NVIDIA GPU adapter or two installed, a special build nmon version and the nvidia ML library available