Produce And Deliver System Utilization Reports (processor, memory, disk, and network)

The sysstat package in Linux automatically logs various system resource information, including processor, memory, disk and network.

We can process these logs with the ‘sadf’ command in order to generate basic reports which outline system resource usage over a defined period of time.

Sysstat is great as it is usually installed and already running by default, at least that was my experience with a fresh install of CentOS 7.

If you don’t have it installed, it’s fairly straightforward to install, enable, and start.

yum install sysstat -y
systemctl enable sysstat
systemctl start sysstat

For further information on basic service management with systemctl, see our guide here.

Sysstat will log resource usage data into files in the /var/log/sa directory, with new data being written every 10 minutes. This is defined by the /etc/cron.d/sysstat cron job outlined below.

[[email protected] ~]# cat /etc/cron.d/sysstat
# Run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# 0 * * * * root /usr/lib64/sa/sa1 600 6 &
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

The number on the end of a ‘sa’ file corresponds to the day of the month that it holds data for.

At 23:53 every day, the ‘sa’ file is taken and becomes a ‘sar’ file which is readable as a text file. These files actually make a pretty good daily report of all available information, however we can go further and pull out specific information that we want to report on with ‘sadf’.

The configuration for sysstat is found in the /etc/sysconfig/sysstat file and is shown below.

# sysstat-10.1.5 configuration file.

# How long to keep log files (in days).
# If value is greater than 28, then log files are kept in
# multiple directories, one for each month.
HISTORY=28

# Compress (using gzip or bzip2) sa and sar files older than (in days):
COMPRESSAFTER=31

# Parameters for the system activity data collector (see sadc manual page)
# which are used for the generation of log files.
SADC_OPTIONS="-S DISK"

# Compression program to use.
ZIP="bzip2"

This file is fairly simple and self explanatory, by default we hold 28 days worth of log files, and compress after 31 days with bzip2. Obviously this means that compression never happens by default, however this can be adjusted as required. The log files are typically fairly small, so you may want to hold a longer period of logs and compress more aggressively.

Generating Reports

As mentioned we can use the ‘sadf’ command to generate reports and pull out specific data that we want, in the format that we want. For full details, check out the manual page for the command, we’ll cover the basics here.

In this example command below, we are using -s to specify a start time of 12:00:00 and -e to specify an end time of 23:59:59 for the log file created on the 15th of the month. This is being stored to /root/report.

sadf -s 12:00:00 -e 23:59:59 /var/log/sa/sa15 > /root/report

By default the output is run with -p, which is a format usable for the ‘awk’ command. We can instead specify -x to get XML output, or -d which will provide the output in a format that can be ingested by a relational database.

[[email protected] sa]# sadf /var/log/sa/sa22 -p | head -n 5
localhost.localdomain 600       2016-05-22 07:10:01 UTC all     %user   0.06
localhost.localdomain 600       2016-05-22 07:10:01 UTC all     %nice   0.00
localhost.localdomain 600       2016-05-22 07:10:01 UTC all     %system 0.16
localhost.localdomain 600       2016-05-22 07:10:01 UTC all     %iowait 0.00
localhost.localdomain 600       2016-05-22 07:10:01 UTC all     %steal  0.00

[[email protected] sa]# sadf /var/log/sa/sa22 -x
        <file-date>2016-05-22</file-date>
        <statistics>
         <timestamp date="2016-05-22" time="07:10:01" utc="1" interval="600">
           <cpu-load>
              <cpu number="all" user="0.06" nice="0.00" system="0.16" iowait="0.00" steal="0.00" idle="99.78"/>
           </cpu-load>
         </timestamp>

[[email protected] sa]# sadf /var/log/sa/sa22 -d | head -n 5
# hostname;interval;timestamp;CPU;%user;%nice;%system;%iowait;%steal;%idle
localhost.localdomain;600;2016-05-22 07:10:01 UTC;-1;0.06;0.00;0.16;0.00;0.00;99.78
localhost.localdomain;600;2016-05-22 07:20:01 UTC;-1;0.03;0.00;0.10;0.00;0.00;99.87
localhost.localdomain;600;2016-05-22 07:30:01 UTC;-1;0.05;0.00;0.13;0.00;0.00;99.82
localhost.localdomain;600;2016-05-22 07:40:01 UTC;-1;0.04;0.00;0.11;0.01;0.00;99.85

By default this pulls out all information, however we can use ‐‐ to specify sar commands, such as -u for cpu, -r for memory, or -d for disk activity. For full information on what is available, checkout the ‘sar’ manual page.

[[email protected] sa]# sadf /var/log/sa/sa22 -- -d | head -n 5
localhost.localdomain 600       2016-05-22 07:10:01 UTC dev8-0  tps     0.19
localhost.localdomain 600       2016-05-22 07:10:01 UTC dev8-0  rd_sec/s        0.00
localhost.localdomain 600       2016-05-22 07:10:01 UTC dev8-0  wr_sec/s        2.00
localhost.localdomain 600       2016-05-22 07:10:01 UTC dev8-0  avgrq-sz        10.60
localhost.localdomain 600       2016-05-22 07:10:01 UTC dev8-0  avgqu-sz        0.00

[[email protected] sa]# sadf /var/log/sa/sa22 -- -r | head -n 5
localhost.localdomain 600       2016-05-22 07:10:01 UTC -       kbmemfree       120708
localhost.localdomain 600       2016-05-22 07:10:01 UTC -       kbmemused       1747988
localhost.localdomain 600       2016-05-22 07:10:01 UTC -       %memused        93.54
localhost.localdomain 600       2016-05-22 07:10:01 UTC -       kbbuffers       76
localhost.localdomain 600       2016-05-22 07:10:01 UTC -       kbcached        828392

With all of these combinations and the logged information you can create some pretty specific and verbose resource usage reports.

Summary

As shown we can make use of sysstat to produce reports of system resource usage, including CPU, RAM, disk and network information. This historical information can be useful when investigating and troubleshooting problems that occurred in the past.

While commands such as ‘top’ are useful for getting an idea of the system at the current point in time, sysstat allows us to go back and better understand the state of the system at a particular time.


This post is part of our Red Hat Certified Engineer (RHCE) exam study guide series. For more RHCE related posts and information check out our full RHCE study guide.

  1. nice tip
    thanks for that!

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>