News  |  Features  |  DTD/XSD  |  GitHub  |  Download  |  FAQ  |  About

Features

---

Sysstat's main features

• Includes four groups of monitoring tools (sar / sadc / sadf, iostat / tapestat / cifsiostat, mpstat, pidstat) for global system performance analysis.
• Can monitor a huge number of different metrics:

1. Input / Output and transfer rate statistics (global, per device, per partition, per network filesystem and per Linux task / PID).
2. CPU statistics (global, per CPU, per NUMA nodes and per Linux task / PID), including support for virtualization architectures.
3. Memory, hugepages and swap space utilization statistics.
4. Virtual memory, paging and fault statistics.
5. Per-task (per-PID) memory and page fault statistics.
6. Global CPU and page fault statistics for tasks and all their children.
7. Process creation activity.
8. Interrupt statistics (global, per CPU and per interrupt, including potential APIC interrupt sources, hardware and software interrupts).
9. Extensive network statistics: network interface activity (number of packets and kB received and transmitted per second, etc.) including failures from network devices; network traffic statistics for IP, TCP, ICMP and UDP protocols based on SNMPv2 standards; support for IPv6-related protocols.
10. Fibre Channel traffic statistics.
11. Software-based network processing (softnet) statistics.
12. NFS server and client activity.
13. Sockets statistics.
14. Run queue and system load statistics.
15. Kernel internal tables utilization statistics.
16. System and per Linux task switching activity.
17. Swapping statistics.
18. TTY devices activity.
19. Power management statistics (instantaneous and average CPU clock frequency, fans speed, devices temperature, voltage inputs)
20. USB devices plugged into the system.
21. Filesystems utilization (inodes and blocks).
22. Tape drives statistics.
23. Pressure-Stall Information statistics.

• Can generate graphs (SVG format - Scalable Vector Graphics) that can be displayed in your favorite web browser!
• Average statistics values are calculated over the sampling period.
• Most system statistics can be saved in a file for future inspection.
• Allows to configure the length of data history to keep.
• On the fly detection of new devices (disks, network interfaces, etc.) that are created or registered dynamically.
• Support for UP and SMP machines, including machines with hyperthreaded or multi-core processors.
• Support for hotplug CPUs (it detects automagically processors that are disabled or enabled on the fly) and tickless CPUs.
• Works on many different architectures, whether 32- or 64-bit.
• Needs very little CPU time to run (written in C).
• System statistics collected by sar/sadc can be exported in various different formats (CSV, XML, JSON, SVG, etc.). DTD and XML Schema documents are included in sysstat package. JSON output format is also available for mpstat and iostat commands.
• sar data can also be exported by sadf to PCP (Performance Co-Pilot) archive.
• Smart color output for easier statistics reading.
• Internationalization support (sysstat has been translated into numerous different languages). Sysstat is now part of the Translation Project.
• Sysstat commands can automatically select the unit used to display sizes for easier reading (see option --human).
• Many programs available on the internet to use sysstat's data to make graphs (one of them, isag, is included in sysstat).
• iostat has support for devices managed by drivers in userspace like spdk.

If you think that other statistics could be useful for the sys admin and so should be added to sysstat, please tell me.

Last but not least, the Matrix of activities lists all the possible activities for sar and the corresponding options to use with sar and sadc.

---

Main features and improvements added to the latest sysstat versions

Sysstat 12.6 stable series - Why should I upgrade from sysstat 12.4.x to 12.6.x?

• sar and sadf commands have been updated so that persistent device names can be displayed with filesystems statistics.
• sar: Code used to collect and display interrupts statistics has been completely reworked. Interrupts are now read from /proc/interrupts file instead of /proc/stat file. A list of interrupts can also be entered on the command line using option --int=.
• sar: A new metric (softnet network backlog) has been added to A_NET_SOFT report.
• Option --rotate has been added to sa1 script to make it easier to handle file rotation.
• Basic colorization has been added to sadf's output.
• Link Time Optimization (LTO) is now supported when building sysstat.
• A debug option (which can be selected with -o debug) has been added to sadf's SVG output.
• Monitoring a process with pidstat's option -e has been improved.
• Option --compact has been added to iostat to indicate that all metrics shall be displayed on a single line.

Sysstat 12.4 stable series - Why should I upgrade from sysstat 12.2.x to 12.4.x?

• sadc/sar now collect and display Pressure-Stall Information (PSI) statistics (metrics added to Linux kernel 4.20).
• iostat can now display flush I/O statistics (statistics which have been made available in Linux kernel 5.5).
• iostat can now display statistics for devices managed by userspace drivers (e.g. spdk). New flags (-f/+f) may be used to specify an alternate location for statistics files.
• mpstat has got a new switch (-T) to display system topology: core, socket and node numbers. It is also possible now to select individual CPU and nodes to be displayed even when option -A (to display all possible statistics) has been selected.
• cifsiostat command has gained support for SMB2 version of statistics file.
• pidstat and cifsiostat have been largely rewritten for better performance.
• A new option (hz=) has been added to sadf to specify kernel's HZ value when an old binary saDD datafile has to be converted to the up-to-date format.
• All sysstat commands now display their statistics in color by default when the output is connected to a terminal.
• sar also pretty-prints the name of the devices (those displayed with option -d) by default. A new option (--pretty) has been added to cifsiostat, sar and iostat.
• sa1 script has been updated to insert a comment in the current daily datafile on system suspend or resume.
• It is now possible to tell sa2 script to wait for a random delay before running. This can be useful to prevent a massive I/O burst on some systems.
• sysstat code has been made compliant with latest gcc version (v10).
• Manual pages are now compressed by default when they are installed.

Sysstat 12.2 stable series - Why should I upgrade from sysstat 12.0.x to 12.2.x?

• sar: A stable identifier has been added for disk statistics. This stable identifier, which won't change across reboots, is the device's World Wide Name (WWN). New metrics have also been added to sar Huge Pages statistics (HugePages_Rsvd and HugePages_Surp).
• sadc: It is now possible to unselect activities by name (e.g. sadc -S XALL,-A_PAGE will collect all possible activities except paging statistics). Also a new flag (-f) has been added to force fdatasync() use. Last, the timezone value is now saved in the binary data files.
• sadf has been improved to make it more robust to corrupted datafiles.
• SVG: New options have been added to sadf (customcol and bwcol) to enable the user to select distinct color palettes to draw the graphs with sadf -g. The color palette can also be customized by the user.
• PCP: sadf can now export sar data in the format expected by PCP (Performance Co-Pilote).
• iostat and sar have gained support for discard I/O statistics which can be displayed on recent linux kernels. iostat has also been largely rewritten for better performance (device structures are now dynamically allocated, iostat better handles the case when devices are removed then inserted again in the system...)
• A new simulation environment has been added to sysstat to run non regression tests. This environment allows for fast, reproducible tests. With the addition of hundreds of tests, introduction of regressions in new sysstat versions is now made more unlikely...

Sysstat 12.0 stable series - Why should I upgrade from sysstat 11.6.x to 12.0.x?

Sysstat 12.0 includes major changes concerning sar’s binary data file format and how sar works. Among them:

• A binary data file should take much less space on disk thanks to new optimizations on how values are stowed in memory. The space saved is estimated between 25% and 45% compared to previous format.
• Structures are no longer statically allocated, meaning that the system can now register as many new devices (disks, network interfaces, etc.) as needed and you will find all of them saved in your saDD data file (provided that you have selected the corresponding activities to collect). Previous version could lead to some devices being ignored and going unnoticed if no free structures were left.
• Sar (and sadf) will now be able to read a binary data file whatever its endianness is: Both big-endian and little-endian files can be read by the same sar or sadf executable.
• More flexibility has been added to sar’s binary data file format which should now make it possible to add new metrics or activities without making the format unreadable by older sar versions (starting with version 11.7.1).

Other changes include:

• Support for offline/online CPU has been improved (for sar and mpstat). Also sar and sadf no longer display offline CPU.
• A new option (-z) has been added to sar to omit output for any devices for which there was no activity during the sample period.
• Another new option (-h) has been added to sar to make output easier to read by a human.
• Options have been added to make it possible to select network interfaces (--iface=), block devices (--dev=) and filesystems (--fs=) for which statistics shall be displayed by sar or sadf.
• A new option (showtoc) can now be used with sadf SVG output. This option displays, at the beginning of the SVG file, the list of activities for which there are graphs (table of contents).
• The user can now select each individual activity that will be collected by sadc.
• It is now possible to select the number of decimal places (in the range 0-2) used by every sysstat command.
• National Language Support has been improved, with a new Korean translation added.

Sysstat 11.6 stable series - Why should I upgrade from sysstat 11.4.x to 11.6.x?

• sar can now display software-based network processing statistics (softnet). New metrics have also been added for other activities, e.g. 'available free memory" with memory statistics.
• A new option (--human) can be used with various sysstat commands to make them display values in human readable format.
• New options can be used with sadf -g to enhance graphs output: Option showidle can be used to display CPU idle state in SVG graphs ; Option packed tells sadf to allow multiple graphs on a row ; Option height=... allows the user to explicitly specify canvas height ; Option showinfo displays extra information such as the date and the host name.
• A new output format has been added to sadf to display the raw contents of sar binary data files.
• The iostat and mpstat commands can now display their output in JSON format.
• A range of CPU or interrupts can now be selected with sar and mpstat commands.
• The pidstat command has been improved: It now displays %wait metric as part of CPU utilization statistics. It also has a new switch (-e) which can be used to pass a program to execute and make pidstat monitor it. Last you can choose to display the timestamps in seconds since the Epoch with its new -H switch.
• The mpstat command can provide CPU statistics based on NUMA node placement.
• The iostat command has a new switch to display a short (narrow) version of its reports that should fit in 80 chars wide screens. New metrics have also been added to its extended statistics report.
• National Language Support has been improved, with a new Friulian translation.

Sysstat 11.4 stable series - Why should I upgrade from sysstat 11.2.x to 11.4.x?

• Sysstat 11.4 includes SVG (Scalable Vector Graphics) support for sar/sadf. It is now possible to draw graphs for (almost) every sar activity using a maintained tool (sadf) which is part of sysstat package. Extra options (skipempty, autoscale, oneday) may be passed to sadf to control SVG output.

Sysstat 11.2 stable series - Why should I upgrade from sysstat 11.0.x to 11.2.x?

• A new command (tapestat) has been added to display statistics for tape drives.
• A new option (-c) has been added to sadf to enable the user to convert an old system activity binary data file (sysstat 9.1.6 at least) to the up-to-date format.
• sar has now support for Fibre Channel HBA statistics. New metrics have also been added to memory statistics.
• All sysstat commands have now smart color output support for easier statistics reading. Colors can be customized by the user.
• The obsolete nfsiostat command has been removed (such a command already exists in the nfs-utils package).
• sysstat is now regularly submitted for static analysis provided by Coverity Scan. As a result, numerous defects have been fixed and sysstat has now a defect density (number of defects per 1000 lines of code) lower than 0.1 as of this writing.

Sysstat 11.0 stable series - Why should I upgrade from sysstat 10.2.x to 11.0.x?

• sar, sadc and sadf commands can now take into account a change of CPU count in binary data files without losing data from current day.
• The standard system activity daily data files used by sar, sadc and sadf may now be named saYYYYMMDD instead of saDD, where YYYY stands for the current year, MM for the current month and DD for the current day.
• It has been made easier for sar, sadc and sadf to take into account alternate locations for standard system activity daily data files (the default location remains /var/log/sa).
• systemd timer units can now be used instead of cron jobs to launch sadc.
• pidstat can now display task scheduling priority and policy information (option -R).
• pidstat can now display only processes whose name matches a regular expression (option -G). All threads of matching processes are displayed.
• National Language Support improved: Added Galician and Hungarian translations.