Performance Co-Pilot (PCP) is a robust framework for collecting, monitoring, and analyzing system performance metrics. Available in the repos for Fedora and RHEL, it allows administrators to gather a wide array of data with minimal configuration. This guide walks you through tuning PCP’s pmlogger service to better fit your needs—whether you’re debugging performance issues or running on constrained hardware.

Is the default setup of PCP right for your use case? Often, it’s not. While PCP’s defaults strike a balance between data granularity and overhead, production workloads vary widely. Later in this article, two scenarios will be used to demonstrate some useful configurations.

Prerequisites: Getting PCP up and running

First, install the PCP packages:

$ sudo dnf install pcp pcp-system-tools

Then enable and start the core services:

$ sudo systemctl enable --now pmcd.service
$ sudo systemctl enable --now pmlogger.service

Verify both services are running:

$ systemctl status pmcd pmlogger

Understanding pmlogger and its configuration

PCP consists of two main components:

• pmcd: collects live performance metrics from various agents.
• pmlogger: archives these metrics over time for analysis.

The behavior of pmlogger is controlled by files in /etc/pcp/pmlogger/control.d/. The most relevant is local, which contains command-line options for how logging should behave.

Sample configuration:

$ cat /etc/pcp/pmlogger/control.d/local

You’ll see a line like:

localhost y y /usr/bin/pmlogger -h localhost ... -t 10s -m note

The -t 10s flag defines the logging interval—every 10 seconds in this case.

Scenario 1: High-frequency monitoring for deep analysis

Use case: Debugging a transient issue on a production server.
Goal: Change the logging interval from 10 seconds to 1 second.

Edit the file (nano editor used in the examples, please use your editor of choice):

$ sudo nano /etc/pcp/pmlogger/control.d/local

Change -t 10s to -t 1s.

Restart the logger:

$ sudo systemctl restart pmlogger.service

Verify:

$ ps aux | grep '[p]mlogger -h localhost'
$ pminfo -f

Expected output snippet:

records: 10, interval: 0:00:01.000

Scenario 2: Lightweight monitoring for constrained systems

Use case: Monitoring on a small VM or IoT device.
Goal: Change the logging interval to once every 60 seconds.

Edit the same file:

$ sudo nano /etc/pcp/pmlogger/control.d/local

Change -t 10s to -t 60s.

Restart the logger:

$ sudo systemctl restart pmlogger.service

Confirm:

$ ps aux | grep '[p]mlogger -h localhost'
$ pminfo -f

Expected output:

records: 3, interval: 0:01:00.000

Managing data retention: logs, size, and cleanup

PCP archives are rotated daily by a cron-like service. Configuration lives in:

$ cat /etc/sysconfig/pmlogger

Default values:

PCP_MAX_LOG_SIZE=100
PCP_MAX_LOG_VERSIONS=14

• PCP_MAX_LOG_SIZE: total archive size (in MB).
• PCP_MAX_LOG_VERSIONS: number of daily logs to keep.

Goal: Keep logs for 30 days.

Edit the file:

$ sudo nano /etc/sysconfig/pmlogger

Change:

PCP_MAX_LOG_VERSIONS=30

No service restart is required. Changes apply during the next cleanup cycle.

Final thoughts

PCP is a flexible powerhouse. With just a few changes, you can transform it from a general-purpose monitor into a specialized tool tailored to your workload. Whether you need precision diagnostics or long-term resource tracking, tuning pmlogger gives you control and confidence.

So go ahead—open that config file and start customizing your system’s performance story.

Note: This article is dedicated to my wife, Rupali Suraj Patil, who inspires me every day.

This article will describe the content and structure of the sosreport output. The aim is to improve its usefullness through a better understanding of its contents.

What is sosreport?

sosreport is a powerful command-line utility available on Fedora, Red Hat Enterprise Linux (RHEL), CentOS, and other RHEL-based systems to collect a comprehensive snapshot of the system’s configuration, logs, services, and state. The primary use is for diagnosing issues, especially during support cases with Red Hat or other vendors.

When executed, sosreport runs a series of modular plugins that collect relevant data from various subsystems like networking, storage, SELinux, Docker, and more. The resulting report is packaged into a compressed tarball, which can be securely shared with support teams to expedite troubleshooting.

In essence, sosreport acts as a black box recorder for Linux — capturing everything from system logs and kernel messages to active configurations and command outputs — helping support engineers trace problems without needing direct access to the system.

How to Generate a sosreport

To use sosreport on Fedora, RHEL, or CentOS, run the following command as root or with sudo:

sudo sosreport

This command collects system configuration, logs, and command outputs using various plugins. After a few minutes, it generates a compressed tarball in /var/tmp/ (or a similar location), typically named like:

sosreport-hostname-20250623-123456.tar.xz

You may be prompted to enter a case ID or other metadata, depending on your system configuration or support workflow.

The sosreport generated tarball contains a detailed snapshot of the system’s health and configuration. It has a well-organized structure which reflects the data collected from the myriad Linux subsystems.

Exploring sosreport output is challenging due to the sheer volume of logs, configuration files, and system command outputs it contains. However, understanding its layout is key for support engineers and sysadmins to quickly locate and interpret crucial diagnostic information.

sosreport directory layout

When the tarball is unpacked, the directory structure typically resembles this:

.
├── ./boot
├── ./etc
├── ./lib -> usr/lib
├── ./opt
├── ./proc
├── ./run
├── ./sos_commands
├── ./sos_logs
├── ./sos_reports
├── ./sos_strings
├── ./sys
├── ./usr
├── ./var
└── ./EXTRAS

CORE Breakdown:

• Most directories mimic a standard Linux root filesystem and primarily contain configuration files.
• The directories that don’t appear in a regular root filesystem include:
  • sos_command
  • sos_logs
  • sos_reports
  • sos_strings
  • EXTRAS

Key directories in detail

sos_commands/

This contains output from commands executed by each plugin. Its structure is plugin-specific:

./sos_commands/
├── apparmor/
├── docker/
├── memory/
├── networkmanager/
├── process/
│ ├── lsof_M_-n_-l_-c
│ ├── pidstat_-p_ALL_-rudvwsRU_--human_-h
│ ├── ps_auxwwwm
│ └── pstree_-lp

Each file name matches the Linux command used, with all options. The contents are the actual command output, making the plugin behavior transparent.

sos_reports/

This directory contains multiple formats that index and summarize the entire sosreport:

• sos.json: A machine-readable index of all collected files and commands.
• manifest.json: Describes how sosreport executed – timestamps, plugins used, obfuscation done, errors, etc.
• HTML output for easy browsing via browser.

sos_logs/

Contains logs from the execution of sosreport itself.

• sos.log: Primary log file that highlights any errors or issues during data collection.

sos_strings/

• Contains journal logs for up to 30 days, extracted using journalctl
• Can be quite large, especially on heavily used systems
• Structured into subdirectories like logs/ or networkmanager/

EXTRAS/

This is not a default part of an sosreport. It is created by the sos_extras plugin and used to collect any custom user-defined files.

Why this layout matters

• Speed: Logical grouping of directories help engineers drill down without manually parsing GigaBytes of log files.
• Traceability: Knowing where each file came from and what command produced it enhances reproducibility.
• Automation: Tools like soscleaner or sos-analyzer rely on this structure for automated diagnostics.

Final thoughts

While sosreport is a powerful diagnostic tool, its effectiveness hinges on understanding its structure. With familiarity, engineers can isolate root causes of failures, uncover misconfigurations, and collaborate more efficiently with support teams. If you haven’t yet opened one up manually, try it — there’s a lot to learn from the insides!

This is my first Fedora Magazine article, dedicated to my wife Rupali Suraj Patil — my constant source of inspiration.

In this fifth article of the “System insights with command-line tools” series we explore free and vmstat, two small utilities that reveal a surprising amount about your Linux system’s health. free gives you an instant snapshot of how RAM and swap are being used. vmstat (the virtual memory statistics reporter) reports a real-time view of memory, CPU, and I/O activity.

By the end of this article you will be able to translate buffers and cache into “breathing room”, read the mysterious available column with confidence, and spot memory leaks or I/O saturation.

A quick tour of free

Basic usage

$ free -h
       total    used    free   shared  buff/cache  available
Mem:    23Gi    14Gi   575Mi    3,3Gi        12Gi      8,8Gi
Swap:  8,0Gi   6,6Gi   1,4Gi

free parses /proc/meminfo and prints totals for physical memory and swap, along with kernel buffers and cache. Use -h for human-readable units, -s 1 to refresh every second, and -c N to stop after N samples which is handy to get a trend when doing something in parallel. For example, free -s 60 -c 1440 gives a 24-hour CSV-friendly record without installing extra monitoring daemons.

Free memory refers to RAM that is entirely unoccupied. It isn’t being used by any process or for caching. On server systems, I tend to view this as wasted since unused memory isn’t contributing to performance. Ideally, after a system has been running for some time, this number should remain low.

Available memory, on the other hand, represents an estimate of how much memory can be used by new or running processes without resorting to swap. It includes free memory plus parts of the cache and buffers that the system can reclaim quickly if needed.

In essence, the distinction in Linux lies here: free memory is idle and unused, while available memory includes both truly free space and memory that can be readily freed up to keep the system responsive without swapping. It is not a problem to have a low free memory, available memory is usually what to be concerned about.

A healthy system might even show used ≈ total yet available remains large; that mostly reflects cache at work. Fedora’s kernel will automatically drop clean cache pages whenever an application needs the space, so cached memory is not wasted. Think of it as a working set that just hasn’t been reassigned yet.

Spotting problems with free

• Rapidly shrinking available combined with rising swap used indicates real memory pressure.
• Large swap-in/out spikes point to thrashing workloads or runaway memory consumers.

vmstat – Report virtual memory statistics

vmstat (virtual memory statistics) displays processes, memory, paging, block-I/O, interrupts, context switches, and CPU utilization in a single line. Run it with an interval and count to watch trends (output shown below has been split into three sections for better readability):

$ vmstat 1 3
procs -----------memory----------     
 r  b   swpd   free   buff  cache     
 2  0 7102404 1392528     36 12335148 
 0  0 7102404 1392560     36 12335188 
 0  0 7102404 1373640     36 12349928 

 ---swap-- -----io---- 
  si   so    bi    bo  
   8   21   130   724  
   0    0     0     0  
   0    0     8    48  

 -system-- -------cpu-------
 in     cs us sy id wa st gu
 2851   19 15  7 77  0  0  0
 5779 7246 14 10 77  0  0  0
 5141 6525 12  9 79  0  0  0

Anatomy of the output

From the vmstat(8) manpage:

Procs
    r: The number of runnable processes (running or waiting
       for run time).
    b: The number of processes blocked waiting for I/O to
       complete.

Memory
    These are affected by the --unit option.
    swpd: the amount of swap memory used.
    free: the amount of idle memory.
    buff: the amount of memory used as buffers.
    cache: the amount of memory used as cache.
    inact: the amount of inactive memory.  (-a option)
    active: the amount of active memory.  (-a option)

Swap
    These are affected by the --unit option.
    si: Amount of memory swapped in from disk (/s).
    so: Amount of memory swapped to disk (/s).

IO
    bi: Kibibyte received from a block device (KiB/s).
    bo: Kibibyte sent to a block device (KiB/s).

System
    in: The number of interrupts per second, including
        the clock.
    cs: The number of context switches per second.

CPU
    These are percentages of total CPU time.
    us: Time spent running non-kernel code.  (user time,
        including nice time)
    sy: Time spent running kernel code.  (system time)
    id: Time spent idle.  Prior to Linux 2.5.41, this
        includes IO-wait time.
    wa: Time spent waiting for IO.  Prior to Linux 2.5.41,
        included in idle.
    st: Time stolen from a virtual machine.  Prior to
        Linux 2.6.11, unknown.
    gu: Time spent running KVM guest code (guest time,
        including guest nice).

Practical diagnostics

Catching a memory leak

Run vmstat 500 in one terminal while your suspect application runs in another. If free keeps falling and si/so climb over successive samples, physical RAM is being exhausted and the kernel starts swapping, which is classic leak behavior.

Finding I/O saturation

When wa (CPU wait) and bo (blocks out) soar while r remains modest, the CPU is idle but stuck waiting for the disk. Consider adding faster storage or tuning I/O scheduler parameters.

Detecting CPU over-commit

A sustained r that is double the number of logical cores with low wa and plenty of free means CPU is the bottleneck, not memory or I/O. Use top or htop to locate the busiest processes, or scale out workloads accordingly.

Conclusion

Mastering free and vmstat gives you a lens into memory usage, swap activity, I/O latency, and CPU load. For everyday debugging: start with free to check if your system is truly out of memory, then use vmstat to reveal the reason, whether it’s memory leaks, disk bottlenecks, or CPU saturation.

Stay tuned for the next piece in our “System insights with command-line tools” series and happy Fedora troubleshooting!