Introduction¶

“CPU Isolation” means leaving a CPU exclusive to a given workload without any undesired code interference from the kernel.

Those interferences, commonly pointed out as “noise”, can be triggered by asynchronous events (interrupts, timers, scheduler preemption by workqueues and kthreads, ...) or synchronous events (syscalls and page faults).

Such noise usually goes unnoticed. After all, synchronous events are a component of the requested kernel service. And asynchronous events are either sufficiently well-distributed by the scheduler when executed as tasks or reasonably fast when executed as interrupt. The timer interrupt can even execute 1024 times per seconds without a significant and measurable impact most of the time.

However some rare and extreme workloads can be quite sensitive to those kinds of noise. This is the case, for example, with high bandwidth network processing that can’t afford losing a single packet or very low latency network processing. Typically those use cases involve DPDK, bypassing the kernel networking stack and performing direct access to the networking device from userspace.

In order to run a CPU without or with limited kernel noise, the related housekeeping work needs to be either shut down, migrated or offloaded.

Housekeeping¶

In the CPU isolation terminology, housekeeping is the work, often asynchronous, that the kernel needs to process in order to maintain all its services. It matches the noises and disturbances enumerated above except when at least one CPU is isolated. Then housekeeping may make use of further coping mechanisms if CPU-tied work must be offloaded.

Housekeeping CPUs are the non-isolated CPUs where the kernel noise is moved away from isolated CPUs.

The isolation can be implemented in several ways depending on the nature of the noise:

• Unbound work, where “unbound” means not tied to any CPU, can be simply migrated away from isolated CPUs to housekeeping CPUs. This is the case of unbound workqueues, kthreads and timers.

• Bound work, where “bound” means tied to a specific CPU, usually can’t be moved away as-is by nature. Either:

  • The work must switch to a locked implementation. E.g.: This is the case of RCU with CONFIG_RCU_NOCB_CPU.

  • The related feature must be shut down and considered incompatible with isolated CPUs. E.g.: Lockup watchdog, unreliable clocksources, etc...

  • An elaborate and heavyweight coping mechanism stands as a replacement. E.g.: the timer tick is shut down on nohz_full CPUs but with the constraint of running a single task on them. A significant cost penalty is added on kernel entry/exit and a residual 1Hz scheduler tick is offloaded to housekeeping CPUs.

In any case, housekeeping work has to be handled, which is why there must be at least one housekeeping CPU in the system, preferably more if the machine runs a lot of CPUs. For example one per node on NUMA systems.

Also CPU isolation often means a tradeoff between noise-free isolated CPUs and added overhead on housekeeping CPUs, sometimes even on isolated CPUs entering the kernel.

Isolation features¶

Different levels of isolation can be configured in the kernel, each of which has its own drawbacks and tradeoffs.

Checklist¶

You have set up each of the above isolation features but you still observe jitters that trash your workload? Make sure to check a few elements before proceeding.

Some of these checklist items are similar to those of real-time workloads:

• Use mlock() to prevent your pages from being swapped away. Page faults are usually not compatible with jitter sensitive workloads.

• Avoid SMT to prevent your hardware thread from being “preempted” by another one.

• CPU frequency changes may induce subtle sorts of jitter in a workload. Cpufreq should be used and tuned with caution.

• Deep C-states may result in latency issues upon wake-up. If this happens to be a problem, C-states can be limited via kernel boot parameters such as processor.max_cstate or intel_idle.max_cstate. More finegrained tunings are described in CPU Idle Time Management page

• Your system may be subject to firmware-originating interrupts - x86 has System Management Interrupts (SMIs) for example. Check your system BIOS to disable such interference, and with some luck your vendor will have a BIOS tuning guidance for low-latency operations.

Full isolation example¶

In this example, the system has 8 CPUs and the 8th is to be fully isolated. Since CPUs start from 0, the 8th CPU is CPU 7.

cd /sys/fs/cgroup
# Activate the cpuset subsystem
echo +cpuset > cgroup.subtree_control
# Create partition to be isolated
mkdir test
cd test
echo +cpuset > cgroup.subtree_control
# Isolate CPU 7
echo 7 > cpuset.cpus
echo "isolated" > cpuset.cpus.partition

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
int main(void)
{
    // Move the current task to the isolated cpuset (bind to CPU 7)
    int fd = open("/sys/fs/cgroup/test/cgroup.procs", O_WRONLY);
    if (fd < 0) {
        perror("Can't open cpuset file...\n");
        return 0;
    }

    write(fd, "0\n", 2);
    close(fd);

    // Run an endless dummy loop until the launcher kills us
    while (1)
    ;

    return 0;
}

# gcc user_loop.c -o user_loop

TRACING=/sys/kernel/tracing/
# Make sure tracing is off for now
echo 0 > $TRACING/tracing_on
# Flush previous traces
echo > $TRACING/trace
# Record disturbance from other tasks
echo 1 > $TRACING/events/sched/sched_switch/enable
# Record disturbance from interrupts
echo 1 > $TRACING/events/irq_vectors/enable
# Now we can start tracing
echo 1 > $TRACING/tracing_on
# Run the dummy user_loop for 10 seconds on CPU 7
./user_loop &
USER_LOOP_PID=$!
sleep 10
kill $USER_LOOP_PID
# Disable tracing and save traces from CPU 7 in a file
echo 0 > $TRACING/tracing_on
cat $TRACING/per_cpu/cpu7/trace > trace.7

<idle>-0 [007] d..2. 1980.976624: sched_switch: prev_comm=swapper/7 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=user_loop next_pid=1553 next_prio=120
user_loop-1553 [007] d.h.. 1990.946593: reschedule_entry: vector=253
user_loop-1553 [007] d.h.. 1990.946593: reschedule_exit: vector=253

Debugging¶

Of course things are never so easy, especially on this matter. Chances are that actual noise will be observed in the aforementioned trace.7 file.

The best way to investigate further is to enable finer grained tracepoints such as those of subsystems producing asynchronous events: workqueue, timer, irq_vector, etc... It also can be interesting to enable the tick_stop event to diagnose why the tick is retained when that happens.

Some tools may also be useful for higher level analysis:

• rtla provides a suite of tools to analyze latency and noise in the system. For example rtla-osnoise runs a kernel tracer that analyzes and output a summary of the noises.

• cpunoise does something similar to rtla-osnoise but in userspace. It is available at https://git.kernel.org/pub/scm/linux/kernel/git/frederic/cpunoise.git