---

Linux Support for NUMA Hardware

Large count multiprocessors are being built with non-uniform memory access (NUMA) times - access times that are  dependent upon where within the machine a piece of memory physically resides.   For optimal performance, the kernel needs to be aware of where memory is located, and keep memory used as close as possible to the user of the memory.  Examples of NUMA machines include the NEC Azusa, the IBM x440 and the IBM NUMAQ.

The 2.5 Linux kernel includes many enhancements in support of NUMA machines.  Data structures and macros are provided within the kernel for determining the layout of the memory and processors on the system.  These enable the VM subsystem to make decisions on the optimal placement of memory for processes.  This topology information is also exported to user-space via sysfs. 

In addition to items that have been incorported into the 2.5 Linux kernel, there are NUMA features that have been developed that continue to be supported as patchsets.  These include NUMA enhancements to the scheduler, multipath I/O and a user-level API that provides user control over the allocation of resources in respect to NUMA nodes.

This page provides links to information about the various Linux on NUMA projects.  Discussions related to Linux on NUMA take place on the lse-tech mailing list and on the linux kernel mailing list.  

Introduction to NUMA:

• Demystify NUMA via the FAQ
• Current NUMA status of Linux

Linux-on-NUMA Projects:

• NUMA-aware scheduler:

•  Discontigmem:

• NUMA-aware locking primitives

• NUMA related APIs:  

  To facilitate user space programs to optimize for NUMA configurations, APIs are necessary to export topology information, and to provide user specification of processor and memory resources to be used.  There are also internal kernel APIs to provide NUMA topology information for use by kernel subsystems.  The following links provide information on various NUMA APIs, some of which are accepted in the mainline kernel, others which exist as independent patchsets, and others which are in the planning/proposal stage.

  • In-Kernel Topology API - Included: 2.5.40
  • Sysfs Topology API - Included: 2.5.46
  • Memory Binding API - Posted to LKML for discussion
  • setaffinity - Included: early 2.5 -  developed for general SMP, but necessary for NUMA resource binding
    
  • launch policies
  • Process Scheduling and Memory Placement Design Notes a.k.a. CpuMemSets

• Multi-path I/O:

Linux-on-NUMA Related Resources:

• [email protected] is a great place for all kinds of discussion regarding the Linux Scalability Effort as a whole.
• NUMA-related email archive
• Linux-kernel mailing list and its archives also contain some valuable information. Check out the lkmlFAQ.

Linux-on-NUMA Meeting Links:

• March 26th Linux-on-NUMA Meeting
• July 25th Linux-on-NUMA Working Group at OLS
• 2.5 Kernel Developers Summit NUMA BOF

NUMA Links by Corporation:

• IBM
• SGI
• Fujitsu
• Sequent
  

Last updated: 11/20/02 Any problems, additions, etc., please send email to this page's maintainer.