Cpuset Library and Linux Kernel Support

   This document describes the 'C' library libcpuset interface to Linux cpusets.

   Cpusets provide system-wide control of the CPUs on which tasks may execute, and
   the memory nodes on which they allocate memory. Each cpuset defines a list of
   allowed CPUs and memory nodes, and each process in the system is attached to a
   cpuset. Cpusets are represented in a hierarchical virtual file system. Cpusets can
   be nested and they have file-like permissions.

   The efficient administration of large multi-processor systems depends on
   dynamically allocating portions of the systems CPU and memory resources to
   different users and purposes. The optimum performance of NUMA systems depends
   optimizing CPU and memory placement of critical applications, and minimizing
   interference between applications. Cpusets provides a convenient means to control
   such CPU and memory placement and usage.
    Author:   Paul Jackson
    Address:  pj@sgi.com
     Date:    14 November 2006
   Copyright: Copyright (c) 2006-2007 SGI. All Rights Reserved.

   Permission is hereby granted, free of charge, to any person obtaining a copy of
   this Documentation, to deal in the Documentation without restriction, including
   without limitation the rights to use, copy, modify, merge, publish, distribute,
   sublicense, and/or sell copies of the Documentation, and to permit persons to whom
   the Documentation is furnished to do so, subject to the following conditions:

   The above copyright notice and this permission notice shall be included in all
   copies or substantial portions of the Documentation.

   THE DOCUMENTATION IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
   IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
   FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL SILICON GRAPHICS,
   INC. BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
   CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
   DOCUMENTATION OR THE USE OR OTHER DEALINGS IN THE DOCUMENTATION.

   Except as contained in this notice, the names of Silicon Graphics and SGI shall
   not be used in advertising or otherwise to promote the sale, use or other dealings
   in this Documentation without prior written authorization from SGI.
     _____________________________________________________________________________

   This document is written using the outline processor Leo, and version
   controlled using CSSC. It is rendered using Python Docutils on
   reStructuredText extracted from Leo, directly into both html and
   LaTeX. The LaTeX is converted into pdf using the pdflatex utility.
   The html is converted into plain text using the lynx utility.
     _____________________________________________________________________________

   Silicon Graphics and SGI are registered trademarks of Silicon Graphics, Inc., in
   the United States and other countries worldwide. Linux is a registered trademark
   of Linus Torvalds in several countries. Novell is a registered trademark, and SUSE
   is a trademark of Novell, Inc. in the United States and other countries. All other
   trademarks mentioned herein are the property of their respective owners.

   Table of Contents
     * 1   Why Cpusets?
     * 2   Linux Cpuset Kernel Support
     * 3   Using Cpusets at the Shell Prompt
     * 4   Cpuset Programming Model
     * 5   CPUs and Memory Nodes
     * 6   Extensible API
     * 7   Cpuset Text Format
     * 8   Basic Cpuset Library Functions
     * 9   Using Cpusets with Hyper-Threads
     * 10   Advanced Cpuset Library Functions
     * 11   System Error Numbers
     * 12   Change History

1   Why Cpusets?

   The essential purpose of cpusets is to provide CPU and memory containers or "soft
   partitions" within which to run sets of related tasks.

   On an SMP (multiple CPU) system without some means of CPU placement, any task can
   run on any CPU. On a NUMA (multiple memory node) system, any memory page can be
   allocated on any node. This can cause both poor cache locality and poor memory
   access times, substantially reducing performance and run-time repeatability. By
   restraining all other jobs from using any of the CPUs or memory nodes assigned to
   critical jobs, interference with critical jobs can be minimized.

   For example, some multi-threaded high performance computing (HPC) jobs consist of
   a number of threads that communicate via message passing interfaces (MPI) and
   other such jobs rely on multi-platform shared-memory parallel programming (OpenMP)
   that can tightly couple parallel computation threads using special language
   directives. It is common that threads in such jobs need to be executing at the
   same time to make optimum progress. In such cases, if a single thread loses a CPU,
   all threads stop making forward progress and spin at a barrier. Cpusets can
   eliminate the need for a gang scheduler, provide isolation of one such job from
   other tasks on a system, and facilitate providing equal resources to each thread
   in such a job. This results in both optimum and repeatable performance.

   This document focuses on the 'C' API provided by the user level libcpuset library.
   This library depends on the following Linux 2.6 kernel facilities:

     * sched_setaffinity (for CPU binding),
     * mbind and set_mempolicy (for memory node binding), and
     * kernel cpusets support.

   The sched_setaffinity, mbind and set_mempolicy calls enable specifying the CPU and
   memory placement for individual tasks. On smaller or limited use systems, these
   calls may be sufficient.

   The kernel cpuset facility provides additional support for system wide management
   of CPU and memory resources, by related sets of tasks. It provides a hierarchical
   structure to the resources, with file system like name-space and permissions, and
   support for guaranteed exclusive use of resources.

   The Linux kernel provides the following support for cpusets:

     * Each task has a link to a cpuset structure that specifies the CPUs and memory
       nodes available for its use.
     * A hook in the sched_setaffinity and mbind system calls ensures that any
       requested CPU or memory node is available in that tasks cpuset.
     * Tasks sharing the same placement constraints reference the same cpuset.
     * These kernel cpusets are arranged in a hierarchical virtual file system,
       reflecting the possible nesting of "soft partitions".
     * The kernel task scheduler is constrained to only schedule a task on the CPUs
       in that task's cpuset.
     * The kernel memory allocator is constrained to only allocate physical memory to
       a task from the memory nodes in that tasks cpuset.
     * The kernel memory allocator provides an economical per-cpuset metric of the
       aggregate memory pressure (frequency of requests for a free memory page not
       easily satisfied by an available free page) of the tasks in a cpuset (see the
       per-cpuset 'memory_pressure' file.)
     * The kernel memory allocator provides the option to request that memory pages
       used for file I/O (the kernel page cache) and associated kernel data
       structures for file inodes and directories be evenly spread across all the
       memory nodes in a cpuset, rather than preferentially allocated on whatever
       memory node the task that first accessed the page was first running (see the
       per-cpuset 'memory_spread_page' and 'memory_spread_slab' files).
     * The memory migration facility in the kernel can be controlled using per-cpuset
       files, so that when the memory nodes allowed to a task by cpusets changes, any
       pages it had on no longer allowed nodes are migrated to nodes now allowed.

   A cpuset constrains the jobs (set of related tasks) running in it to a subset of
   the systems memory and CPUs. They enable administrators and system service
   software to:

     * Create and delete named cpusets.
     * Decide which CPUs and memory nodes are available to a cpuset.
     * Attach a task to a particular cpuset.
     * Identify all tasks sharing the same cpuset.
     * Exclude any other cpuset from overlapping a given cpuset, giving the tasks
       running in that cpuset exclusive use of those CPUs and memory nodes.
     * Perform bulk operations on all tasks associated with a cpuset, such as varying
       the resources available to that cpuset, or hibernating those tasks in
       temporary favor of some other job.
     * Perform sub-partitioning with hierarchical permissions and resource
       management.

   Cpusets are exposed by the kernel to user space by mounting the cpuset virtual
   file system (VFS) at /dev/cpuset, rather than by additional system calls. Such a
   VFS is a natural way to represent nested resource allocations and the associated
   hierarchical permission model.

   Within a single cpuset, other facilities such as dplace, first-touch memory
   placement, pthreads, sched_setaffinity and mbind can be used to manage processor
   and memory placement to a more fine-grained level.

   There is a single set of kernel mechanisms that supports all these facilities and
   provides a consistent processor and memory model regardless of what mix of
   utilities and API's you use to manage it. This provides a consistent execution
   model for all users.

2   Linux Cpuset Kernel Support

   Several developers in the Open Source community provided the Linux 2.6 kernel
   support for CPU and memory placement, including the following:

     * Robert Love - Tech9, Novell (USA) - Scheduler attributes such as CPU affinity
     * Simon Derr - Bull (France) - CPU placement, core architecture and file system
       interface to cpusets
     * Andi Kleen - SUSE (Germany) - NUMA memory placement, mbind and mempolicy
     * Paul Jackson - SGI (USA) - kernel bitmask improvements, kernel cpuset
       integration, libbitmask, libcpuset

   At least a couple of command line utilities have been developed that use these
   affinity calls to allow placing a process on a specific CPU. Robert Love has a
   package schedutils with a command taskset. The numactl command in Andi Kleen's
   work has options to run a specified command on specified CPUs and memory nodes.

   Andi Kleen led a session at the 2003 Kernel Summit in Ottawa in NUMA memory
   management, and has been developing a NUMA library and kernel support for memory
   placement. Minutes from that session are available at:
   http://lwn.net/Articles/40626/.

   The cpuset kernel facility and file system for Linux 2.6 kernels is based on the
   work of Simon Derr, with integration and refinements by Paul Jackson. In addition
   this document of the libcpuset user library, the kernel cpuset facility is
   documented in the kernel source file Documentation/cpusets.txt, and Simon Derr has
   provided a document of cpusets at
   http://www.bullopensource.org/cpuset/.

   The user level bitmask library supports convenient manipulation of multi-word
   bitmasks useful for CPUs and memory nodes. This bitmask library is required by and
   designed to work with the cpuset library. The design notes of libbitmask are
   available in a separate document, Bitmask_Library.html or Bitmask_Library.pdf.

   Unlike sched_setaffinity() and mbind(), which are implemented as additional kernel
   system calls, the primary kernel interface for accessing the cpuset facility is
   The Cpuset File System, usually mounted at /dev/cpuset. The cpuset library
   libcpuset provides convenient access to these facilities from 'C' programs.

   Cpusets extend the usefulness of the Linux 2.6 kernel mechanisms
   sched_setaffinity() for CPU placement, and mbind() and set_mempolicy() for memory
   placement. On smaller or dedicated use systems, these other mechanisms are often
   sufficient. The libcpuset library provides a convenient API to these other
   mechanisms that has the added advantage of being robustly adapting to memory
   migration.

   On larger NUMA systems, running more than one, performance critical, job, it is
   necessary to be able to manage jobs in their entirety. This includes providing a
   job with exclusive CPU and memory that no other job can use and being able to list
   all tasks currently in a cpuset.

   You can use both these other placement mechanisms and cpusets together, using the
   Advanced Cpuset Library Functions to manage overall job placement, and using
   the other mechanisms, perhaps via the Basic Cpuset Library Functions within
   each given job to manage the details of thread and memory page placement.

2.1   The Cpuset File System

   Cpusets are named, nested sets of CPUs and memory nodes. Each cpuset is
   represented by a directory in the cpuset virtual file system, normally mounted at
   /dev/cpuset.

   New cpusets are created using the mkdir system call or shell command. The
   properties of a cpuset, such as its flags, allowed CPUs and memory nodes, and
   attached tasks, are queried and modified by reading or writing to the appropriate
   file in that cpusets directory.

   The state of each cpuset is represented by small text files in that cpusets
   directory. These files may be read and written using traditional shell utilities
   such as cat(1) and echo(1), or using ordinary file access routines from
   programmatic languages, such as open(2), read(2), write(2) and close(2) from the
   'C' library.

   These per-cpuset files represent internal kernel state and do not have any
   persistent image on disk. These files are automatically created when the cpuset is
   created, as a result of the mkdir invocation. It is not allowed to add or remove
   files from a cpuset directory.

   Each of these per-cpuset files is listed and described below:

   tasks:
          List of the process ID's (PIDs) of the tasks in that cpuset. The list is
          formatted as a series of ASCII decimal numbers, each followed by a newline.
          A task may be added to a cpuset (removing it from the cpuset previously
          containing it) by writing its PID to that cpusets tasks file (with or
          without a trailing newline.)

          Beware that only one PID may be written to the tasks file at a time. If a
          string is written that contains more than one PID, all but the first will
          be ignored.

   notify_on_release:
          Flag (0 or 1). If set (1), then that cpuset will receive special handling
          whenever its last using task and last child cpuset goes away. For more
          information, see the Notify On Release section, below.

   cpus:
          List of CPUs on which tasks in that cpuset are allowed to execute. See
          List Format below for a description of the format of cpus.

          The CPUs allowed to a cpuset may be changed by writing a new list to its
          cpus file. Note however, such a change does not take affect until the PIDs
          of the tasks in the cpuset are rewritten to the cpusets tasks file.

   cpu_exclusive:
          Flag (0 or 1). If set (1), then the cpuset has exclusive use of its CPUs
          (no sibling or cousin cpuset may overlap CPUs). By default this is off (0).
          Newly created cpusets also initially default this to off (0).

   mems:
          List of memory nodes on which tasks in that cpuset are allowed to allocate
          memory. See List Format below for a description of the format of mems.

   mem_exclusive:
          Flag (0 or 1). If set (1), then the cpuset has exclusive use of its memory
          nodes (no sibling or cousin may overlap). By default this is off (0). Newly
          created cpusets also initially default this to off (0).

   memory_migrate:
          Flag (0 or 1). If set (1), then memory migration is enabled. For more
          information, see the Memory Migration section, below.

   memory_pressure:
          A measure of how much memory pressure the tasks in this cpuset are causing.
          For more information, see the Memory Pressure section, below. Always has
          value zero (0) unless memory_pressure_enabled is enabled in the top cpuset.
          This file is read-only.

   memory_pressure_enabled:
          Flag (0 or 1). This file is only present in the root cpuset, normally
          /dev/cpuset. If set (1), then memory_pressure calculations are enabled for
          all cpusets in the system. For more information, see the Memory Pressure
          section, below.

   memory_spread_page:
          Flag (0 or 1). If set (1), then the kernel page cache (file system buffers)
          are uniformly spread across the cpuset. For more information, see the
          Memory Spread section, below.

   memory_spread_slab:
          Flag (0 or 1). If set (1), then the kernel slab caches for file i/o
          (directory and inode structures) are uniformly spread across the cpuset.
          For more information, see the Memory Spread section, below.

   In addition one new file type is added to the /proc file system:

   /proc/<pid>/cpuset:
          For each task (pid), list its cpuset path, relative to the root of the
          cpuset file system. This file is read-only.

   Finally, the two control fields actually used by the kernel scheduler and memory
   allocator to constrain scheduling and allocation to the allowed CPUs are exposed
   as two more fields in the status file of each task:

     /proc/<pid>/status:

   Cpus_allowed:
          bit vector of CPUs on which this task may be scheduled.

   Mems_allowed:
          bit vector of memory nodes on which this task may obtain memory.

   There are several reasons why a tasks Cpus_allowed and Mems_allowed values may
   differ from the cpus and mems that are allowed in its current cpuset, as follows:

    1. A task might use sched_setaffinity, mbind or set_mempolicy to restrain its
       placement to less than its cpuset.
    2. Various temporary changes to Cpus_allowed are done by kernel internal code.
    3. Attaching a task to a cpuset doesn't change its Mems_allowed until the next
       time that task needs kernel memory.
    4. Changing a cpusets cpus doesn't change the Cpus_allowed of the tasks attached
       to it until those tasks are reattached to that cpuset (to avoid a hook in the
       hotpath scheduler code in the kernel).
    5. If hotplug is used to remove all the CPUs, or all the memory nodes, in a
       cpuset, then the tasks attached to that cpuset will have their Cpus_allowed or
       Mems_allowed altered to the CPUs or memory nodes of the closest ancestor to
       that cpuset that is not empty.

   Beware of items D and E, above. Due to item D, user space action is required to
   update a tasks Cpus_allowed after changing its cpuset. Use the routine
   cpuset_reattach to perform this update after a changing the cpus allowed to a
   cpuset.

   Due to item E, the confines of a cpuset can be violated after a hotplug removal
   that empties a cpuset. To avoid having a cpuset without CPU or memory resources,
   update your system's cpuset configuration to reflect the new hardware
   configuration. The kernel prefers misplacing a task, over starving a task of
   essential compute resources.

   There is one other condition under which the confines of a cpuset may be violated.
   A few kernel critical internal memory allocation requests, marked GFP_ATOMIC, must
   be satisfied immediately. The kernel may drop some request or malfunction if one
   of these allocations fail. If such a request cannot be satisfied within the
   current tasks cpuset, then the kernel relaxes the cpuset, and looks for memory
   anywhere it can find it. It's better to violate the cpuset than stress the kernel.

   New cpusets are created using mkdir (at the shell or in C). Old ones are removed
   using rmdir. The above files are accessed using read(2) and write(2) system calls,
   or shell commands such as cat(1) and echo(1).

   The CPUs and memory nodes in a given cpuset are always a subset of its parent. The
   root cpuset has all possible CPUs and memory nodes in the system. A cpuset may be
   exclusive (cpu or memory) only if its parent is similarly exclusive.

   Each task has a pointer to a cpuset. Multiple tasks may reference the same cpuset.
   Requests by a task, using the sched_setaffinity(2) system call to include CPUs in
   its CPU affinity mask, and using the mbind(2) and set_mempolicy(2) system calls to
   include memory nodes in its memory policy, are both filtered through that tasks
   cpuset, filtering out any CPUs or memory nodes not in that cpuset. The scheduler
   will not schedule a task on a CPU that is not allowed in its cpus_allowed vector,
   and the kernel page allocator will not allocate a page on a node that is not
   allowed in the requesting tasks mems_allowed vector.

   If a cpuset is cpu or mem exclusive, no other cpuset, other than a direct ancestor
   or descendant, may share any of the same CPUs or memory nodes.

   User level code may create and destroy cpusets by name in the cpuset virtual file
   system, manage the attributes and permissions of these cpusets and which CPUs and
   memory nodes are assigned to each cpuset, specify and query to which cpuset a task
   is assigned, and list the task pids assigned to a cpuset.

   Cpuset names are limited in length by the kernel's VFS implementation. No single
   component of a cpuset name may exceed 255 characters, and the full pathname of a
   cpuset including the /dev/cpuset mount point may not exceed 4095 characters in
   length.

2.2   Exclusive Cpusets

   If a cpuset is marked cpu_exclusive or mem_exclusive, no other cpuset, other than
   a direct ancestor or descendant, may share any of the same CPUs or memory nodes.

   A cpuset that is cpu_exclusive has a scheduler (sched) domain associated with it.
   The sched domain consists of all CPUs in the current cpuset that are not part of
   any exclusive child cpusets. This ensures that the scheduler load balancing code
   only balances against the CPUs that are in the sched domain as defined above and
   not all of the CPUs in the system. This removes any overhead due to load balancing
   code trying to pull tasks outside of the cpu_exclusive cpuset only to be prevented
   by the Cpus_allowed mask of the task.

   A cpuset that is mem_exclusive restricts kernel allocations for page, buffer, and
   other data commonly shared by the kernel across multiple users. All cpusets,
   whether mem_exclusive or not, restrict allocations of memory for user space. This
   enables configuring a system so that several independent jobs can share common
   kernel data, such as file system pages, while isolating each jobs user allocation
   in its own cpuset. To do this, construct a large mem_exclusive cpuset to hold all
   the jobs, and construct child, non-mem_exclusive cpusets for each individual job.
   Only a small amount of typical kernel memory, such as requests from interrupt
   handlers, is allowed to be taken outside even a mem_exclusive cpuset.

2.3   Notify On Release

   If the notify_on_release flag is enabled (1) in a cpuset, then whenever the last
   task in the cpuset leaves (exits or attaches to some other cpuset) and the last
   child cpuset of that cpuset is removed, the kernel runs the command
   /sbin/cpuset_release_agent, supplying the pathname (relative to the mount point of
   the cpuset file system) of the abandoned cpuset. This enables automatic removal of
   abandoned cpusets.

   The default value of notify_on_release in the root cpuset at system boot is
   disabled (0). The default value of other cpusets at creation is the current value
   of their parents notify_on_release setting.

   The command /sbin/cpuset_release_agent is invoked, with the name (/dev/cpuset
   relative path) of that cpuset in argv. This supports automatic cleanup of
   abandoned cpusets.

   The usual contents of the command /sbin/cpuset_release_agent is simply the shell
   script:
#!/bin/sh
rmdir /dev/cpuset/$1

   By default notify_on_release is off (0). Newly created cpusets inherit their
   notify_on_release setting from their parent cpuset.

   As with other flag values below, this flag can be changed by writing an ASCII
   number 0 or 1 (with optional trailing newline) into the file, to clear or set the
   flag, respectively.

2.4   Memory Pressure

   The memory_pressure of a cpuset provides a simple per-cpuset metric of the rate
   that the tasks in a cpuset are attempting to free up in use memory on the nodes of
   the cpuset to satisfy additional memory requests.

   This enables batch schedulers monitoring jobs running in dedicated cpusets to
   efficiently detect what level of memory pressure that job is causing.

   This is useful both on tightly managed systems running a wide mix of submitted
   jobs, which may choose to terminate or re-prioritize jobs that are trying to use
   more memory than allowed on the nodes assigned them, and with tightly coupled,
   long running, massively parallel scientific computing jobs that will dramatically
   fail to meet required performance goals if they start to use more memory than
   allowed to them.

   This mechanism provides a very economical way for the batch scheduler to monitor a
   cpuset for signs of memory pressure. It's up to the batch scheduler or other user
   code to decide what to do about it and take action.

   If the memory_pressure_enabled flag in the top cpuset is not set (0), then the
   kernel does not compute this filter, and the per-cpuset files memory_pressure
   always contain the value zero (0).

   If the memory_pressure_enabled flag in the top cpuset is set (1), then the kernel
   computes this filter for each cpuset in the system, and the memory_pressure file
   for each cpuset reflects the recent rate of such low memory page allocation
   attempts by tasks in said cpuset.

   Reading the memory_pressure file of a cpuset is very efficient. The expectation is
   that batch schedulers can poll these files and detect jobs that are causing memory
   stress, so that action can be taken to avoid impacting the rest of the system with
   a job that is trying to aggressively exceed its allowed memory.

   Note well: unless enabled by setting memory_pressure_enabled in the top cpuset,
   memory_pressure is not computed for any cpuset, and always reads a value of zero.

   Why a per-cpuset, running average:

     Because this meter is per-cpuset, rather than per-task or memory region, the
     system load imposed by a batch scheduler monitoring this metric is sharply
     reduced on large systems, because a scan of the system-wide tasklist can be
     avoided on each set of queries.

     Because this meter is a running average, instead of an accumulating counter, a
     batch scheduler can detect memory pressure with a single read, instead of
     having to read and accumulate results for a period of time.

     Because this meter is per-cpuset rather than per-task or memory region, the
     batch scheduler can obtain the key information, memory pressure in a cpuset,
     with a single read, rather than having to query and accumulate results over all
     the (dynamically changing) set of tasks in the cpuset.

   A per-cpuset simple digital filter is kept within the kernel, and updated by any
   task attached to that cpuset, if it enters the synchronous (direct) page reclaim
   code.

   The per-cpuset memory_pressure file provides an integer number representing the
   recent (half-life of 10 seconds) rate of direct page reclaims caused by the tasks
   in the cpuset, in units of reclaims attempted per second, times 1000.

   The kernel computes this value using a single-pole low-pass recursive (IIR)
   digital filter coded with 32 bit integer arithmetic. The value decays at an
   exponential rate.

   Given the simple 32 bit integer arithmetic used in the kernel to compute this
   value, this meter works best for reporting page reclaim rates between one per
   millisecond (msec) and one per 32 (approx) seconds. At constant rates faster than
   one per msec it maxes out at values just under 1,000,000. At constant rates
   between one per msec, and one per second it will stabilize to a value N*1000,
   where N is the rate of events per second. At constant rates between one per second
   and one per 32 seconds, it will be choppy, moving up on the seconds that have an
   event, and then decaying until the next event. At rates slower than about one in
   32 seconds, it decays all the way back to zero between each event.

2.5   Memory Spread

   There are two Boolean flag files per cpuset that control where the kernel
   allocates pages for the file system buffers and related in kernel data structures.
   They are called memory_spread_page and memory_spread_slab.

   If the per-cpuset Boolean flag file memory_spread_page is set, the kernel will
   spread the file system buffers (page cache) evenly over all the nodes that the
   faulting task is allowed to use, instead of preferring to put those pages on the
   node where the task is running.

   If the per-cpuset Boolean flag file memory_spread_slab is set, the kernel will
   spread some file system related slab caches, such as for inodes and directory
   entries, evenly over all the nodes that the faulting task is allowed to use;
   instead of preferring to put those pages on the node where the task is running.

   The setting of these flags does not affect anonymous data segment or stack segment
   pages of a task.

   By default, both kinds of memory spreading are off, and memory pages are allocated
   on the node local to where the task is running, except perhaps as modified by the
   tasks NUMA mempolicy or cpuset configuration, as long as sufficient free memory
   pages are available.

   When new cpusets are created, they inherit the memory spread settings of their
   parent.

   Setting memory spreading causes allocations for the affected page or slab caches
   to ignore the tasks NUMA mempolicy and be spread instead. Tasks using mbind() or
   set_mempolicy() calls to set NUMA mempolicies will not notice any change in these
   calls as a result of their containing tasks memory spread settings. If memory
   spreading is turned off, the currently specified NUMA mempolicy once again applies
   to memory page allocations.

   Both memory_spread_page and memory_spread_slab are Boolean flag files. By default
   they contain "0", meaning that the feature is off for that cpuset. If a "1" is
   written to that file, that turns the named feature on.

   This memory placement policy is also known (in other contexts) as round-robin or
   interleave.

   This policy can provide substantial improvements for jobs that need to place
   thread local data on the corresponding node, but that need to access large file
   system data sets that need to be spread across the several nodes in the jobs
   cpuset in order to fit. Without this policy, especially for jobs that might have
   one thread reading in the data set, the memory allocation across the nodes in the
   jobs cpuset can become very uneven.

2.6   Memory Migration

   Normally, under the default setting (disabled) of memory_migrate, once a page is
   allocated (given a physical page of main memory) that page stays on whatever node
   it was allocated, as long as it remains allocated. If the cpuset memory placement
   policy mems subsequently changes, currently allocated pages are not moved. If
   pages are swapped out to disk and back, then on return to main memory, they may be
   allocated on different nodes, depending on the cpuset mems setting in affect at
   the time the page is swapped back in.

   When memory migration is enabled in a cpuset, if the mems setting of the cpuset is
   changed, then any memory page in use by any task in the cpuset that is on a memory
   node no longer allowed will be migrated to a memory node that is allowed.

   Also if a task is moved into a cpuset with memory_migrate enabled, any memory
   pages it uses that were on memory nodes allowed in its previous cpuset, but which
   are not allowed in its new cpuset, will be migrated to a memory node allowed in
   the new cpuset.

   The relative placement of a migrated page within the cpuset is preserved during
   these migration operations if possible. For example, if the page was on the second
   valid node of the prior cpuset, the page will be placed on the second valid node
   of the new cpuset, if possible.

   In order to maintain the cpuset relative position of pages, even pages on memory
   nodes allowed in both the old and new cpusets may be migrated. For example, if
   memory_migrate is enabled in a cpuset, and that cpusets mems file is written,
   changing it from say memory nodes "4-7", to memory nodes "5-8", then the following
   page migrations will be done, in order, for all pages in the address space of
   tasks in that cpuset:
First, migrate pages on node 7 to node 8
Second, migrate pages on node 6 to node 7
Third, migrate pages on node 5 to node 6
Fourth, migrate pages on node 4 to node 5

   In this example, pages on any memory node other than "4-7" will not be migrated.
   The order in which nodes are handled in a migration is intentionally chosen so as
   to avoid migrating memory to a node until any migrations from that node have first
   been accomplished.

2.7   Mask Format

   The Mask Format is used to represent CPU and memory node bitmasks in the
   /proc/<pid>/status file.

   It is hexadecimal, using ASCII characters "0" - "9" and "a" - "f". This format
   displays each 32-bit word in hex (zero filled), and for masks longer than one
   word, uses a comma separator between words. Words are displayed in big-endian
   order most significant first. And hex digits within a word are also in big-endian
   order.

   The number of 32-bit words displayed is the minimum number needed to display all
   bits of the bitmask, based on the size of the bitmask.

   Examples of the Mask Format:

00000001                        # just bit 0 set
80000000,00000000,00000000      # just bit 95 set
00000001,00000000,00000000      # just bit 64 set
000000ff,00000000               # bits 32-39 set
00000000,000E3862               # bits 1,5,6,11-13,17-19 set

   A mask with bits 0, 1, 2, 4, 8, 16, 32 and 64 set displays as
   "00000001,00000001,00010117". The first "1" is for bit 64, the second for bit 32,
   the third for bit 16, the fourth for bit 8, the fifth for bit 4, and the "7" is
   for bits 2, 1 and 0.

2.8   List Format

   The List Format is used to represent CPU and memory node bitmasks (sets of CPU
   and memory node numbers) in the /dev/cpuset file system.

   It is a comma separated list of CPU or memory node numbers and ranges of numbers,
   in ASCII decimal.

   Examples of the List Format:

0-4,9           # bits 0, 1, 2, 3, 4, and 9 set
0-3,7,12-15     # bits 0, 1, 2, 3, 7, 12, 13, 14, and 15 set

3   Using Cpusets at the Shell Prompt

   There are multiple ways to use cpusets, including:

     * They can be queried and changed from a shell prompt, using such command line
       utilities as echo, cat, mkdir and ls.
     * They can be queried and changed via the libcpuset 'C' programming API. The
       primary emphasis of this document is on the 'C' API.

   This section describes the use of cpusets using shell commands.

   One convenient way to learn how cpusets work is to experiment with them at the
   shell prompt, before doing extensive 'C' coding.

   Note that there is one significant difference between these two interfaces.

   Modifying the CPUs in a cpuset the shell prompt requires an additional step, due
   to intentional limitations in the kernel support for cpusets. The
   cpuset_reattach routine can be used to perform this step when using libcpuset.
   The extra step consists of writing the pid of each task attached to that cpuset
   back into the cpusets tasks file:

     In order to minimize the impact of cpusets on critical kernel code, such as the
     scheduler, and due to the fact that the kernel does not support one task
     updating the memory placement of another task directly, the impact on a task of
     changing its cpuset CPU or memory node placement, or of changing to which
     cpuset a task is attached, is subtle.

     If a cpuset has its memory nodes modified, then for each task attached to that
     cpuset, the next time that the kernel attempts to allocate a page of memory for
     that task, the kernel will notice the change in the tasks cpuset, and update
     its per-task memory placement to remain within the new cpusets memory
     placement. If the task was using mempolicy MPOL_BIND, and the nodes to which it
     was bound overlap with its new cpuset, then the task will continue to use
     whatever subset of MPOL_BIND nodes are still allowed in the new cpuset. If the
     task was using MPOL_BIND and now none of its MPOL_BIND nodes are allowed in the
     new cpuset, then the task will be essentially treated as if it was MPOL_BIND
     bound to the new cpuset (even though its NUMA placement, as queried by
     get_mempolicy(), doesn't change). If a task is moved from one cpuset to
     another, then the kernel will adjust the tasks memory placement, as above, the
     next time that the kernel attempts to allocate a page of memory for that task.

     If a cpuset has its CPUs modified, then each task using that cpuset does _not_
     change its behavior automatically. In order to minimize the impact on the
     critical scheduling code in the kernel, tasks will continue to use their prior
     CPU placement until they are rebound to their cpuset, by rewriting their pid to
     the 'tasks' file of their cpuset. If a task had been bound to some subset of
     its cpuset using the sched_setaffinity() call, the effect of this is lost on
     the rebinding. The rebound tasks cpus_allowed is set to include all cpus in the
     tasks new cpuset. If a task is moved from one cpuset to another, its CPU
     placement is updated in the same way as if the tasks pid is rewritten to the
     'tasks' file of its current cpuset.

     In summary, the memory placement of a task whose cpuset is changed is
     automatically updated by the kernel, on the next allocation of a page for that
     task, but the processor placement is not updated, until that tasks pid is
     rewritten to the 'tasks' file of its cpuset. The delay in rebinding a tasks
     memory placement is necessary because the kernel does not support one task
     changing another tasks memory placement. The added user level step in rebinding
     a tasks CPU placement is necessary to avoid impacting the scheduler code in the
     kernel with a check for changes in a tasks processor placement.

   To create a new cpuset and attach the current command shell to it, the steps are:

    1. mkdir /dev/cpuset (if not already done)
    2. mount -t cpuset cpuset /dev/cpuset (if not already done)
    3. Create the new cpuset using mkdir(1).
    4. Assign CPUs and memory nodes to the new cpuset.
    5. Attach the shell to the new cpuset.

   For example, the following sequence of commands will setup a cpuset named
   "Charlie", containing just CPUs 2 and 3, and memory node 1, and then attach the
   current shell to that cpuset:

mkdir /dev/cpuset
mount -t cpuset cpuset /dev/cpuset
cd /dev/cpuset
mkdir Charlie
cd Charlie
/bin/echo 2-3 > cpus
/bin/echo 1 > mems
/bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
cat /proc/self/cpuset

   Migrating a job (the set of tasks attached to a cpuset) to different CPUs and
   memory nodes in the system, including moving the memory pages currently allocated
   to that job, can be done as follows. Lets say you want to move the job in cpuset
   alpha (CPUs 4-7 and memory nodes 2-3) to a new cpuset beta (CPUs 16-19 and memory
   nodes 8-9).

    1. First create the new cpuset beta.
    2. Then allow CPUs 16-19 and memory nodes 8-9 in beta.
    3. Then enable memory_migration in beta.
    4. Then move each task from alpha to beta.

   The following sequence of commands accomplishes this:
cd /dev/cpuset
mkdir beta
cd beta
/bin/echo 16-19 > cpus
/bin/echo 8-9 > mems
/bin/echo 1 > memory_migrate
while read i; do /bin/echo $i; done < ../alpha/tasks > tasks

   The above should move any tasks in alpha to beta, and any memory held by these
   tasks on memory nodes 2-3 to memory nodes 8-9, respectively.

   Notice that the last step of the above sequence did not do:
cp ../alpha/tasks tasks     # Doesn't work (ignores all but first task)

   The while loop, rather than the seemingly easier use of the cp(1) command, is
   necessary because only one task PID at a time may be written to the tasks file.

   The same affect (writing one pid at a time) as the while loop can be accomplished
   more efficiently, in fewer keystrokes and in syntax that works in any shell, but
   alas more obscurely, by using the sed -u [unbuffered] option:
sed -un p < ../alpha/tasks > tasks

4   Cpuset Programming Model

   The libcpuset programming model for cpusets provides a hierarchical cpuset name
   space that integrates smoothly with Linux 2.6 kernel support for simple processor
   and memory placement. As systems become larger, with more complex memory,
   processor and bus architectures, the hierarchical cpuset model for managing
   processor and memory resources will become increasingly important.

   The cpuset name space remains visible to all tasks on a system. Once created, a
   cpuset remains in existence until it is deleted or until the system is rebooted,
   even if no tasks are currently running in that cpuset.

   The key properties of a cpuset are its pathname, the list of which CPUs and memory
   nodes it contains, and whether the cpuset has exclusive rights to these resources.

   Every task (process) in the system is attached to (running inside) a cpuset. Tasks
   inherit their parents cpuset attachment when forked. This binding of task to a
   cpuset can subsequently be changed, either by the task itself, or externally from
   another task, given sufficient authority.

   Tasks have their CPU and memory placement constrained to whatever their containing
   cpuset allows. A cpuset may have exclusive rights to its CPUs and memory, which
   provides certain guarantees that other cpusets will not overlap.

   At system boot, a top level root cpuset is created, which includes all CPUs and
   memory nodes on the system. The usual mount point of the cpuset file system, and
   hence the usual file system path to this root cpuset, is /dev/cpuset.

   Changing the cpuset binding of a task does not by default move the memory the
   tasks might have currently allocated, even if that memory is on memory nodes no
   longer allowed in the tasks cpuset. On kernels that support such memory migration,
   use the [optional] cpuset_migrate function to move allocated memory as well.

   To create a cpuset from 'C' code, one obtains a handle to a new struct cpuset,
   sets the desired attributes via that handle, and issues a cpuset_create() to
   actually create the desired cpuset and bind it to the specified name. One can also
   list by name what cpusets exist, query and modify their properties, move tasks
   between cpusets, list what tasks are currently attached to a cpuset, and delete
   cpusets.

   The cpuset_alloc() call applies a hidden undefined mark to each attribute of the
   allocated struct cpuset. Calls to the various cpuset_set*() routines mark the
   attribute being set as defined. Calls to cpuset_create() and cpuset_modify() only
   set the attributes of the cpuset marked defined. This is primarily noticable when
   creating a cpuset. Code in the kernel sets some attributes of new cpusets, such as
   memory_spread_page, memory_spread_slab and notify_on_release, by default to the
   value inherited from their parent. Unless the application using libcpuset
   explicitly overrides the setting of these attributes in the struct cpuset, between
   the calls to cpuset_alloc() and cpuset_create(), the kernel default settings will
   prevail. These hidden marks have no noticeable affect when modifying an existing
   cpuset using the sequence of calls cpuset_alloc(), cpuset_query(), and
   cpuset_modify(), because the cpuset_query() call sets all attributes and marks
   them defined, while reading the attributes from the cpuset.

   The names of cpusets in this 'C' library are either relative to the root cpuset
   mount point (typically /dev/cpuset) if the name starts with a slash '/' character
   or else relative to the current tasks cpuset.

   Cpusets can be renamed using the rename(2) system call. The per-cpuset files
   within each cpuset directory cannot be renamed, and the rename(2) system call
   cannot be used to modify the cpuset hierarchy. You cannot change the parent cpuset
   of a cpuset using rename(2).

   Despite its name, the pid parameter to various libcpuset routines is actually a
   thread id, and each thread in a threaded group can be attached to a different
   cpuset. The value returned from a call to gettid(2) can be passed in the argument
   pid.

4.1   Permission Model

   Cpusets have a permission structure which determines which users have rights to
   query, modify and attach to any given cpuset. The permissions of a cpuset are
   determined by the permissions of the special files and directories in the cpuset
   file system. The cpuset filesystem is normally mounted at /dev/cpuset.

   The directory representing each cpuset, and the special per-cpuset files in each
   such directory, both have traditional Unix hierarchical file system permissions
   for read, write and execute (or search), for each of the owning user, the owning
   group, and all others.

   For instance, a task can put itself in some other cpuset (than its current one) if
   it can write the tasks file for that cpuset (requires execute permission on the
   encompassing directories and write permission on that tasks file).

   Because the cpuset file system is not persistent, changes in permissions, and even
   the existence of cpusets other than the root cpuset, are not preserved across
   system reboots.

   An additional constraint is applied to requests to place some other task in a
   cpuset. One task may not attach another to a cpuset unless it would have
   permission to send that task a signal.

   A task may create a child cpuset if it can access and write the parent cpuset
   directory. It can modify the CPUs or memory nodes in a cpuset if it can access
   that cpusets directory (execute permissions on the encompassing directories) and
   write the corresponding cpus or mems file.

   It should be noted, however, that changes to the CPUs of a cpuset do not apply to
   any task in that cpuset until the task is reattached to that cpuset. If a task can
   write the cpus file, it should also be able to write the tasks file and might be
   expected to have permission to reattach the tasks therein (equivalent to
   permission to send them a signal).

   Some utilities and libraries read the cpuset special files of the cpuset in which
   the calling task is placed, and may be hindered if a task is placed in a cpuset
   that is not owned by the task's user-id, unless special care is taken to allow
   that task continued read access to its own cpuset special files (and search access
   to the directory containing them). For example, the Message Passing Toolkit's MPI
   library's placement functionality activated through the use of MPI_DSM_DISTRIBUTE
   will fail to place ranks correctly if it cannot read the cpuset special files
   through insufficient permissions on the cpuset directory or the special files
   within it.

   There is one minor difference between the manner in which cpuset path permissions
   are evaluated by libcpuset and the manner in which file system operation
   permissions are evaluated by direct system calls. System calls that operate on
   file pathnames, such as the open(2) system call, rely on direct kernel support for
   a tasks current directory. Therefore such calls can successfully operate on files
   in or below a tasks current directory even if the task lacks search permission on
   some ancestor directory. Calls in libcpuset that operate on cpuset pathnames, such
   as the cpuset_query() call, rely on libcpuset internal conversion of all cpuset
   pathnames to full, root-based paths, so cannot successfully operate on a cpuset
   unless the task has search permission on all ancestor directories, starting with
   the usual cpuset mount point (/dev/cpuset.)

4.2   Other Placement Mechanisms

   The Linux 2.6 kernel supports additional processor and memory placement
   mechanisms.
     * The sched_setaffinity(2) and sched_getaffinity(2) system calls set and get a
       process's CPU affinity mask, which determines the set of CPUs on which it is
       eligible to run. The taskset(1) command provides a command line utility for
       manipulating a process's CPU affinity mask using these system calls. Only CPUs
       within a tasks cpuset are allowed, but CPUs are numbered using system wide CPU
       numbers, not cpuset relative numbers.
     * The set_mempolicy(2), and get_mempolicy(2) system calls set and get a tasks
       NUMA memory policy for a process and its children, which defines on which
       nodes memory is allocated for the process. The numactl(8) command provides a
       command line utility and the libnuma library provides a 'C' API for
       manipulating a process's NUMA memory policy using these system calls. Only
       memory nodes within a tasks cpuset are allowed, but nodes are numbered using
       system wide node numbers, not cpuset relative numbers.
     * The mbind(2) system call sets the NUMA memory policy for the pages in a
       specific range of a tasks virtual address space.

   Cpusets are designed to interact cleanly with these other mechanisms, to support
   for example having a batch scheduler use cpusets to control the CPU and memory
   placement of various jobs, while within each job, these other mechanisms are used
   to manage placement in more detail. It is possible for a batch scheduler to change
   a jobs cpuset placement while preserving the internal CPU affinity and NUMA memory
   placement policy.

   The CPU and memory node placement constraints imposed by cpusets always constrain
   those of these other mechanisms. You can use these other mechanisms to reduce
   further the set of CPUs or memory nodes allowed to a task by cpusets, but you can
   not use these other mechanisms to escape a tasks cpuset confinement.

   Calls to sched_setaffinity(2) to modify a tasks CPU affinity automatically mask
   off CPUs that are not allowed by the affected tasks cpuset. If that results in all
   the requested CPUs being masked off, then the call fails with errno set to EINVAL.
   If some of the requested CPUs are allowed by the tasks cpuset, then the call
   proceeds as if only the allowed CPUs were requested, silently ignoring the other,
   unallowed CPUs. If a task is moved to a different cpuset, or if the cpus of a
   cpuset are changed, then the CPU affinity of the affected task or tasks is lost.
   If a batch scheduler needs to preserve the CPU affinity of the tasks in a job
   being moved, it should use the sched_getaffinity(2) and sched_setaffinity(2) calls
   to save and restore each affected tasks CPU affinity across the move, relative to
   the cpuset. The cpu_set_t mask data type supported by the 'C' library for use with
   the CPU affinity calls is different then the libbitmask bitmasks used by
   libcpuset, so some coding is required to convert between the two, in order to
   calculate and preserve cpuset relative CPU affinity.

   Similar to CPU affinity, calls to modify a tasks NUMA memory policy silently mask
   off requested memory nodes outside the tasks allowed cpuset, and will fail if that
   results in requesting an empty set of memory nodes. Unlike CPU affinity, the NUMA
   memory policy system calls do not support one task querying or modifying another
   tasks policy. So the kernel automatically handles preserving cpuset relative NUMA
   memory policy when either a task is attached to a different cpuset, or a cpusets
   mems setting is changed. If the old and new mems sets have the same size, then the
   cpuset relative offset of affected NUMA memory policies is preserved. If the new
   mems is smaller, then the old mems relative offsets are folded onto the new mems,
   modulo the size of the new mems. If the new mems is larger, then just the first N
   nodes are used, where N is the size of the old mems.

4.3   Cpuset Aware Thread Pinning

   If a job intends to use the Other Placement Mechanisms described above, then
   that job cannot be guaranteed safe operation under the control of a batch
   scheduler if that job might be migrated to different CPUs or memory nodes. This is
   because these Other Placement Mechanisms use system wide numbering of CPUs and
   memory nodes, not cpuset relative numbering, and the job might be migrated without
   its knowledge while it is trying to adjust its placement.

   That is, between the point where such an application computes the CPU or memory
   node on which it wants to place a thread, and the point where it issues the
   sched_setaffinity(2), mbind(2) or set_mempolicy(2) call to direct such a
   placement, the thread might be migrated to a different cpuset, or its cpuset
   changed to different CPUs or memory nodes, invalidating the CPU or memory node
   number it just computed.

   This potential race condition is not a significant problem for applications that
   only use these Other Placement Mechanisms early in the job run for initial
   placement setup, if the job is only migrated by a batch scheduler after it has
   been running for a while.

   This libcpuset library provides the following mechanisms to support cpuset
   relative thread placement that is robust even if the job is being concurrently
   migrated such as by a batch scheduler.

   If a job needs to pin a thread to a single CPU, then it can use the convenient
   cpuset_pin function. This is the most common case.

   If a job needs to implement some other variation of placement, such as to specific
   memory nodes, or to more than one CPU, then it can use the following functions to
   safely guard such code from placement changes caused by job migration:

     * cpuset_get_placement
     * cpuset_equal_placement
     * cpuset_free_placement

   The libcpuset functions cpuset_c_rel_to_sys_cpu and variations provide a
   convenient means to convert between system wide and cpuset relative CPU and memory
   node numbering.

4.4   Safe Job Migration

   Jobs that make proper use of Cpuset Aware Thread Pinning, rather than the
   unsafe Other Placement Mechanisms, can be safely migrated to a different
   cpuset, or have their cpuset's CPUs or memory nodes safely changed, without
   destroying the per-thread placement done within the job.

   A batch scheduler can safely migrate jobs while preserving per-thread placement of
   a job that is concurrently using Cpuset Aware Thread Pinning.

   A batch scheduler can accomplish this with the following steps:
    1. Suspend the tasks in the job, perhaps by sending their process group a
       SIGSTOP.
    2. Use the cpuset_init_pidlist and related pidlist functions to determine the
       list of tasks in the job.
    3. Use sched_getaffinity(2) to query the CPU affinity of each task in the job.
    4. Create a new cpuset, under a temporary name, with the new desired CPU and
       memory placement.
    5. Invoke cpuset_migrate_all to move the jobs tasks from the old cpuset to
       the new.
    6. Use cpuset_delete to delete the old cpuset.
    7. Use rename(2) on the /dev/cpuset based path of the new temporary cpuset to
       rename that cpuset to the to the old cpuset name.
    8. Convert the results of the previous sched_getaffinity(2) calls to the new
       cpuset placement, preserving cpuset relative offset by using the
       cpuset_c_rel_to_sys_cpu and related functions.
    9. Use sched_setaffinity(2) to reestablish the per-task CPU binding of each
       thread in the job.
   10. Resume the tasks in the job, perhaps by sending their process group a SIGCONT.

   The sched_getaffinity(2) and sched_setaffinity(2) 'C' library calls are limited by
   'C' library internals to systems with 1024 CPUs or fewer. To write code that will
   work on larger systems, one should use the syscall(2) indirect system call wrapper
   to directly invoke the underlying system call, bypassing the 'C' library API for
   these calls. Perhaps in the future the libcpuset library will provide functions
   that make it easier for a batch scheduler to obtain, migrate, and set a tasks CPU
   affinity.

   The suspend and resume are required in order to keep tasks in the job from
   changing their per thread CPU placement between step 3 and step 6. The kernel
   automatically migrates the per-thread memory node placement during step 4, which
   it has to as there is no way for one task to modify the NUMA memory placement
   policy of another task. But the kernel does not automatically migrate the
   per-thread CPU placement, as that can be handled by a user level process such as a
   batch scheduler doing the migration, as above.

   Migrating a job from a larger cpuset (more CPUs or nodes) to a smaller cpuset will
   lose placement information, and subsequently moving that cpuset back to a larger
   cpuset will not recover that information. Such migrations lose track of
   information on a jobs placement. This loss of information of the jobs CPU affinity
   can be avoided as described above, using sched_getaffinity(2) and
   sched_setaffinity(2) to save and restore the placement (affinity) across such a
   pair of moves. This loss of information of the jobs NUMA memory placement cannot
   be avoided because one task (the one doing the migration) cannot save nor restore
   the NUMA memory placement policy of another. So if a batch scheduler wants to
   migrate jobs without causing them to lose their mbind(2) or set_mempolicy(2)
   placement, it should only migrate to cpusets with at least as many memory nodes as
   the original cpuset.

5   CPUs and Memory Nodes

   As of this writing, the Linux kernel NUMA support presumes that there are some
   memory nodes and some CPUs, and that for each CPU, there is exactly one preferred
   or local memory node. Frequently, multiple (two or four perhaps) CPUs will be
   local to the same memory node. Some memory nodes have no local CPUs - these are
   called headless nodes.

   However, this is not the only possible architecture, and architectures are
   constantly changing, usually toward the more complex. Driven by nonstop increases
   in logic density for a half century now, the bus, cache, CPU, memory and storage
   hierarchy of large systems continues to evolve.

   This cpuset interface should remain stable over a long period of time, and be
   usable over a variety of system architectures. So this interface avoids presuming
   that there is exactly one memory node local to each CPU. Rather it just presumes
   that there are some CPUs and memory nodes, that some zero or more memory nodes are
   local to each CPU, and that some zero or more CPUs are local to each memory node.

   This interface provides the functions cpuset_localmems to identify, for a CPU,
   which memory node(s) are local to that CPU, and cpuset_localcpus to identify
   for a memory node, which CPU(s) are local. Applications using this interface can
   explicitly specify just one of CPU or memory placement, and then use these
   functions to determine the corresponding memory or CPU placement.

   In some situations, applications will want to explicitly place both CPUs and
   memory nodes, not necessarily according to the default relation between a CPU and
   its local memory node. Such situations include the following.
     * You can control specific CPU and memory node placement when precise placement
       control is required, such as when optimizing performance for a specific
       important application.
     * A higher level system service such as a batch scheduler can control specific
       CPU and memory node placement.
     * On systems having some memory on headless nodes (memory nodes with no
       associated local CPU), you can explicitly use a particular headless node with
       a particular CPU.
     * On systems supporting Hyper-Threadthreading, one could affectively disable
       Hyper-Threading, by using just one out of the two or more execution engines
       (CPUs) available on a processor die.

6   Extensible API

   In order to provide for the convenient and robust extensibility of this cpuset API
   over time, the following function enables dynamically obtaining pointers for
   optional functions by name, at run-time:

     void *cpuset_function(const char * function_name) - returns function pointer,
     or NULL if function name unrecognized

   For maximum portability, you should not reference any optional cpuset function by
   name by explicit name.

   However, if you are willing to presume that an optional function will always be
   available on the target systems of interest, you might decide to explicitly
   reference it by name, in order to improve the clarity and simplicity of the
   software in question.

   Also to support robust extensibility, flags and integer option values have names
   dynamically resolved at run-time, not via preprocessor macros.

   All functions use only the primitive types of int, char *, pid_t (for process id
   of a task), size_t (for buffer sizes), pointers to opaque structures, functions
   whose signatures use these types, and pointers to such functions. They use no
   structure members, special types or magic define constants.

   Some functions in Advanced Cpuset Library Functions are marked [optional].
   They are not available in all implementations of libcpuset. Additional [optional]
   cpuset_* functions may also be added in the future. Functions that are not marked
   [optional] are available on all implementations of libcpuset.so, and can be called
   directly without using cpuset_function(). However, any of them can also be called
   indirectly via cpuset_function().

   To safely invoke an optional function, such as for example cpuset_migrate(), use
   the following call sequence:
/* fp points to function of the type of cpuset_migrate() */
int (*fp)(struct cpuset *fromcp, struct cpuset *tocp, pid_t pid);
fp = cpuset_function("cpuset_migrate");
if (fp) {
        fp( ... );
} else {
        puts ("cpuset migration not supported");
}

   If you invoke an [optional] function directly, then your resulting program will
   not be able to link with any version of libcpuset.so that does not define that
   particular function.

7   Cpuset Text Format

   Cpuset settings may be exported to, and imported from, text files using a text
   format representation of cpusets.

   The permissions of files holding these text representations have no special
   significance to the implementation of cpusets. Rather, the permissions of the
   special cpuset files in the cpuset file system, normally mounted at /dev/cpuset,
   control reading and writing of and attaching to cpusets.

   The text representation of cpusets is not essential to the use of cpusets. One can
   directly manipulate the special files in the cpuset file system. This text
   representation provides an alternative that may be convenient for some uses.

   The cpuset text format supports one directive per line. Comments begin with the
   '#' character and extend to the end of line.

   After stripping comments, the first white space separated token on each remaining
   line selects from the following possible directives:

   cpus
          Specify which CPUs are in this cpuset. The second token on the line must be
          a comma-separated list of CPU numbers and ranges of numbers, optionally
          modified by a stride operator (see below).

   mems
          Specify which memory nodes are in this cpuset. The second token on the line
          must be a comma-separated list of memory node numbers and ranges of
          numbers, optionally modified by a stride operator (see below).

   cpu_exclusive
          The cpu_exclusive flag is set.

   mem_exclusive
          The mem_exclusive flag is set.

   notify_on_release
          The notify_on_release flag is set

   Additional unnecessary tokens on a line are quietly ignored. Lines containing only
   comments and white space are ignored.

   The token "cpu" is allowed for "cpus", and "mem" for "mems". Matching is case
   insensitive.

   The stride operator for "cpus" and "mems" values is used to designate every N-th
   CPU or memory node in a range of numbers. It is written as a colon ":" followed by
   the number N, with no spaces on either side of the colon. For example, the range
   0-31:2 designates the 16 even numbers 0, 2, 4, ... 30.

   The libcpuset routines cpuset_import and cpuset_export to handle
   converting the internal 'struct cpuset' representation of cpusets to (export) and
   from (import) this text representation.

8   Basic Cpuset Library Functions

   The basic cpuset API provides functions usable from 'C' for processor and memory
   placement within a cpuset.

   These functions enable an application to place various threads of its execution on
   specific CPUs within its current cpuset, and perform related functions such as
   asking how large the current cpuset is, and on which CPU within the current cpuset
   a thread is currently executing.

   These functions do not provide the full power of the advanced cpuset API, but they
   are easier to use, and provide some common needs of multi-threaded applications.

   Unlike the rest of this document, the functions described in this section use
   cpuset relative numbering. In a cpuset of N CPUs, the relative cpu numbers range
   from zero to N - 1.

   Unlike the underlying system calls sched_setaffinity, mbind or set_mempolicy,
   these basic cpuset API functions are robust in the presence of cpuset migration.
   If you pin a thread in your job to one of the CPUs in your jobs cpuset, it will
   stay properly pinned even if your jobs cpuset is migrated later to another set of
   CPUs and memory nodes, or even if the migration is occurring at the same time as
   your calls.

   memory placement is done automatically, preferring the node local to the requested
   CPU. Threads may only be placed on a single CPU. This avoids the need to allocate
   and free the bitmasks required to specify a set of several CPUs. These functions
   do not support creating or removing cpusets, only the placement of threads within
   an existing cpuset. This avoids the need to explicitly allocate and free cpuset
   structures. Operations only apply to the current thread, avoiding the need to pass
   the process id of the thread to be affected.

   If more powerful capabilities are needed, use the Advanced Cpuset Library
   Functions. These basic functions do not provide any essential new capability. They
   are implemented using the advanced functions, and are fully interoperable with
   them.

   On error, these routines return -1 and set errno. If invoked on an operating
   system kernel that does not support cpusets, errno is set to ENOSYS. If invoked on
   a system that supports cpusets, but when the cpuset file system is not currently
   mounted at /dev/cpuset, then errno is set to ENODEV.

   The following inclusion and linkage provides access to the cpuset API from 'C'
   code:

#include <cpuset.h>
/* link with -lcpuset */

   The following functions are supported in the basic cpuset 'C' API:

     * cpuset_pin - Pin the current thread to a CPU, preferring local memory
     * cpuset_size - Return the number of CPUs are in the current tasks cpuset
     * cpuset_where - On which CPU in current tasks cpuset did the task most
       recently execute
     * cpuset_unpin - Remove affect of cpuset_pin, let task have run of its
       entire cpuset

8.1   cpuset_pin

int cpuset_pin(int relcpu);

     Pin the current task to execute only on the CPU relcpu, which is a relative CPU
     number within the current cpuset of that task. Also, automatically pin the
     memory allowed to be used by the current task to prefer the memory on that same
     node (as determined by the cpuset_cpu2node function), but to allow any
     memory in the cpuset if no free memory is readily available on the same node.

     Return 0 on success, -1 on error. Errors include relcpu being too large
     (greater than cpuset_size() - 1). They also include running on a system that
     doesn't support cpusets (ENOSYS) and running when the cpuset file system is not
     mounted at /dev/cpuset (ENODEV).

8.2   cpuset_size

int cpuset_size();

     Return the number of CPUs in the current tasks cpuset. The relative CPU numbers
     that are passed to the cpuset_pin function and that are returned by the
     cpuset_where function, must be between 0 and N - 1 inclusive, where N is
     the value returned by cpuset_size.

     Returns -1 and sets errno on error. Errors include running on a system that
     doesn't support cpusets (ENOSYS) and running when the cpuset file system is not
     mounted at /dev/cpuset (ENODEV).

8.3   cpuset_where

int cpuset_where();

     Return the CPU number, relative to the current tasks cpuset, of the CPU on
     which the current task most recently executed. If a task is allowed to execute
     on more than one CPU, then there is no guarantee that the task is still
     executing on the CPU returned by cpuset_where, by the time that the user
     code obtains the return value.

     Returns -1 and sets errno on error. Errors include running on a system that
     doesn't support cpusets (ENOSYS) and running when the cpuset file system is not
     mounted at /dev/cpuset (ENODEV).

8.4   cpuset_unpin

int cpuset_unpin();

     Remove the CPU and memory pinning affects of any previous cpuset_pin call,
     allowing the current task to execute on any CPU in its current cpuset and to
     allocate memory on any memory node in its current cpuset. Return -1 on error, 0
     on success.

     Returns -1 and sets errno on error. Errors include running on a system that
     doesn't support cpusets (ENOSYS) and running when the cpuset file system is not
     mounted at /dev/cpuset (ENODEV).

9   Using Cpusets with Hyper-Threads

   Threading in a software application splits instructions into multiple streams so
   that multiple processors can act on them.

   Hyper-Threading (HT) Technology, developed by Intel Corporation, provides
   thread-level parallelism on each processor, resulting in more efficient use of
   processor resources, higher processing throughput, and improved performance. One
   physical CPU can appear as two logical CPUs by having additional registers to
   overlap two instruction streams or a single processor can have dual-cores
   executing instructions in parallel.

   In addition to their traditional use to control the placement of jobs on the CPUs
   and memory nodes of a system, cpusets also provides a convenient mechanism to
   control the use of Hyper-Threading.

   Some jobs achieve better performance using both of the Hyper-Thread sides, A and
   B, of a processor core, and some run better using just one of the sides, allowing
   the other side to idle.

   Since each logical (Hyper-Threaded) processor in a core has a distinct CPU number,
   one can easily specify a cpuset that contains both sides, or contains just one
   side from each of the processor cores in the cpuset.

   Cpusets can be configured to include any combination of the logical CPUs in a
   system.

   For example, the following cpuset configuration file called cpuset.cfg includes
   the A sides of an HT enabled system, along with all the memory, on the first 32
   nodes (assuming 2 cores per node). The colon ':' prefixes the stride. The stride
   of '2' in this example means use every other logical CPU.

cpus 0-127:2    # even numbered CPUs 0, 2, 4, ... 126
mems 0-31       # memory nodes 0, 1, 2, ... 31

   To create a cpuset called foo and run a command called bar in that cpuset, defined
   by the cpuset configuration file cpuset.cfg shown above, use the following
   commands:

cpuset -c /foo < cpuset.cfg
cpuset -i /foo -I bar

   To specify both sides of the first 64 cores, use the following entry in your
   cpuset configuration file:

cpus 0-127

   To specify just the B sides, use the following entry in your cpuset configuration
   file:

cpus 1-127:2

   The examples above assume that CPUs are uniformly numbered, with the even numbers
   for the A side and odd numbers for the B side. This is usually the case, but not
   guaranteed. One could still place a job on a system that was not uniformly
   numbered, but currently it would involve a longer argument list to the cpus
   option, explicitly listing the desired CPUs.

   Typically, the logical numbering of CPUs puts the even numbered CPUs on the A
   sides, and the odd numbered CPUs on the B side. The stride operator (":2", above)
   makes it easy to specify that only every other side will be used. If the CPU
   number range starts with an even number, this will be the A sides, and if the
   range starts with an odd number, this will be the B sides.

   A ps(1) or top(1) invocation will show a handful of threads on unused CPUs, but
   these are kernel threads assigned to every CPU in support of user applications
   running on those CPUs, to handle tasks such as asynchronous file system writes and
   task migration between CPUs. If no application is actually using a CPU, then the
   kernel threads on that CPU will be almost always idle and will consume very little
   memory.

10   Advanced Cpuset Library Functions

   The advanced cpuset API provides functions usable from 'C' for managing cpusets on
   a system-wide basis.

   These functions primarily deal with the three entities (1) struct cpuset *, (2)
   system cpusets and (3) tasks.

     The struct cpuset * provides a transient in-memory structure used to build up a
     description of an existing or desired cpuset. These structs can be allocated,
     freed, queried and modified.

     Actual kernel cpusets are created under /dev/cpuset, which is the usual mount
     point of the kernels virtual cpuset filesystem. These cpusets are visible to
     all tasks (with sufficient authority) in the system, and persist until the
     system is rebooted or until the cpuset is explicitly deleted. These cpusets can
     be created, deleted, queried, modified, listed and examined.

     Every task (also known as a process) is bound to exactly one cpuset at a time.
     You can list which tasks are bound to a given cpuset, and to which cpuset a
     given task is bound. You can change to which cpuset a task is bound.

   The primary attributes of a cpuset are its lists of CPUs and memory nodes. The
   scheduling affinity for each task, whether set by default or explicitly by the
   sched_setaffinity() system call, is constrained to those CPUs that are available
   in that tasks cpuset. The NUMA memory placement for each task, whether set by
   default or explicitly by the mbind() system call, is constrained to those memory
   nodes that are available in that tasks cpuset. This provides the essential purpose
   of cpusets - to constrain the CPU and memory usage of tasks to specified subsets
   of the system.

   The other essential attribute of a cpuset is its pathname beneath /dev/cpuset. All
   tasks bound to the same cpuset pathname can be managed as a unit, and this
   hierarchical name space describes the nested resource management and hierarchical
   permission space supported by cpusets. Also, this hierarchy is used to enforce
   strict exclusion, using the following rules:

     * A cpuset may only be marked strictly exclusive for CPU or memory if its parent
       is also.
     * A cpuset may not make any CPUs or memory nodes available that are not also
       available in its parent.
     * If a cpuset is exclusive for CPU or memory, then it may not overlap CPUs or
       memory with any of its siblings.

   The combination of these rules enables checking for strict exclusion just by
   making various checks on the parent, siblings and existing child cpusets of the
   cpuset being changed, without having to check all cpusets in the system.

   On error, some of these routines return -1 or NULL and set errno. If one of the
   routines below that requires cpuset kernel support is invoked on an operating
   system kernel that does not support cpusets, then that routine returns failure and
   errno is set to ENOSYS. If invoked on a system that supports cpusets, but when the
   cpuset file system is not currently mounted at /dev/cpuset, then it returns
   failure and errno is set to ENODEV.

   The following inclusion and linkage provides access to the cpuset API from 'C'
   code:

#include <bitmask.h>
#include <cpuset.h>
/* link with -lcpuset */

   The following functions are supported in the advanced cpuset 'C' API:
     * Cpuset library (libcpuset) version
          + cpuset_version - [optional] Version (simple integer) of the library
     * Allocate and free struct cpuset *
          + cpuset_alloc - Return handle to newly allocated struct cpuset *
          + cpuset_free - Discard no longer needed struct cpuset *
     * Lengths of CPUs and memory nodes bitmasks - needed to allocate bitmasks
          + cpuset_cpus_nbits - Number of bits needed for a CPU bitmask on
            current system
          + cpuset_mems_nbits - Number of bits needed for a memory bitmask on
            current system
     * Set various attributes of a struct cpuset *
          + cpuset_setcpus - Specify CPUs in cpuset
          + cpuset_setmems - Specify memory nodes in cpuset
          + cpuset_set_iopt - Specify an integer value option of cpuset
          + cpuset_set_sopt - [optional] Specify a string value option of cpuset
     * Query various attributes of a struct cpuset *
          + cpuset_getcpus - Query CPUs in cpuset
          + cpuset_getmems - Query memory nodes in cpuset
          + cpuset_cpus_weight - Number of CPUs in a cpuset
          + cpuset_mems_weight - Number of memory nodes in a cpuset
          + cpuset_get_iopt - Query an integer value option of cpuset
          + cpuset_get_sopt - [optional] Query a string value option of cpuset
     * Local CPUs and memory nodes
          + cpuset_localcpus - Query the CPUs local to specified memory nodes
          + cpuset_localmems - Query the memory nodes local to specified CPUs
          + cpuset_cpumemdist - [optional] Hardware distance from CPU to memory
            node
          + cpuset_cpu2node - Return system number of memory node closest to
            specified CPU.
          + cpuset_addr2node - Return sytem number of memory node holding page
            at specified address.
     * Create, delete, query, modify, list and examine cpusets.
          + cpuset_create - Create named cpuset as specified by struct cpuset *
          + cpuset_delete - Delete the specified cpuset (if empty)
          + cpuset_query - Set struct cpuset to settings of specified cpuset
          + cpuset_modify - Modify a cpuset's settings to those specified in a
            struct cpuset
          + cpuset_getcpusetpath - Get path of a tasks (0 for current) cpuset.
          + cpuset_cpusetofpid - Set struct cpuset to settings of cpuset of
            specified task
          + cpuset_mountpoint - Return path at which cpuset filesystem is
            mounted
          + cpuset_collides_exclusive - [optional] True if would collide
            exclusive
          + cpuset_nuke - [optional] Remove cpuset anyway possible
     * List tasks currently attached to a cpuset
          + cpuset_init_pidlist - Initialize a list of tasks attached to a
            cpuset
          + cpuset_pidlist_length - Return the number of tasks in such a list
          + cpuset_get_pidlist - Return a specific task from such a list
          + cpuset_freepidlist - Deallocate such a list
     * Attach tasks to cpusets.
          + cpuset_move - Move task (0 for current) to a cpuset
          + cpuset_move_all - Move all tasks in a list of pids to a cpuset
          + cpuset_move_cpuset_tasks - [optional] Move all tasks in a cpuset to
            another cpuset
          + cpuset_migrate - [optional] Move a task and its memory to a cpuset
          + cpuset_migrate_all - [optional] Move all tasks with memory in a list
            of pids to a cpuset
          + cpuset_reattach - Rebind cpus_allowed of each task in a cpuset after
            changing its cpus
     * Determine memory pressure
          + cpuset_open_memory_pressure - [optional] Open handle to read
            memory_pressure
          + cpuset_read_memory_pressure - [optional] Read cpuset current
            memory_pressure
          + cpuset_close_memory_pressure - [optional] Close handle to read
            memory pressure
     * Map between cpuset relative and system-wide CPU and memory node numbers
          + cpuset_c_rel_to_sys_cpu - Map cpuset relative CPU number to system
            wide number
          + cpuset_c_sys_to_rel_cpu - Map system wide CPU number to cpuset
            relative number
          + cpuset_c_rel_to_sys_mem - Map cpuset relative memory node number to
            system wide number
          + cpuset_c_sys_to_rel_mem - Map system wide memory node number to
            cpuset relative number
          + cpuset_p_rel_to_sys_cpu - Map task cpuset relative CPU number to
            system wide number
          + cpuset_p_sys_to_rel_cpu - Map system wide CPU number to task cpuset
            relative number
          + cpuset_p_rel_to_sys_mem - Map task cpuset relative memory node
            number to system wide number
          + cpuset_p_sys_to_rel_mem - Map system wide memory node number to task
            cpuset relative number
     * Placement operations - for detecting cpuset migration
          + cpuset_get_placement - [optional] Return current placement of task
            pid
          + cpuset_equal_placement - [optional] True if two placements equal
          + cpuset_free_placement - [optional] Free placement
     * Traverse a cpuset hierarchy.
          + cpuset_fts_open - [optional] Open cpuset hierarchy
          + cpuset_fts_read - [optional] Next entry in hierarchy
          + cpuset_fts_reverse - [optional] Reverse order of cpusets
          + cpuset_fts_rewind - [optional] Rewind to first cpuset in list
          + cpuset_fts_get_path - [optional] Get entry's cpuset path
          + cpuset_fts_get_stat - [optional] Get entry's stat(2) pointer
          + cpuset_fts_get_cpuset - [optional] Get entry's cpuset pointer
          + cpuset_fts_get_errno - [optional] Get entry's errno
          + cpuset_fts_get_info - [optional] Get operation causing error
          + cpuset_fts_close - [optional] Close cpuset hierarchy
     * Bind to a CPU or memory node within the current cpuset
          + cpuset_cpubind - Bind to a single CPU within a cpuset (uses
            sched_setaffinity(2))
          + cpuset_latestcpu - Most recent CPU on which a task has executed
          + cpuset_membind - Bind to a single memory node within a cpuset (uses
            set_mempolicy(2))
     * Export cpuset settings to a regular file, and import them from a regular file
          + cpuset_export - Export cpuset settings to a text file
          + cpuset_import - Import cpuset settings from a text file
     * Support calls to [optional] cpuset_* API routines
          + cpuset_function - Return pointer to a libcpuset.so function, or NULL

   A typical calling sequence would use the above functions in the following order to
   create a new cpuset named "xyz" and attach itself to it.

struct cpuset *cp = cpuset_alloc();
various cpuset_set*(cp, ...) calls
cpuset_create(cp, "xyz");
cpuset_free(cp);
cpuset_move(0, "xyz");

   Some functions above are marked [optional]. For more information, see the
   Extensible API section, above, for an explanation of this marking and how to
   invoke such functions in a portable manner.

10.1   cpuset_version

int cpuset_version();

     Version (simple integer) of the cpuset library (libcpuset). The version number
     returned by cpuset_version() is incremented anytime that any changes or
     additions are made to its API or behaviour. Other mechanims are provided to
     maintain full upward compatibility with this libraries API. This
     cpuset_version() call is intended to provide a fallback mechanism in case
     an application needs to distinguish between two previous versions of this
     library.

     This is an [optional] function. Use cpuset_function to invoke it.

10.2   cpuset_alloc

struct cpuset *cpuset_alloc();

     Creates, initializes and returns a handle to a struct cpuset, which is an
     opaque data structure used to describe a cpuset.

     After obtaining a struct cpuset handle with this call, one can use the various
     cpuset_set() methods to specify which CPUs and memory nodes are in the cpuset
     and other attributes. Then one can create such a cpuset with the
     cpuset_create() call and free cpuset handles with the cpuset_free() call.

     The cpuset_alloc function returns a zero pointer (NULL) and sets errno in
     the event that malloc(3) fails. See the malloc(3) man page for possible values
     of errno (ENOMEM being the most likely).

10.3   cpuset_free

void cpuset_free(struct cpuset *cp);

     Frees the memory associated with a struct cpuset handle, which must have been
     returned by a previous cpuset_alloc() call. If cp is NULL, no operation is
     performed.

10.4   cpuset_cpus_nbits

int cpuset_cpus_nbits();

     Return the number of bits in a CPU bitmask on current system. Useful when using
     bitmask_alloc() to allocate a CPU mask. Some other routines below return
     cpuset_cpus_nbits() as an out-of-bounds indicator.

10.5   cpuset_mems_nbits

int cpuset_mems_nbits();

     Return the number of bits in a memory node bitmask on current system. Useful
     when using bitmask_alloc() to allocate a memory node mask. Some other routines
     below return cpuset_mems_nbits() as an out-of-bounds indicator.

10.6   cpuset_setcpus

int cpuset_setcpus(struct cpuset *cp, const struct bitmask *cpus);

     Given a bitmask of CPUs, the cpuset_setcpus() call sets the specified cpuset cp
     to include exactly those CPUs.

     Returns 0 on success, else -1 on error, setting errno. This routine can fail if
     malloc(3) fails. See the malloc(3) man page for possible values of errno
     (ENOMEM being the most likely).

10.7   cpuset_setmems

void cpuset_setmems(struct cpuset *cp, const struct bitmask *mems);

     Given a bitmask of memory nodes, the cpuset_setmems() call sets the specified
     cpuset cp to include exactly those memory nodes.

     Returns 0 on success, else -1 on error, setting errno. This routine can fail if
     malloc(3) fails. See the malloc(3) man page for possible values of errno
     (ENOMEM being the most likely).

10.8   cpuset_set_iopt

int cpuset_set_iopt(struct cpuset *cp, const char *optionname, int value);

     Sets cpuset integer valued option optionname to specified integer value.
     Returns 0 if optionname is recognized and value is an allowed value for that
     option. Returns -1 if optionname is recognized, but value is not allowed.
     Returns -2 if optionname is not recognized. Boolean options accept any non-zero
     value as equivalent to a value of one (1).

     The following optionname's are recognized:
     * cpu_exclusive - sibling cpusets not allowed to overlap cpus (see section
       Exclusive Cpusets, above)
     * mem_exclusive - sibling cpusets not allowed to overlap mems (see section
       Exclusive Cpusets, above)
     * notify_on_release - invoke /sbin/cpuset_release_agent when cpuset released
       (see section Notify On Release, above)
     * memory_migrate - causes memory pages to migrate to new mems (see section
       Memory Migration, above)
     * memory_spread_page - causes kernel buffer (page) cache to spread over cpuset
       (see section Memory Spread, above)
     * memory_spread_slab - causes kernel file i/o data (directory and inode slab
       caches) to spread over cpuset(see section Memory Spread, above)

10.9   cpuset_set_sopt

int cpuset_set_sopt(struct cpuset *cp, const char *optionname, const char *value);

     Sets cpuset string valued option optionname to specified string value. Returns
     0 if optionname is recognized and value is an allowed value for that option.
     Returns -1 if optionname is recognized, but value is not allowed. Returns -2 if
     optionname is not recognized.

     This is an [optional] function. Use cpuset_function to invoke it.

10.10   cpuset_getcpus

int cpuset_getcpus(const struct cpuset *cp, struct bitmask *cpus);

     Query CPUs in cpuset cp, by writing them to the bitmask cpus. Pass cp == NULL
     to query the current tasks cpuset

     If the CPUs have not been set in cpuset cp, then no operation is performed, -1
     is returned, and errno is set to EINVAL.

     Returns 0 on success, else -1 on error, setting errno. This routine can fail if
     malloc(3) fails. See the malloc(3) man page for possible values of errno
     (ENOMEM being the most likely).

10.11   cpuset_getmems

int cpuset_getmems(const struct cpuset *cp, struct bitmask *mems);

     Query memory nodes in cpuset cp, by writing them to the bitmask mems. Pass cp
     == NULL to query the current tasks cpuset.

     If the memory nodes have not been set in cpuset cp, then no operation is
     performed, -1 is returned, and errno is set to EINVAL.

     Returns 0 on success, else -1 on error, setting errno. This routine can fail if
     malloc(3) fails. See the malloc(3) man page for possible values of errno
     (ENOMEM being the most likely).

10.12   cpuset_cpus_weight

int cpuset_cpus_weight(const struct cpuset *cp);

     Query number of CPUs in cpuset cp. Pass cp == NULL to query the current tasks
     cpuset

     If the CPUs have not been set in cpuset cp, then zero(0) is returned.

10.13   cpuset_mems_weight

int cpuset_mems_weight(const struct cpuset *cp);

     Query number of memory nodes in cpuset cp. Pass cp == NULL to query the current
     tasks cpuset

     If the memory nodes have not been set in cpuset cp, then zero (0) is returned.

10.14   cpuset_get_iopt

int cpuset_get_iopt(const struct cpuset *cp, const char *optionname);

     Query value of integer option optionname in cpuset cp. Returns value of
     optionname if it is recognized, else returns -1. Integer values in an
     uninitialized cpuset have value 0.

     The following optionname's are recognized:
     * cpu_exclusive - sibling cpusets not allowed to overlap cpus (see section
       Exclusive Cpusets, above)
     * mem_exclusive - sibling cpusets not allowed to overlap mems (see section
       Exclusive Cpusets, above)
     * notify_on_release - invoke /sbin/cpuset_release_agent when cpuset released
       (see section Notify On Release, above)
     * memory_migrate - causes memory pages to migrate to new mems (see section
       Memory Migration, above)
     * memory_spread_page - causes kernel buffer (page) cache to spread over cpuset
       (see section Memory Spread, above)
     * memory_spread_slab - causes kernel file i/o data (directory and inode slab
       caches) to spread over cpuset(see section Memory Spread, above)

10.15   cpuset_get_sopt

const char *cpuset_get_sopt(const struct cpuset *cp, const char *optionname);

     Query value of string option optionname in cpuset cp. Returns pointer to value
     of optionname if it is recognized, else returns NULL. String values in an
     uninitialized cpuset have value NULL.

     This is an [optional] function. Use cpuset_function to invoke it.

10.16   cpuset_localcpus

int cpuset_localcpus(const struct bitmask *mems, struct bitmask *cpus);

     Query the CPUs local to specified memory nodes mems, by writing them to the
     bitmask cpus. Return 0 on success, -1 on error, setting errno.

10.17   cpuset_localmems

int cpuset_localmems(const struct bitmask *cpus, struct bitmask *mems);

     Query the memory nodes local to specified CPUs cpus, by writing them to the
     bitmask mems. Return 0 on success, -1 on error, setting errno.

10.18   cpuset_cpumemdist

unsigned int cpuset_cpumemdist(int cpu, int mem);

     Distance between hardware CPU cpu and memory node mem, on a scale which has the
     closest distance of a CPU to its local memory valued at ten (10), and other
     distances more or less proportional. Distance is a rough metric of the
     bandwidth and delay combined, where a higher distance means lower bandwidth and
     longer delays.

     If either cpu or mem is not known to the current system, or if any internal
     error occurs while evaluating this distance, or if a node has no CPUs nor
     memory (I/O only), then the distance returned is UCHAR_MAX (from limits.h).

     These distances are obtained from the systems ACPI SLIT table, and should
     conform to:

     System Locality Information Table Interface Specification
     Version 1.0, July 25, 2003

     This is an [optional] function. Use cpuset_function to invoke it.

10.19   cpuset_cpu2node

int cpuset_cpu2node(int cpu);

     Return system wide number of memory node closest to CPU cpu. For NUMA
     architectures common as of this writing, this would be the number of the node
     on which cpu is located. If an architecture did not have memory on the same
     node as a CPU, then it would be the node number of the memory node closest to
     or preferred by that cpu.

     This is an [optional] function. Use cpuset_function to invoke it.

10.20   cpuset_addr2node

int cpuset_addr2node(void *addr);

     Return system wide number of memory node on which is located the physical page
     of memory at virtual address addr of the current tasks address space. Returns
     -1 if addr is not a valid address in the address space of the current process,
     with errno set to EFAULT. If the referenced physical page was not allocated
     (faulted in) by the kernel prior to this call, it will be during the call.

     This is an [optional] function. Use cpuset_function to invoke it.

10.21   cpuset_create

int cpuset_create(const char *cpusetpath, const struct *cp);

     Create a cpuset at the specified cpusetpath, as described in the provided
     struct cpuset *cp. The parent cpuset of that pathname must already exist.

     The parameter cp refers to a handle obtained from a cpuset_alloc() call. If the
     parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, else -1 on error, setting errno. This routine can fail if
     malloc(3) fails. See the malloc(3) man page for possible values of errno
     (ENOMEM being the most likely).

10.22   cpuset_delete

int cpuset_delete(const char *cpusetpath);

     Delete a cpuset at the specified cpusetpath. The cpuset of that pathname must
     already exist, be empty (no child cpusets) and be unused (no using tasks).

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, else -1 on error, setting errno.

10.23   cpuset_query

int cpuset_query(struct cpuset *cp, const char *cpusetpath);

     Set struct cpuset to settings of cpuset at specified path cpusetpath. Struct
     cpuset *cp must have been returned by a previous cpuset_alloc() call. Any
     previous settings of cp are lost.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, or -1 on error, setting errno. Errors include cpusetpath
     not referencing a valid cpuset path relative to /dev/cpuset, or the current
     task lacking permission to query that cpuset.

10.24   cpuset_modify

int cpuset_modify(const char *cpusetpath, const struct *cp);

     Modify the cpuset at the specified cpusetpath, as described in the provided
     struct cpuset *cp. The cpuset at that pathname must already exist.

     The parameter cp refers to a handle obtained from a cpuset_alloc() call.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, else -1 on error, setting errno.

10.25   cpuset_getcpusetpath

char *cpuset_getcpusetpath(pid_t pid, char *buf, size_t size);

     The cpuset_getcpusetpath() function copies an absolute pathname of the cpuset
     to which task of process id pid is attached, to the array pointed to by buf,
     which is of length size. Use pid == 0 for the current process.

     The provided path is relative to the cpuset file system mount point.

     If the cpuset path name would require a buffer longer than size elements, NULL
     is returned, and errno is set to ERANGE an application should check for this
     error, and allocate a larger buffer if necessary.

     Returns NULL on failure with errno set accordingly, and buf on success. The
     contents of buf are undefined on error.

     ERRORS

     EACCES Permission to read or search a component of the file name was denied.

     EFAULT buf points to a bad address.

     ESRCH The pid does not exist.

     E2BIG Larger buffer needed.

     ENOSYS Kernel does not support cpusets.

10.26   cpuset_cpusetofpid

int cpuset_cpusetofpid(struct cpuset *cp, pid_t pid);

     Set struct cpuset to settings of cpuset to which specified task pid is
     attached. Struct cpuset *cp must have been returned by a previous
     cpuset_alloc() call. Any previous settings of cp are lost.

     Returns 0 on success, or -1 on error, setting errno.

     ERRORS

     EACCES Permission to read or search a component of the file name was denied.

     EFAULT buf points to a bad address.

     ESRCH The pid does not exist.

     ERANGE Larger buffer needed.

     ENOSYS Kernel does not support cpusets.

10.27   cpuset_mountpoint

const char *cpuset_mountpoint();

     Return the filesystem path at which the cpuset file system is mounted. The
     current implementation of this routine returns /dev/cpuset, or the string
     [cpuset filesystem not mounted] if the cpuset file system is not mounted, or
     the string [cpuset filesystem not supported] if the system does not support
     cpusets.

     In general, if the first character of the return string is a slash (/) then the
     result is the mount point of the cpuset file system, otherwise the result is an
     error message string.

     This is an [optional] function. Use cpuset_function to invoke it.

10.28   cpuset_collides_exclusive

int cpuset_collides_exclusive(const char *cpusetpath, const struct *cp);

     Return true (1) if cpuset cp would collide with any sibling of the cpuset at
     cpusetpath due to overlap of cpu_exclusive cpus or mem_exclusive mems. Return
     false (0) if no collision, or for any error.

     cpuset_create fails with errno == EINVAL if the requested cpuset would
     overlap with any sibling, where either one is cpu_exclusive or mem_exclusive.
     This is a common, and not obvious error. cpuset_collides_exclusive() checks for
     this particular case, so that code creating cpusets can better identify the
     situation, perhaps to issue a more informative error message.

     Can also be used to diagnose cpuset_modify failures. This routine ignores
     any existing cpuset with the same path as the given cpusetpath, and only looks
     for exclusive collisions with sibling cpusets of that path.

     In case of any error, returns (0) -- does not collide. Presumably any actual
     attempt to create or modify a cpuset will encounter the same error, and report
     it usefully.

     This routine is not particularly efficient; most likely code creating or
     modifying a cpuset will want to try the operation first, and then if that fails
     with errno EINVAL, perhaps call this routine to determine if an exclusive
     cpuset collision caused the error.

     This is an [optional] function. Use cpuset_function to invoke it.

10.29   cpuset_nuke

int cpuset_nuke(const char *cpusetpath, unsigned int seconds);

     Remove a cpuset, including killing tasks in it, and removing any descendent
     cpusets and killing their tasks.

     Tasks can take a long time (minutes on some configurations) to exit. Loop up to
     seconds seconds, trying to kill them.

     The following steps are taken to remove a cpuset:

    1. First, kill all the pids, looping until there are no more pids in this cpuset
       or below, or until the 'seconds' timeout limit is exceeded.
    2. Then depth first recursively rmdir the cpuset directories.
    3. If by this point the original cpuset is gone, return success.

     If the timeout is exceeded, and tasks still exist, fail with errno == ETIME.

     This routine sleeps a variable amount of time. After the first attempt to kill
     all the tasks in the cpuset or its descendents, it sleeps one second, the next
     time two seconds, increasing one second each loop up to a max of ten seconds.
     If more loops past ten are required to kill all the tasks, it sleeps ten
     seconds each subsequent loop. In any case, before the last loop, it sleeps
     however many seconds remain of the original timeout seconds requested. The
     total time of all sleeps will be no more than the requested seconds.

     If the cpuset started out empty of any tasks, or if the passed in seconds was
     zero, then this routine will return quickly, having not slept at all.
     Otherwise, this routine will at a minimum send a SIGKILL to all the tasks in
     this cpuset subtree, then sleep one second, before looking to see if any tasks
     remain. If tasks remain in the cpuset subtree, and a longer seconds timeout was
     requested (more than one), it will continue to kill remaining tasks and sleep,
     in a loop, for as long as time and tasks remain.

     The signal sent for the kill is hardcoded to SIGKILL. If some other signal
     should be sent first, use a separate code loop, perhaps based on
     cpuset_init_pidlist and cpuset_get_pidlist, to scan the task pids in a cpuset.
     If SIGKILL should -not- be sent, this cpuset_nuke() routine can still be called
     to recursively remove a cpuset subtree, by specifying a timeout of zero
     seconds.

     On success, returns 0 with errno == 0.

     On failure, returns -1, setting errno.

     ERRORS

     EACCES search permission denied on intervening directory

     ETIME timed out - tasks remain after 'seconds' timeout

     EMFILE too many open files

     ENODEV /dev/cpuset not mounted

     ENOENT component of cpuset path doesn't exist

     ENOMEM out of memory

     ENOSYS kernel doesn't support cpusets

     ENOTDIR component of cpuset path is not a directory

     EPERM lacked permission to kill a task

     EPERM lacked permission to read cpusets or files therein

     This is an [optional] function. Use cpuset_function to invoke it.

10.30   cpuset_init_pidlist

struct cpuset_pidlist *cpuset_init_pidlist(const char *cpusetpath, int recursiveflag);

     Initialize and return a list of tasks (PIDs) attached to cpuset cpusetpath. If
     recursiveflag is zero, include only the tasks directly in that cpuset,
     otherwise include all tasks in that cpuset or any descendant thereof.

     Beware that tasks may come and go from a cpuset, after this call is made.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     On error, return NULL and set errno.

10.31   cpuset_pidlist_length

int cpuset_pidlist_length(const struct cpuset_pidlist *pl);

     Return the number of elements (PIDs) in cpuset_pidlist pl.

10.32   cpuset_get_pidlist

pid_t cpuset_get_pidlist(const struct cpuset_pidlist *pl, int i);

     Return the i'th element of a cpuset_pidlist. The elements of a cpuset_pidlist
     of length N are numbered 0 through N-1. Return (pid_t)-1 for any other index i.

10.33   cpuset_freepidlist

void cpuset_freepidlist(struct cpuset_pidlist *pl);

     Deallocate a list of attached pids.

10.34   cpuset_move

int cpuset_move(pid_t p, const char *cpusetpath);

     Move task whose process id is p to cpuset cpusetpath. If pid is zero, then the
     current task is moved.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, else -1 on error, setting errno.

10.35   cpuset_move_all

int cpuset_move_all(struct cpuset_pid_list *pl, const char *cpusetpath);

     Move all tasks in list pl to cpuset cpusetpath.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, else -1 on error, setting errno.

10.36   cpuset_move_cpuset_tasks

int cpuset_move_cpuset_tasks(const char *fromrelpath, const char *torelpath);

     Move all tasks in cpuset fromrelpath to cpuset torelpath. This may race with
     tasks being added to or forking into fromrelpath. Loop repeatedly, reading the
     tasks file of cpuset fromrelpath and writing any task pid's found there to the
     tasks file of cpuset torelpath, up to ten attempts, or until the tasks file of
     cpuset fromrelpath is empty, or until the cpuset fromrelpath is no longer
     present.

     Returns 0 with errno == 0 if able to empty the tasks file of cpuset
     fromrelpath. Of course it is still possible that some independent task could
     add another task to cpuset fromrelpath at the same time that such a successful
     result is being returned, so there can be no guarantee that a successful return
     means that fromrelpath is still empty of tasks.

     The cpuset fromrelpath might disappear during this operation, perhaps because
     it has notify_on_release set and was automatically removed as soon as its last
     task was detached from it. Consider a missing fromrelpath to be a successful
     move.

     If called with fromrelpath and torelpath pathnames that evaluate to the same
     cpuset, then treat that as if cpuset_reattach() was called, rebinding each
     task in this cpuset one time, and return success or failure depending on the
     return of that cpuset_reattach() call.

     On failure, returns -1, setting errno.

     ERRORS

     EACCES search permission denied on intervening directory

     ENOTEMPTY tasks remain after multiple attempts to move them

     EMFILE too many open files

     ENODEV /dev/cpuset not mounted

     ENOENT component of cpuset path doesn't exist

     ENOMEM out of memory

     ENOSYS kernel doesn't support cpusets

     ENOTDIR component of cpuset path is not a directory

     EPERM lacked permission to read cpusets or files therein

     EACCES`      lacked permission to write a cpuset ``tasks file

     This is an [optional] function. Use cpuset_function to invoke it.

10.37   cpuset_migrate

int cpuset_migrate(pid_t pid, const char *cpusetpath);

     Migrate task whose process id is p to cpuset cpusetpath, moving its currently
     allocated memory to nodes in that cpuset, if not already there. If pid is zero,
     then the current task is migrated.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, else -1 on error, setting errno

     For more information, see the Memory Migration section, above.

     This is an [optional] function. Use cpuset_function to invoke it.

10.38   cpuset_migrate_all

int cpuset_migrate_all(struct cpuset_pid_list *pl, const char *cpusetpath);

     Move all tasks in list pl to cpuset cpusetpath, moving their currently
     allocated memory to nodes in that cpuset, if not already there.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     For more information, see the Memory Migration section, above.

     This is an [optional] function. Use cpuset_function to invoke it.

10.39   cpuset_reattach

int cpuset_reattach(const char *cpusetpath);

     Reattach all tasks in cpuset cpusetpath to itself. This additional step is
     necessary anytime that the cpus of a cpuset have been changed, in order to
     rebind the cpus_allowed of each task in the cpuset to the new value. This
     routine writes the pid of each task currently attached to the named cpuset to
     the tasks file of that cpuset. If additional tasks are being spawned too
     rapidly into the cpuset at the same time, there is an unavoidable race
     condition, and some tasks may be missed.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     Returns 0 on success, else -1 on error, setting errno.

10.40   cpuset_open_memory_pressure

int cpuset_open_memory_pressure(const char *cpusetpath);

     Open a file descriptor from which to read the memory_pressure of the cpuset
     cpusetpath.

     If the parameter cpusetpath starts with a slash (/) character, then this a path
     relative to /dev/cpuset, otherwise it is relative to the current tasks cpuset.

     By default, computation by the kernel of memory_pressure is disabled. Set the
     memory_pressure_enabled flag in the top cpuset to enable it.

     On error, return -1 and set errno.

     For more information, see the Memory Pressure section, above.

     This is an [optional] function. Use cpuset_function to invoke it.

10.41   cpuset_read_memory_pressure

int cpuset_read_memory_pressure(int fd);

     Read and return the current memory_pressure of the cpuset for which file
     descriptor fd was opened using cpuset_open_memory_pressure.

     Uses the system call pread(2). On success, returns a non-negative number, as
     described in section Memory Pressure. On failure, returns -1 and sets
     errno.

     By default, computation by the kernel of memory_pressure is disabled. Set the
     memory_pressure_enabled flag in the top cpuset to enable it.

     For more information, see the Memory Pressure section, above.

     This is an [optional] function. Use cpuset_function to invoke it.

10.42   cpuset_close_memory_pressure

void cpuset_close_memory_pressure(int fd);

     Close the file descriptor fd which was opened using
     cpuset_open_memory_pressure.

     If fd is not a valid open file descriptor, then this call does nothing. No
     error is returned in any case.

     By default, computation by the kernel of memory_pressure is disabled. Set the
     memory_pressure_enabled flag in the top cpuset to enable it.

     For more information, see the Memory Pressure section, above.

     This is an [optional] function. Use cpuset_function to invoke it.

10.43   cpuset_c_rel_to_sys_cpu

int cpuset_c_rel_to_sys_cpu(const struct cpuset *cp, int cpu);

     Return the system-wide CPU number that is used by the cpu-th CPU of the
     specified cpuset cp. Returns result of cpuset_cpus_nbits() if cpu is not in the
     range [0, bitmask_weight(cpuset_getcpus(cp))).

10.44   cpuset_c_sys_to_rel_cpu

int cpuset_c_sys_to_rel_cpu(const struct cpuset *cp, int cpu);

     Return the cpu-th CPU of the specified cpuset cp that is used by the
     system-wide CPU number. Returns result of cpuset_cpus_nbits() if
     bitmask_isbitset(cpuset_getcpus(cp), cpu) is false.

10.45   cpuset_c_rel_to_sys_mem

int cpuset_c_rel_to_sys_mem(const struct cpuset *cp, int mem);

     Return the system-wide memory node number that is used by the mem-th memory
     node of the specified cpuset cp. Returns result of cpuset_mems_nbits() if mem
     is not in the range [0, bitmask_weight(cpuset_getmems(cp))).

10.46   cpuset_c_sys_to_rel_mem

int cpuset_c_sys_to_rel_mem(const struct cpuset *cp, int mem);

     Return the mem-th memory node of the specified cpuset cp that is used by the
     system-wide memory node number. Returns result of cpuset_mems_nbits() if
     bitmask_isbitset(cpuset_getmems(cp), mem) is false.

10.47   cpuset_p_rel_to_sys_cpu

int cpuset_p_rel_to_sys_cpu(pid_t pid, int cpu);

     Return the system-wide CPU number that is used by the cpu-th CPU of the cpuset
     to which task pid is attached. Returns result of cpuset_cpus_nbits() if that
     cpuset doesn't encompass that relative cpu number.

10.48   cpuset_p_sys_to_rel_cpu

int cpuset_p_sys_to_rel_cpu(pid_t pid, int cpu);

     Return the cpu-th CPU of the cpuset to which task pid is attached that is used
     by the system-wide CPU number. Returns result of cpuset_cpus_nbits() if that
     cpuset doesn't encompass that system-wide cpu number.

10.49   cpuset_p_rel_to_sys_mem

int cpuset_p_rel_to_sys_mem(pid_t pid, int mem);

     Return the system-wide memory node number that is used by the mem-th memory
     node of the cpuset to which task pid is attached. Returns result of
     cpuset_mems_nbits() if that cpuset doesn't encompass that relative memory node
     number.

10.50   cpuset_p_sys_to_rel_mem

int cpuset_p_sys_to_rel_mem(pid_t pid, int mem);

     Return the mem-th memory node of the cpuset to which task pid is attached that
     is used by the system-wide memory node number. Returns result of
     cpuset_mems_nbits() if that cpuset doesn't encompass that system-wide memory
     node.

10.51   cpuset_get_placement

   cpuset_get_placement(pid) - [optional Return current placement of task pid

     This is an [optional] function. Use cpuset_function to invoke it.

     This function returns an opaque struct placement * pointer. The results of
     calling cpuset_get_placement() twice at different points in a program can
     be compared by calling cpuset_equal_placement() to determine if the
     specified task has had its cpuset CPU and memory placement modified between
     those two cpuset_get_placement() calls.

     When finished with a struct placement * pointer, free it by calling
     cpuset_free_placement().

10.52   cpuset_equal_placement

   cpuset_equal_placement(plc1, plc2) - [optional] True if two placements equal

     This is an [optional] function. Use cpuset_function to invoke it.

     This function compares two struct placement * pointers, returned by two
     separate calls to cpuset_get_placement(). This is done to determine if the
     specified task has had its cpuset CPU and memory placement modified between
     those two cpuset_get_placement() calls.

     When finished with a struct placement * pointer, free it by calling
     cpuset_free_placement().

     Two struct placement * pointers will compare equal if they have the same CPU
     placement cpus, the same memory placement mems, and the same cpuset path.

10.53   cpuset_free_placement

   cpuset_free_placement(plc) - [optional] Free placement

     This is an [optional] function. Use cpuset_function to invoke it.

     Use this routine to free a struct placement * pointer returned by a previous
     call to cpuset_get_placement().

10.54   cpuset_fts_open

struct cpuset_fts_tree *cpuset_fts_open(const char *cpusetpath);

     Open a cpuset hierarchy. Returns a pointer to a cpuset_fts_tree structure,
     which can be used to traverse all cpusets below the specified cpuset
     cpusetpath.

     If the parameter cpusetpath starts with a slash (/) character, then this path
     is relative to /dev/cpuset, otherwise it is relative to the current tasks
     cpuset.

     The cpuset_fts_open routine is implemented internally using the fts(3)
     library routines for traversing a file hierarchy. The entire cpuset subtree
     below cpusetpath is traversed as part of the cpuset_fts_open() call, and
     all cpuset state and directory stat information is captured at that time. The
     other cpuset_fts_* routines just access this captured state. Any changes to the
     traversed cpusets made after the return of the cpuset_fts_open() call will
     not be visible via the returned cpuset_fts_tree structure.

     Internally, the fts(3) options FTS_NOCHDIR and FTS_XDEV are used, to avoid
     changing the invoking tasks current directory, and to avoid descending into any
     other file systems mounted below /dev/cpuset. The order in which cpusets will
     be returned by the cpuset_fts_read routine corresponds to the fts
     pre-order (FTS_D) visitation order. The internal fts scan by
     cpuset_fts_open ignores the post-order (FTS_DP) results.

     Because the cpuset_fts_open() call collects all the information at once
     from an entire cpuset subtree, a simple error return would not provide
     sufficient information as to what failed, and on what cpuset in the subtree.
     So, except for malloc(3) failures, errors are captured in the list of entries.

     See cpuset_fts_get_info for details of the info field.

     This is an [optional] function. Use cpuset_function to invoke it.

10.55   cpuset_fts_read

const struct cpuset_fts_entry *cpuset_fts_read(struct cpuset_fts_tree *cs_tree);

     Returns next cs_entry in cpuset_fts_tree cs_tree obtained from an
     cpuset_fts_open() call. One cs_entry is returned for each cpuset directory
     that was found in the subtree scanned by the cpuset_fts_open() call. Use
     the info field obtained from a cpuset_fts_get_info() call to determine
     which fields of a particular cs_entry are valid, and which fields contain error
     information or are not valid.

     This is an [optional] function. Use cpuset_function to invoke it.

10.56   cpuset_fts_reverse

void cpuset_fts_reverse(struct cpuset_fts_tree *cs_tree);

     Reverse order of cs_entry's in the cpuset_fts_tree cs_tree obtained from a
     cpuset_fts_open() call.

     An open cpuset_fts_tree stores a list of cs_entry cpuset entries, in pre-order,
     meaning that a series of cpuset_fts_read() calls will always return a
     parent cpuset before any of its child cpusets. Following a
     cpuset_fts_reverse() call, the order of cpuset entries is reversed,
     putting it in post-order, so that a series of cpuset_fts_read() calls will
     always return any children cpusets before their parent cpuset. A second
     cpuset_fts_reverse() call would put the list back in pre-order again.

     To avoid exposing confusing inner details of the implementation across the API,
     a cpuset_fts_rewind() call is always automatically performed on a
     cpuset_fts_tree whenever cpuset_fts_reverse() is called on it.

     This is an [optional] function. Use cpuset_function to invoke it.

10.57   cpuset_fts_rewind

void cpuset_fts_rewind(struct cpuset_fts_tree *cs_tree);

     Rewind a cpuset tree cs_tree obtained from a cpuset_fts_open() call, so
     that subsequent cpuset_fts_read() calls start from the beginning again.

     This is an [optional] function. Use cpuset_function to invoke it.

10.58   cpuset_fts_get_path

const char *cpuset_fts_get_path(const struct cpuset_fts_entry *cs_entry);

     Return the cpuset path, relative to /dev/cpuset, as nul-terminated string, of a
     cs_entry obtained from a cpuset_fts_read() call.

     The results of this call are valid for all cs_entry's returned from
     cpuset_fts_read() calls, regardless of the value returned by
     cpuset_fts_get_info() for that cs_entry.

     This is an [optional] function. Use cpuset_function to invoke it.

10.59   cpuset_fts_get_stat

const struct stat *cpuset_fts_get_stat(const struct cpuset_fts_entry *cs_entry);

     Return pointer to stat(2) information about the cpuset directory of a cs_entry
     obtained from a cpuset_fts_read() call.

     The results of this call are valid for all cs_entry's returned from
     cpuset_fts_read() calls, regardless of the value returned by
     cpuset_fts_get_info() for that cs_entry, except in the cases that:

     * the info field returned by cpuset_fts_get_info contains
       CPUSET_FTS_ERR_DNR, in which case, a directory in the path to the cpuset could
       not be read and this call will return a NULL pointer, or
     * the info field returned by cpuset_fts_get_info contains
       CPUSET_FTS_ERR_STAT, in which case a stat(2) failed on this cpuset directory
       and this call will return a pointer to a struct stat containing all zeros.

     This is an [optional] function. Use cpuset_function to invoke it.

10.60   cpuset_fts_get_cpuset

const struct cpuset *cpuset_fts_get_cpuset(const struct cpuset_fts_entry *cs_entry);

     Return the struct cpuset pointer of a cs_entry obtained from a
     cpuset_fts_read() call. The struct cpuset so referenced describes the
     cpuset represented by one directory in the cpuset hierarchy, and can be used
     with various other calls in this library.

     The results of this call are only valid for a cs_entry if the
     cpuset_fts_get_info() call returns CPUSET_FTS_CPUSET for the info field of
     a cs_entry. If the info field contained CPUSET_FTS_ERR_CPUSET, then
     cpuset_fts_get_cpuset returns a pointer to a struct cpuset that is all
     zeros. If the info field contains any other CPUSET_FTS_ERR_* value, then
     cpuset_fts_get_cpuset returns a NULL pointer.

     This is an [optional] function. Use cpuset_function to invoke it.

10.61   cpuset_fts_get_errno

int cpuset_fts_get_errno(const struct cpuset_fts_entry *cs_entry);

     Return the err field of a cs_entry obtained from a cpuset_fts_read() call.

     If an entry (obtained from cpuset_fts_read) has one of the
     CPUSET_FTS_ERR_* values in the info field (as described in
     cpuset_fts_get_info), then this err field captures the failing errno value
     for that operation. If an entry has the value CPUSET_FTS_CPUSET in its info
     field, then this err field will have the value 0.

     This is an [optional] function. Use cpuset_function to invoke it.

10.62   cpuset_fts_get_info

int cpuset_fts_get_info(const struct cpuset_fts_entry *cs_entry);

     Return the info field of a cs_entry obtained from a cpuset_fts_read()
     call.

     If this info field has one of the following CPUSET_FTS_ERR_* values, then it
     indicates which operation failed, the err field (returned by
     cpuset_fts_get_errno) captures the failing errno value for that operation,
     the path field (returned by cpuset_fts_get_path) indicates which cpuset
     failed, and some of the other entry fields may not be valid, depending on the
     value. If an entry has the value CPUSET_FTS_CPUSET for its info field, then the
     err field will have the value 0, and the other fields will be contain valid
     information about that cpuset.

     info field values:

   CPUSET_FTS_CPUSET = 0:
          Valid cpuset

   CPUSET_FTS_ERR_DNR = 1:
          Error - couldn't read directory

   CPUSET_FTS_ERR_STAT = 2:
          Error - couldn't stat directory

   CPUSET_FTS_ERR_CPUSET = 3:
          Error - cpuset_query failed

     The above info field values are defined using an anonymous enum in the cpuset.h
     header file. If it necessary to maintain source code compatibility with earlier
     versions of the cpuset.h header file lacking the above CPUSET_FTS_* values, one
     can conditionally check that the C preprocessor symbol
     CPUSET_FTS_INFO_VALUES_DEFINED is not defined and provide alternative coding
     for that case.

     This is an [optional] function. Use cpuset_function to invoke it.

10.63   cpuset_fts_close

void cpuset_fts_close(struct cpuset_fts_tree *cs_tree);

     Close a cs_tree obtained from a cpuset_fts_open() call, freeing any
     internally allocated memory for that cs_tree.

     This is an [optional] function. Use cpuset_function to invoke it.

10.64   cpuset_cpubind

int cpuset_cpubind(int cpu);

     Bind the scheduling of the current task to CPU cpu, using the
     sched_setaffinity(2) system call.

     Fails with a return of -1, and errno set to EINVAL, if cpu is not allowed in
     the current tasks cpuset.

     The following code will bind the scheduling of a thread to the n-th CPU of the
     current cpuset:

/*
 * Confine current task to only run on the n-th CPU
 * of its current cpuset. In a cpuset of N CPUs,
 * valid values for n are 0 .. N-1.
 */
cpuset_cpubind(cpuset_p_rel_to_sys_cpu(0, n));

10.65   cpuset_latestcpu

int cpuset_latestcpu(pid_t pid);

     Return the most recent CPU on which the specified task pid executed. If pid is
     0, examine current task.

     The cpuset_latestcpu() call returns the number of the CPU on which the
     specified task pid most recently executed. If a process can be scheduled on two
     or more CPUs, then the results of cpuset_lastcpu() may become invalid even
     before the query returns to the invoking user code.

     The last used CPU is visible for a given pid as field #39 (starting with #1) in
     the file /proc/pid/stat. Currently this file has 41 fields, so its the 3rd to
     the last field.

10.66   cpuset_membind

int cpuset_membind(int mem);

     Bind the memory allocation of the current task to memory node mem, using the
     set_mempolicy(2) system call with a policy of MPOL_BIND.

     Fails with a return of -1, and errno set to EINVAL, if mem is not allowed in
     the current tasks cpuset.

     The following code will bind the memory allocation of a thread to the n-th
     memory node of the current cpuset:

/*
 * Confine current task to only allocate memory on
 * n-th node of its current cpuset.  In a cpuset of
 * N memory nodes, valid values for n are 0 .. N-1.
 */
cpuset_membind(cpuset_p_rel_to_sys_mem(0, n));

10.67   cpuset_export

int cpuset_export(const struct cpuset *cp, char *buf, int buflen);

     Write the settings of cpuset cp to file. If no file exists at the path
     specified by file, create one. If a file already exists there, overwrite it.

     Returns -1 and sets errno on error. Upon successful return, returns the number
     of characters printed (not including the trailing '0' used to end output to
     strings). The function cpuset_export does not write more than size bytes
     (including the trailing '0'). If the output was truncated due to this limit
     then the return value is the number of characters (not including the trailing
     '0') which would have been written to the final string if enough space had been
     available. Thus, a return value of size or more means that the output was
     truncated.

     See Cpuset Text Format for details of the format of an exported cpuset
     file.

10.68   cpuset_import

int cpuset_import(struct cpuset *cp, const char *file, int *errlinenum_ptr, char
*errmsg_bufptr, int errmsg_buflen);

     Read the settings of cpuset cp from file. If no file exists at the path
     specified by file, create one. If a file already exists there, overwrite it.

     Struct cpuset *cp must have been returned by a previous cpuset_alloc() call.
     Any previous settings of cp are lost.

     Returns 0 on success, or -1 on error, setting errno. Errors include file not
     referencing a readable file.

     If parsing errors are encountered reading the file, and if errlinenum_ptr is
     not NULL, then the number of the first line (numbers start with one) with an
     error is written to *errlinenum_ptr. If an error occurs on the open, and
     errlinenum_ptr is not NULL, then zero is written to *errlinenum_ptr.

     If parsing errors are encountered reading the file and if errmsg_bufptr is not
     NULL, then it is presumed to point to a character buffer of at least
     errmsg_buflen characters, and a nul terminated error message is written to
     *errmsg_bufptr providing a human readable error message explaining the error
     message in more detail. As of this writing, the possible error messages are:

     * "Token 'CPU' requires list"
     * "Token 'MEM' requires list"
     * "Invalid list format: %s"
     * "Unrecognized token: %s"
     * "Insufficient memory"

     See Cpuset Text Format for details of the format required for imported
     cpuset file.

10.69   cpuset_function

cpuset_function(const char *function_name);

     Return pointer to the named libcpuset.so function, or NULL. For base functions
     that are in all implementations of libcpuset, there is no particular value in
     using cpuset_function() to obtain a pointer to the function dynamically. But
     for [optional] cpuset functions, the use of cpuset_function() enables
     dynamically adapting to run-time environments that may or may not support that
     function.

     For more information, see the Extensible API section, above.

11   System Error Numbers

   The Linux kernel implementation of cpusets sets errno to specify the reason for a
   failed system call affecting cpusets. These errno values are available when a
   cpuset library call fails. Most of these values can also be displayed by shell
   commands used to directly manipulate files below /dev/cpuset.

   The possible errno settings and their meaning when set on a failed cpuset call are
   as listed below.

   ENOSYS
          Invoked on an operating system kernel that does not support cpusets.

   ENODEV
          Invoked on a system that supports cpusets, but when the cpuset file system
          is not currently mounted at /dev/cpuset.

   ENOMEM
          Insufficient memory is available.

   EBUSY
          Attempted cpuset_delete() on a cpuset with attached tasks.

   EBUSY
          Attempted cpuset_delete() on a cpuset with child cpusets.

   ENOENT
          Attempted cpuset_create() in a parent cpuset that doesn't exist.

   EEXIST
          Attempted cpuset_create() for a cpuset that already exists.

   EEXIST
          Attempted to rename(2) a cpuset to a name that already exists.

   ENOTDIR
          Attempted to rename(2) a non-existent cpuset.

   E2BIG
          Attempted a write(2) system call on a special cpuset file with a length
          larger than some kernel determined upper limit on the length of such
          writes.

   ESRCH
          Attempted to cpuset_move() a non-existent task.

   EACCES
          Attempted to cpuset_move() a task which one lacks permission to move.

   EACCES
          Attempted to write(2) a memory_pressure file.

   ENOSPC
          Attempted to cpuset_move() a task to an empty cpuset.

   EINVAL
          The relcpu argument to cpuset_pin() is out of range (not between "zero" and
          "cpuset_size() - 1").

   EINVAL
          Attempted to change a cpuset in a way that would violate a cpu_exclusive or
          mem_exclusive attribute of that cpuset or any of its siblings.

   EINVAL
          Attempted to write an empty cpus or mems bitmask to the kernel. The kernel
          creates new cpusets (via mkdir) with empty cpus and mems, and the user
          level cpuset and bitmask code works with empty masks. But the kernel will
          not allow an empty bitmask (no bits set) to be written to the special cpus
          or mems files of a cpuset.

   EIO
          Attempted to write(2) a string to a cpuset tasks file that does not begin
          with an ASCII decimal integer.

   EIO
          Attempted to rename(2) a cpuset outside of its current directory.

   ENOSPC
          Attempted to write(2) a list to a cpus file that did not include any online
          cpus.

   ENOSPC
          Attempted to write(2) a list to a mems file that did not include any online
          memory nodes.

   EACCES
          Attempted to add a cpu or mem to a cpuset that is not already in its
          parent.

   EACCES
          Attempted to set cpu_exclusive or mem_exclusive on a cpuset whose parent
          lacks the same setting.

   ENODEV
          The cpuset was removed by another task at the same time as a write(2) was
          attempted on one of the special files in the cpuset directory.

   EBUSY
          Attempted to remove a cpu or mem from a cpuset that is also in a child of
          that cpuset.

   EFAULT
          Attempted to read or write a cpuset file using a buffer that was outside
          your accessible address space.

   ENAMETOOLONG
          Attempted to read a /proc/<pid>/cpuset file for a cpuset path that was
          longer than the kernel page size.

   ENAMETOOLONG
          Attempted to create a cpuset whose base directory name was longer than 255
          characters.

   ENAMETOOLONG
          Attempted to create a cpuset whose full pathname including the
          "/dev/cpuset" is longer than 4095 characters.

   ENXIO
          Attempted to create a cpu_exclusive cpuset whose cpus covered just part of
          one or more physical processor packages, such as including just one of the
          two Cores on a package. For Linux kernel version 2.6.16 on i386 and x86_64,
          this operation is rejected with this error to avoid a fatal kernel bug.
          Otherwise, this is a normal and supported operation.

   EINVAL
          Specified a cpus or mems list to the kernel which included a range with the
          second number smaller than the first number.

   EINVAL
          Specified a cpus or mems list to the kernel which included an invalid
          character in the string.

   ERANGE
          Specified a cpus or mems list to the kernel which included a number too
          large for the kernel to set in its bitmasks.

   ETIME
          Time limit for cpuset_nuke operation reached without successful
          completion of operation.

   ENOTEMPTY
          Tasks remain after multiple attempts by cpuset_move_cpuset_tasks to
          move them to a different cpuset.

   EPERM
          Lacked permission to kill (send a signal to) a task.

   EPERM
          Lacked permission to read a cpuset or its files.

   EPERM
          Attempted to unlink a per-cpuset file. Such files can not be unlinked. They
          can only be removed by removing (rmdir) the directory representing the
          cpuset that contains these files.

12   Change History

   Here is the history of changes to this document and associated software.
     * 2006-04-26 Paul Jackson <pj@sgi.com>

          + First published.

     * 2006-11-14 Paul Jackson <pj@sgi.com> Version 1 (cpuset_version())

          + Added cpuset_nuke routine.
          + Added various cpuset_fts_* routines.
          + Added various cpuset_*_affinity routines.
          + Added cpuset_move_cpuset_tasks routine.
          + Converted from using ftw(3) to fts(3).
          + Various minor documentation errors fixed.
          + Added cpuset_version routine.
          + Improved error checking by cpuset_move_all.
          + Correct system call numbering for sched_setaffinity on i386 arch.
          + Correct cpuset_latestcpu result if command basename has space
            character.
          + Remove 256 byte limit on cpuset_export output.

     * 2007-01-14 Paul Jackson <pj@sgi.com> Version 2 (cpuset_version())

          + Fix cpuset_create, cpuset_modify to not zero undefined
            attributes such as memory_spread_page, memory_spread_slab and
            notify_on_release. See further the explanation of an "undefined mark", in
            Cpuset Programming Model. cpuset_create now respects default
            kernel cpuset creation settings, and cpuset_modify now respects the
            existing settings of the cpuset, unless explicitly set.

     * 2007-04-01 Paul Jackson <pj@sgi.com> Version 3 (cpuset_version())

          + Fix cpuset_setcpus() and cpuset_setmems() to mark the cpus and
            mems fields as defined, so that setting them before doing a
            cpuset_create() has affect.