1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
|
The essential purpose of cpusets is to provide CPU and Memory containers or
"soft partitions" within which to run sets of related tasks.
On an SMP (multiple CPU) system without some means of CPU placement, any task
can run on any CPU. On a NUMA (multiple Memory Node) system, any memory page can
be allocated on any node. This can cause both poor cache locality and poor
memory access times, substantially reducing performance and runtime
repeatability. By restraining all other jobs from using any of the CPUs or
Memory Nodes assigned to critical jobs, interference with critical jobs can be
minimized.
For example, MPI jobs are frequently consist of a number of threads that
communicate via message passing interfaces. All threads need to be executing at
the same time. If a single thread loses a cpu, all threads stop making forward
progress and spin at a barrier. Cpusets can eliminate the need for a gang
scheduler.
The libcpuset library depends on the following Linux 2.6 kernel
facilities:
* sched_setaffinity (for CPU binding),
* mbind and set_mempolicy (for Memory Node binding), and
* kernel cpusets support.
The sched_setaffinity, mbind and set_mempolicy calls enable specifying
the CPU and Memory placement for individual tasks. On smaller or limited
use systems, these calls may be sufficient.
The kernel cpuset facility provides additional support for system wide
management of CPU and Memory resources, by related sets of tasks.
It provides a hierarchical structure to the resources, with file system
like namespace and permissions, and support for guaranteed exclusive use
of resources.
The Linux kernel provides the following support for cpusets:
* Each task has a link to a cpuset structure that specifies the
CPUs and Memory Nodes available for its use.
* A hook in the sched_setaffinity and mbind system calls ensures that
any requested CPU or Memory Node is available in that tasks cpuset.
* tasks sharing the same placement constraints reference the same cpuset.
* These kernel cpusets are arranged in a hierarchical virtual file system,
reflecting the possible nesting of "soft partitions".
* The kernel task scheduler is constrained to only schedule a task on
the CPUs in that tasks cpuset.
* The kernel memory allocator is constrained to only allocate physical
memory to a task from the Memory Nodes in that tasks cpuset.
A cpuset constrains the jobs (set of related tasks) running in it to a subset of
the systems memory and CPUs. They enable administrators and system service
software to:
* Create and delete named cpusets.
* Decide which CPUs and Memory Nodes are available to a cpuset.
* Attach a task to a particular cpuset.
* Identify all tasks sharing the same cpuset.
* Exclude any other cpuset from overlapping a given cpuset,
giving the tasks running in that cpuset exclusive use
of those CPUs and Memory Nodes.
* Perform bulk operations on all tasks associated with a cpuset,
such as varying the resources available to that cpuset, or
hibernating those tasks in temporary favor of some other job.
* Perform sub-partitioning with hierarchical permissions and
resource management.
Cpusets are exposed by the kernel to userspace by mounting the `cpuset`
virtual file system (VFS) at `/dev/cpuset`, rather than by additional
system calls. Such a VFS is a natural way to represent nested resource
allocations and the associated hierarchical permission model.
Within a single cpuset, other facilities such as `runon`, `dplace`,
first-touch memory placement, pthreads, `sched_setaffinity` and `mbind`
can be used to manage processor and memory placement to a more fine
grained level.
There is a single set of kernel mechanisms that supports all these
facilities and provides a consistent processor and memory model
regardless of what mix of utilities and API's you use to manage it.
This provides a consistent execution model for all users.
Dependencies
============
The libcpuset library depends on the libbitmask library.
License
=======
Libcpuset is licensed under the terms of the
GNU Lesser Public License (LGPL) - see the file
COPYING.LIB for details.
Paul Jackson
pj@sgi.com
12 Feb 2006
|
