Core Specialization

Core specialization is a feature designed to isolate system overhead (system interrupts, etc.) to designated cores on a compute node. This can reduce applications interrupts ranks to improve completion time. The job will be charged for all allocated cores, but will not be able to directly use the specialized cores.

Command Options

All job allocation commands (salloc, sbatch and srun) accept the -S or --core-spec option with a core count value argument (e.g. "-S 1" or "--core-spec=2"). The count identifies the number of cores to be reserved for system overhead on each allocated compute node. Each job's core specialization count can be viewed using the scontrol, sview or squeue command. Specification of a core specialization count for a job step is ignored (i.e. for the srun command within a job allocation created using the salloc or sbatch command). Use the squeue command with the "%X" format option to see the count (it is not reported in the default output format). The scontrol and sview commands can also be used to modify the count for pending jobs.

Explicitly setting a job's specialized core value implicitly sets its --exclusive option, reserving entire nodes for the job, however the number of CPUs available for the job's steps will not exceed that explicitly requested by the job. For example "salloc -n1 --core-spec=1 bash" will result in one entire node being reserved for the job, but only one CPU will be available for subsequent job steps. The remaining CPUs on the node (at least one core because of "--core-spec=1") will only be available for system use. The job will be charged for all CPUs on the node and the job's NumCPUs value reported by the scontrol, sview and squeue commands will reflect all CPUS on all allocated nodes as will the job's accounting.

Core Selection

The specific resources to be used for specialization may be identified using the CPUSpecList configuration parameter associated with each node in the slurm.conf file. Note that the core_spec/cray_aries does not currently support identification of specific cores, so that plugin should not be used in conjunction with the CPUSpecList configuration parameter, even on Cray systems. If CoreSpecCount is configured, but not CPUSpecList, the cores selected for specialization will follow the assignment algorithm described below . The first core selected will be the highest numbered core on the highest numbered socket. Subsequent cores selected will be the highest numbered core on lower numbered sockets. If additional cores are required, they will come from the next highest numbered cores on each socket. By way of example, consider a node with two sockets, each with four cores. The specialized cores will be selected in the following order:

  1. socket: 1 core: 3
  2. socket: 0 core: 3
  3. socket: 1 core: 2
  4. socket: 0 core: 2
  5. socket: 1 core: 1
  6. socket: 0 core: 1
  7. socket: 1 core: 0
  8. socket: 0 core: 0

Slurm can be configured to specialize the last, rather than the first cores by configuring SchedulerParameters=spec_cores_first. In that case, the first core selected will be the lowest numbered core on the lowest numbered socket. Subsequent cores selected will be the lowest numbered core on higher numbered sockets. If additional cores are required, they well come from the next lowest numbered cores on each socket.

Note that core specialization reservation may impact the use of some job allocation request options, especially --cores-per-socket.

System Configuration

There are two fundamentally different mechanisms for core specialization; one for Cray systems and a different model for other systems.

For Cray systems, configure SelectType=select/cray_aries and CoreSpecPlugin=core_spec/cray_aries. By default, no resources will be reserved for system use. The user must explicitly set a specialized core count as described above.

For all other systems, configure SelectType=select/cons_res, CoreSpecPlugin=core_spec/none and TaskPlugin=task/cgroup. In addition, specialized resources should be configured in slurm.conf on the node specification line using the CoreSpecCount or CPUSpecList option to identify the CPUs to reserve. The MemSpecLimit option can be used to reserve memory. These resources will be reserved using Linux cgroups. Users wanting a different number of specialized cores should use the --core-spec option as described above.

A job's core specialization option will be silently cleared on other configurations. In addition, each compute node's core count must be configured or the CPUs count must be configured to the node's core count. If the core count is not configured and the CPUs value is configured to the count of hyperthreads, then hyperthreads rather than cores will be reserved for system use.

If users are to be granted the right to control the number of specialized cores for their job, the configuration parameter AllowSpecResourceUsage must be set to a value of 1.

Last modified 6 September 2018