Intel Knights Landing (KNL) User and Administrator Guide
Overview
This document describes the unique features of Slurm on the computers with the Intel Knights Landing processor. You should be familiar with the Slurm's mode of operation on Linux clusters before studying the relatively few differences in Intel KNL system operation described in this document.
User Tools
The desired NUMA and MCDRAM modes for a KNL processor should be specified using the -C or --constraints option of Slurm's job submission commands: salloc, sbatch, and srun. Currently available NUMA and MCDRAM modes are shown in the table below. Each node's available and current NUMA and MCDRAM modes are visible in the "available features" and "active features" fields respectively, which may be seen using the scontrol, sinfo, or sview commands. Note that a node may need to be rebooted to get the desired NUMA and MCDRAM modes and nodes may only be rebooted when they contain no running jobs (i.e. sufficient resources may be available to run a pending job, but until the node is idle and can be rebooted, the pending job may not be allocated resources). Also note that the job will be charged for resources from the time of resource allocation, which may include time to reboot a node into the desired NUMA and MCDRAM configuration.
Slurm supports a very rich set of options for the node constraint options (exclusive OR, node counts for each constraint, etc.). See the man pages for the salloc, sbatch and srun commands for more information about the constraint syntax. Jobs may specify their desired NUMA and/or MCDRAM configuration. If no NUMA and/or MCDRAM configuration is specified, then a node with any possible value for that configuration will be used.
| Type | Name | Description |
|---|---|---|
| MCDRAM | cache | All of MCDRAM to be used as cache |
| MCDRAM | equal | MCDRAM to be used partly as cache and partly combined with primary memory |
| MCDRAM | flat | MCDRAM to be combined with primary memory into a "flat" memory space |
| NUMA | a2a | All to all |
| NUMA | hemi | Hemisphere |
| NUMA | snc2 | Sub-NUMA cluster 2 |
| NUMA | snc4 | Sub-NUMA cluster 4 (NOTE) |
| NUMA | quad | Quadrant (NOTE) |
Jobs requiring some or all of the KNL high bandwidth memory (HBM) should explicitly request that memory using Slurm's Generic RESource (GRES) options. The HBM will always be known by Slurm GRES name of "hbm". Examples below demonstrate use of HBM.
Sorting of the free cache pages at step startup using Intel's zonesort module can be configred as the default for all steps using the "LaunchParameters=mem_sort" option in the slurm.conf file. Individual job steps can enable or disable sorting using the "--mem-bind=sort" or "--mem-bind=nosort" command line options for srun. Sorting will be performed only for the NUMA nodes allocated to the job step.
NOTE: Slurm provides limited support for restricting use of HBM. At some point in the future, the amount of HBM requested by the job will be compared with the total amount of HBM and number of memory-containing NUMA nodes available on the KNL processor. The job will then be bound to specific NUMA nodes in order to limit the total amount of HBM available to the job, and thus reserve the remaining HBM for other jobs running on that KNL processor.
NOTE: Slurm can only support homogeneous nodes (e.g. the same number of cores per NUMA node). KNL processors with 68 cores (a subset of KNL models) will not have homogeneous NUMA nodes in snc4 mode, but each NUMA node will have either 16 or 18 cores. This will result in Slurm using the lower core count, finding a total of 256 threads rather than 272 threads and setting the node to a DOWN state.
Accounting
If a node requires rebooting for a job's required configuration, the job will be charged for the resource allocation from the time of allocation through the lifetime of the job, including the time consumed for booting the nodes. The job's time limit will be calculated from the time that all nodes are ready for use. For example, a job with a 10 minute time limit may be allocated resources at 10:00:00. If the nodes require rebooting, they might not be available for use until 10:20:00, 20 minutes after allocation, and the job will begin execution at that time. The job must complete no later than 10:30:00 in order to satisfy its time limit (10 minutes after execution actually begins). However, the job will be charged for 30 minutes of resource use, which includes the boot time.
Sample Use Cases
$ sbatch -C flat,a2a -N2 --gres=hbm:8g --exclusive my.script $ srun --constraint=hemi,cache -n36 a.out $ srun --constraint=flat --gres=hbm:2g -n36 a.out $ sinfo -o "%30N %20b %f" NODELIST ACTIVE_FEATURES AVAIL_FEATURES nid000[10-11] nid000[12-35] flat,a2a flat,a2a,snc2,hemi nid000[36-43] cache,a2a flat,equal,cache,a2a,hemi
Network Topology
Slurm will optimize performance using those resources available without rebooting. If node rebooting is required, then it will optimize layout with respect to network bandwidth using both nodes currently in the desired configuration and those which can be made available after rebooting. This can result in more nodes being rebooted than strictly needed, but will improve application performance.
Users can specify they want all resources allocated on a specific count of leaf switches (Dragonfly group) using Slurm's --switches option. They can also specify how much additional time they are willing to wait for such a configuration. If the desired configuration can not be made available within the specified time interval, the job will be allocated nodes optimized with respect to network bandwidth to the extent possible. On a Dragonfly network, this means allocating resources over either single group or distributed evenly over as many groups as possible. For example:
srun --switches=1@10:00 N16 a.out
Note that system administrators can disable use of the --switches option or limit the amount of time the job can be deferred using the SchedulerParameters max-switch-wait option.
Booting Problems
If node boots fail, those nodes are drained and the job is requeued so that it can be allocated a different set of nodes. The nodes originally allocated to the job will remain available to the job, so likely a small number of additional nodes will be required.
System Administration
Four important components are required to use Slurm on an Intel KNL system.
- Slurm needs a mechanism to determine the node's current topology (e.g. how many NUMA exist and which cores are associated with each NUMA). Slurm relies upon Portable Hardware Locality (HWLOC) for this functionality. Please install HWLOC before building Slurm.
- The node features plugin manages the available and active features information available for each KNL node.
- A configuration file is used to define various timeouts, default configuration, etc. The configuration file name and contents will depend upon the node features plugins used. See the knl.conf man page for more information.
- A mechanism is required to boot nodes in the desired configuration. This mechanism must be integrated with existing Slurm infrastructure for rebooting nodes on user request (--reboot) plus (for Cray systems only) power saving (powering down idle nodes and restarting them on demand).
In addition, there is a DebugFlags option of "NodeFeatures" which will generate detailed information about KNL operations.
The KNL-specific available and active features are configured differently
based upon the plugin configured.
For the knl_cray plugin, KNL-specific available and active features are
not included in the "slurm.conf" configuration file, but are set and the managed
by the NodeFeatures plugin when the slurmctld daemon starts.
For the knl_generic plugin, KNL-specific features should be defined
in the "slurm.conf" configuration file. When the slurmd daemon starts on each
compute node, it will update the available and active features as needed.
Features which are not KNL-specific (e.g. rack number, "knl", etc.) will be
copied from the node's "Features" configuration in "slurm.conf" to both the
available and active feature fields and not modified by the NodeFeatures
plugin.
NOTE: For Dell KNL systems you must also include the SystemType=Dell option for successful operation and will likely need to increase the SyscfgTimeout to allow enough time for the command to successfully complete. Experience at one site has shown that a 10 second timeout may be necessary, configured as SyscfgTimeout=10000.
Slurm does not support the concept of multiple NUMA nodes within a single socket. If a KNL node is booted with multiple NUMA, then each NUMA node will appear in Slurm as a separate socket. In the slurm.conf configuration file, set node socket and core counts to values which are appropriate for some NUMA mode to be used on the node. When the node boots and the slurmd daemon on the node starts, it will report to the slurmctld daemon the node's actual socket (NUMA) and core counts, which will update Slurm data structures for the node to the values which are currently configured. Note that Slurm currently does not support the concept of differing numbers of cores in each socket (or NUMA node). We are currently working to address these issues.
Mode of Operation
- The node's configured "Features" are copied to the available and active feature fields.
- The node features plugin determines the node's current MCDRAM and NUMA values as well as those which are available and adds those values to the node's active and available feature fields respectively. Note that these values may not be available until the node has booted and the slurmd daemon on the compute node sends that information to the slurmctld daemon.
- Jobs will be allocated nodes already in the requested MCDRAM and NUMA mode if possible. If insufficient resources are available with the requested configuration then other nodes will be selected and booted into the desired configuration once no other jobs are active on the node. Until a node is idle, its configuration can not be changed. Note that node reboot time is roughly on the order of 20 minutes.
Cray Configuration
On Cray systems, NodeFeaturesPlugins should be configured as "knl_cray".
The configuration file will be named "knl_cray.conf". The file will include the path to the capmc program (CapmcPath), which is used to get a node's available MCDRAM and NUMA modes, change the modes, power the node down, reboot it, etc. Note the "CapmcTimeout" parameter is the time required for the capmc program to respond to a request and NOT the time for a boot operation to complete.
Power saving mode is integrated with rebooting nodes in the desired mode. Programs named "capmc_resume" and "capmc_suspend" are provided to boot nodes in the desired mode. The programs are included in the main Slurm RPM and installed in the "sbin" directory and must be installed on the Cray "sdb" node. If powering down of idle nodes is not desired, then configure "ResumeProgram" in "slurm.conf" to the path of the "capmc_resume" file and configure "SuspendTime" to a huge value (e.g. "SuspendTime=30000000" will only power down a node which has been idle for about one year).
Note that getting a compute node's current MCDRAM and NUMA mode, modifying its MCDRAM and NUMA mode, and rebooting it are operations performed by the slurmctld daemon on the head node.
The GresTypes configuration parameter should include "hbm" to identify High Bandwidth Memory (HBM) as being a consumable resources on compute nodes. Additional GresTypes can be specified as needed in a comma separated list. The amount of HBM on each node should not be configured in a Slurm configuration file, but that information will be loaded by the knl_cray plugin using information provided by the capmc program.
Sample knl_cray.conf file
# Sample knl_cray.conf CapmcPath=/opt/cray/capmc/default/bin/capmc CapmcTimeout=2000 # msec DefaultNUMA=a2a # NUMA=all2all AllowNUMA=a2a,snc2,hemi DefaultMCDRAM=cache # MCDRAM=cache
Sample slurm.conf File
# Sample slurm.conf NodeFeaturesPlugins=knl_cray DebugFlags=NodeFeatures GresTypes=hbm # ResumeProgram=/opt/slurm/default/sbin/capmc_resume SuspendProgram=/opt/slurm/default/sbin/capmc_suspend SuspendTime=30000000 ResumeTimeout=1800 ... Nodename=default Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000 Feature=knl NodeName=nid[00000-00127] State=UNKNOWN
Generic Cluster Configuration
All other clusters should have NodeFeaturesPlugins configured to "knl_generic". This plugin performs all operations directly on the compute nodes using Intel's syscfg program to get and modify the node's MCDRAM and NUMA mode and uses the Linux reboot program to reboot the compute node in order for modifications in MCDRAM and/or NUMA mode to take effect. Make sure that RebootProgram is defined in the slurm.conf file. This plugin currently does not permit the specification of ResumeProgram, SuspendProgram, SuspendTime, etc. in slurm.conf, however that limitation may be removed in the future (the ResumeProgram currently has no means of changing the node's MCDRAM and/or NUMA mode prior to reboot).
NOTE:The syscfg program reports the MCDRAM and NUMA mode to be used when the node is next booted. If the syscfg program is used to modify the MCDRAM or NUMA mode of a node, but it is not rebooted, then Slurm will be making scheduling decisions based upon incorrect state information. If you want to change node state information outside of Slurm then use the following procedure:
- Drain the nodes of interest
- Change their MCDRAM and/or NUMA mode
- Reboot the nodes, then
- Restore them to service in Slurm
Sample knl_generic.conf File
# Sample knl_generic.conf SyscfgPath=/usr/bin/syscfg DefaultNUMA=a2a # NUMA=all2all AllowNUMA=a2a,snc2,hemi DefaultMCDRAM=cache # MCDRAM=cache
Sample slurm.conf File
# Sample slurm.conf NodeFeaturesPlugins=knl_generic DebugFlags=NodeFeatures GresTypes=hbm RebootProgram=/sbin/reboot ... Nodename=default Sockets=1 CoresPerSocket=68 ThreadsPerCore=4 RealMemory=128000 Feature=knl NodeName=nid[00000-00127] State=UNKNOWN
Last modified 9 October 2019