Slurm Workload Manager - Hierarchical Resource (HRES) Scheduling

# Hierarchical Resource (HRES) Scheduling

## Overview

Hierarchical Resources (HRES) were added in Slurm 25.05 in a Beta state. While in beta, the configuration file format and other aspects of this feature may undergo substantial changes in future versions and require major changes to avoid errors. Sites that utilize this feature while in a beta state should pay particularly close attention to changes noted in RELEASE_NOTES and the CHANGELOGs when upgrading.

Hierarchical Resources allow for license-like resources to be defined as part of independent hierarchical topologies and associated with specific nodes. Jobs may request any integer count of that resource. Multiple resources may be defined in the configuration (in a resources.yaml file), and will use independently defined hierarchies.

Although these hierarchical topologies have some similarities to network topologies, the definitions for each are completely separate.

## Planning Modes

Three modes of resource planning are provided. In either case, a layer may be defined with a count of zero or infinite (`count: -1`) resources, with the impact of this depending on the mode used.

### Mode 1

In Mode 1, sufficient resources are only required on one layer that overlaps with job allocation. In many cases only a single level is needed. However, multiple levels may be defined if some additional flexibility in where resources are used is preferred, as described in the example below.

A layer with a zero count of a resource will have no impact on scheduling. Since that layer will never satisfy the request, it will never alter the calculated list of nodes eligible to run a given job. It would be semantically equivalent to removing that layer definition.

A layer with an infinite count of a resource will the always allow a new job allocation to succeed. Unless prevented due to other requirements, jobs will be able to execute immediately.

For example, consider the natural resource defined in the example resources.yaml file below. A job that requests this resource and is allocated `node[01-04]` will pull from the pool of 100 available to `node[01-16]`. With a count of 50 specified at the higher level (`node[01-32]`), they can be allocated in addition to the resources specified in each group of 16. That is, 100 each would be available to each group of 16, and 50 more would be available on any of them, for a total of 250 of this resource.

### Mode 2

In Mode 2, sufficient resources must be available on all layers on all levels that overlap with job allocation. Note that scheduling stalls are possible if higher levels do not make enough resources available.

A layer with a zero count of a resource will have a significant impact on scheduling. All nodes underneath that layer would never be able to satisfy the allocation constraints. This could be used to mark that resource unavailable for a specific portion of the cluster, e.g., for hardware maintenance, without altering the individual lower-layer resource counts.

A layer with an infinite count of a resource will have no impact on scheduling. Since that layer will always satisfy the request, it will never constrain execution and would be semantically equivalent to removing that layer definition.

For example, consider the flat resource defined in the example resources.yaml file below. A job that requests this resource and is allocated `node[01-04]` will pull from all levels that include those nodes. So it would pull from the 12 available on `node[01-08]`, from the 16 available on `node[01-16]`, and from the 24 available on `node[01-32]`. Since the highest level only contains 24, that is the maximum that can be allocated, even though a larger total is available on lower levels. This example allows a fixed set of resources to have some flexibility in where they are used.

### Mode 3

In Mode 3, resources are allocated on a per-node basis, with the total consumption being calculated from the most granular layer and summed up into the successively higher layers. This mode is specifically designed for consumable resources like power, where the total usage at a higher level (e.g., cluster-wide) is the sum of the usage of its components (e.g., chassis level, then rack level summing multiple chassis, then datacenter row ).

The scheduler ensures that the total summed resources at every layer in the hierarchy do not exceed that layer's defined count.

Important Note: Layers defined under Mode 3 must be able to be represented as a rooted uniform depth tree.

For example, consider the power resource defined in the example resources.yaml file below. A job that requests 1000 of the power resource and is allocated `node[01-16]` will pull from all levels that include those nodes. So it would pull 8000 from `node[01-08]`, 8000 from `node[09-16]`, 16000 from `node[01-16]`, and 16000 from `node[01-32]`.

## Limitations

- The resource counts cannot be dynamically changed; all changes must be made through resources.yaml and applied with a restart or reconfigure
- Dynamic nodes are not supported
- This implementation is in a beta state (see overview)
- Resource names defined through the hierarchical resources configuration must not conflict with any cluster licenses

## Examples

After defining a resource hierarchy in resources.yaml (see below for example), you can interact with these resources in several ways, using the syntax already used for licenses:

- View HRES

      scontrol show license

- Reserve HRES

      scontrol create reservation account=root licenses=natural(node[01-016]):30,flat(node[09-12]):4,flat(node[29-32]):2

  Note that the layer must be specified for each reserved HRES, identified by the node list present available through that layer
- Request HRES on a job

      sbatch --license=flat:2,natural:1 my_script.sh

### resources.yaml

    ---
    - resource: flat
      mode: MODE_2
      layers:
        - nodes:			# highest level
            - "node[01-32]"
          count: 24
        - nodes:			# middle levels
            - "node[01-16]"
          count: 16
        - nodes:
            - "node[17-32]"
          count: 16
        - nodes:			# lowest levels
            - "node[01-08]"
          count: 12
        - nodes:
            - "node[09-16]"
          count: 12
        - nodes:
            - "node[17-24]"
          count: 12
        - nodes:
            - "node[25-32]"
          count: 12
    - resource: natural
      mode: MODE_1
      layers:
        - nodes:			# highest level
            - "node[01-32]"
          count: 50
        - nodes:			# lowest levels
            - "node[01-16]"
          count: 100
        - nodes:
            - "node[17-32]"
          count: 100
    - resource: power
      mode: MODE_3
      topology: blok1
      variables:
        - name: full_node
          value: 1000
        - name: full_gpu_node
          value: 2000
      layers:
        - nodes:
            - "node[01-08]"
          count: 40000
          base:
            - name: storage
              value: 5000
        - nodes:
            - "node[17-24]"
          count: 40000
        - nodes:
            - "node[09-16]"
          count: 40000
        - nodes:
            - "node[25-32]"
          count: 40000
        - nodes:
            - "node[01-16]"
          count: 60000
          base:
            - name: network1
              value: 3000
        - nodes:
            - "node[17-32]"
          count: 80000
          base:
            - name: network2
              value: 2000
        - nodes:
            - "node[01-32]"
          count: 130000
          base:
            - name: acUnit1
              value: 10000
            - name: acUnit2
              value: 8000