10.4. Scheduling processes across hosts

Open MPI provides many options for scheduling application processes across hosts, including oversubscribing processes to processors. This section describes how to define that mapping.

10.4.1. Scheduling overview

If you are not oversubscribing your hosts (i.e., trying to run more processes than slots available on that host), scheduling is pretty simple and occurs either on a by-slot or by-node round robin schedule. If you’re oversubscribing, the issue gets much more complicated — keep reading.

The more complete answer is: Open MPI schedules processes to nodes by asking two questions from each application on the mpirun command line:

How many processes should be launched?
Where should those processes be launched?

The “how many” question is directly answered with the -n switch to mpirun. If -n is not specified on the mpirun command line, its value is the sum of the slots on all the nodes.

The “where” question is a little more complicated, and depends on three factors:

The final node list (e.g., after -hostname / --host exclusionary or inclusionary processing)
The scheduling policy (which applies to all applications in a single job)
The default and maximum number of slots on each host

Open MPI currently supports two scheduling policies: by slot and by node:

By slot: This is the default scheduling policy, but can also be explicitly requested by using either the --map-by slot option to mpirun or by setting the MCA parameter rmaps_default_mapping_policy to the string slot.

In this mode, Open MPI will schedule processes on a node until all of its default slots are exhausted before proceeding to the next node. In MPI terms, this means that Open MPI tries to maximize the number of adjacent ranks in MPI_COMM_WORLD on the same host without oversubscribing that host.

For example:
```
shell$ cat my-hosts
node0 slots=2 max_slots=20
node1 slots=2 max_slots=20
shell$ mpirun --hostfile my-hosts -n 8 --map-by slot hello | sort
Hello World I am rank 0 of 8 running on node0
Hello World I am rank 1 of 8 running on node0
Hello World I am rank 2 of 8 running on node1
Hello World I am rank 3 of 8 running on node1
Hello World I am rank 4 of 8 running on node0
Hello World I am rank 5 of 8 running on node0
Hello World I am rank 6 of 8 running on node1
Hello World I am rank 7 of 8 running on node1
```

By node: This policy can be requested either by using the --map-by node option to mpirun or by setting the MCA parameter rmaps_default_mapping_policy to the string “node”.

In this mode, Open MPI will schedule a single process on each node in a round-robin fashion (looping back to the beginning of the node list as necessary) until all processes have been scheduled. Nodes are skipped once their default slot counts are exhausted.

For example:

shell$ cat my-hosts
node0 slots=2 max_slots=20
node1 slots=2 max_slots=20
shell$ mpirun --hostname my-hosts -n 8 --map-by node hello | sort
Hello World I am rank 0 of 8 running on node0
Hello World I am rank 1 of 8 running on node1
Hello World I am rank 2 of 8 running on node0
Hello World I am rank 3 of 8 running on node1
Hello World I am rank 4 of 8 running on node0
Hello World I am rank 5 of 8 running on node1
Hello World I am rank 6 of 8 running on node0
Hello World I am rank 7 of 8 running on node1

In both policies, if the default slot count is exhausted on all nodes while there are still processes to be scheduled, Open MPI will trigger an oversubscription condition.

If :OVERSUBSCRIBE is added as a modifier to the --map-by option (e.g., mpirun --map-by node:OVERSUBSCRIBE ... — see this section for more details), Open MPI will continue to loop through the list of nodes again and try to schedule one more process to each node until all processes are scheduled. Nodes are skipped in this process if their maximum slot count is exhausted. If the maximum slot count is exhausted on all nodes while there are still processes to be scheduled, Open MPI will abort without launching any processes.

If :OVERSUBSCRIBE is not specified and an oversubscription condition occurs, Open MPI will abort without launching any processes.

10.4.2. Scheduling with the –hostfile option

The --hostfile option to mpirun takes a filename that lists hosts on which to launch MPI processes.

Important

The hosts listed in a hostfile have nothing to do with which network interfaces are used for MPI communication. They are only used to specify on which hosts to launch MPI processes.

Hostfiles are simple text files with hosts specified, one per line. Each host can also specify a default and maximum number of slots to be used on that host (i.e., the maximum number of processes that will be launched on that node). Comments are also supported, and blank lines are ignored. For example:

# This is an example hostfile.  Comments begin with #.
#
# Since no slots are specified, the number of slots defaults to the
# number of processor cores available on the machine.
foo.example.com

# We want to allow launching a maximum of 2 processes on this host
# (e.g., potentially because it has two processor cores):
bar.example.com slots=2

Slots are discussed in much more detail in this section.

Hostfiles works in two different ways:

Exclusionary: If a list of hosts to run on has been provided by another source (e.g., by a hostfile or a batch scheduler such as Slurm, PBS/Torque, SGE, etc.), the hosts provided by the hostfile must be in the already-provided host list. If the hostfile-specified nodes are not in the already-provided host list, mpirun will abort without launching anything.

In this case, hostfiles act like an exclusionary filter — they limit the scope of where processes will be scheduled from the original list of hosts to produce a final list of hosts.

For example, say that a scheduler job contains hosts node01 through node04. If you run:
```
shell$ cat my_hosts
node03
shell$ mpirun -n 1 --hostfile my_hosts hostname
```
This will run a single copy of hostname on the host node03.

However, presuming your job was allocated only to node03 and you run the following:
```
shell$ cat my_hosts
node17
shell$ mpirun -n 1 --hostfile my_hosts hostname
```
This is an error (because node17 is not allocated to your job), and mpirun will abort.

Finally, note that in exclusionary mode, processes will only be executed on the hostfile-specified hosts, If this ends up causing an oversubscription situation, mpirun will abort by default.
Inclusionary: If a list of hosts has not been provided by another source, then the hosts provided by the --hostfile option will be used as the original and final host list.

In this case, --hostfile acts as an inclusionary agent; all --hostfile-supplied hosts become available for scheduling processes. For example (assume that you are not in a scheduling environment where a list of nodes is being transparently supplied):
```
shell$ cat my_hosts
node01.example.com slots=1
node02.example.com slots=1
node03.example.com slots=1
shell$ mpirun -n 3 --hostfile my_hosts hostname
```
This will launch a single copy of hostname on the hosts node01.example.com, node02.example.com, and node03.example.com.

Note, too, that --hostfile is essentially a per-application switch. Hence, if you specify multiple applications (as in an MPMD job), --hostfile can be specified multiple times:

shell$ cat hostfile_1
node01.example.com
shell$ cat hostfile_2
node02.example.com
shell$ mpirun -n 1 --hostfile hostfile_1 hostname : -n 1 --hostfile hostfile_2 uptime
node01.example.com
 06:11:45 up 1 day,  2:32,  0 users,  load average: 21.65, 20.85, 19.84

Notice that hostname was launched on node01.example.com and uptime was launched on node02.example.com.

10.4.3. Scheduling with the –host option

The --host option to mpirun takes a comma-delimited list of hosts on which to run. For example:

shell$ mpirun -n 3 --host a,b,c hostname

Will launch one copy of hostname on each of hosts a, b, and c. Specifically: each host defaults to 1 slot, unless specified by the :N suffix. For example:

shell$ mpirun --host a,b:2,c:3 hostname

Will launch one copy of hostname on a, two copies of hostname on b, and three copies of hostname and c.

Slots are discussed in much more detail in this section.

Important

The hosts specified by the --host option have nothing to do with which network interfaces are used for MPI communication. They are only used to specify on which hosts to launch MPI processes.

--host works in two different ways:

Exclusionary: If a list of hosts to run on has been provided by another source (e.g., by a hostfile or a batch scheduler such as Slurm, PBS/Torque, SGE, etc.), the hosts provided by the --host option must be in the already-provided host list. If the --host-specified nodes are not in the already-provided host list, mpirun will abort without launching anything.

In this case, the --host option acts like an exclusionary filter — it limits the scope of where processes will be scheduled from the original list of hosts to produce a final list of hosts.

For example, say that the hostfile my_hosts contains the hosts node1 through node4. If you run:
```
shell$ mpirun -n 1 --hostfile my_hosts --host node3 hostname
```
This will run a single copy of hostname on the host node3. However, if you run:
```
shell$ mpirun -n 1 --hostfile my_hosts --host node17 hostname
```
This is an error (because node17 is not listed in my_hosts); mpirun will abort.

Finally, note that in exclusionary mode, processes will only be executed on the --host-specified hosts. If this ends up causing an oversubscription situation, mpirun will abort by default.
Inclusionary: If a list of hosts has not been provided by another source, then the hosts provided by the --host option will be used as the original and final host list.

In this case, --host acts as an inclusionary agent; all --host-supplied hosts become available for scheduling processes. For example (assume that you are not in a scheduling environment where a list of nodes is being transparently supplied):
```
shell$ mpirun -n 3 --host a,b,c hostname
```
This will launch a single copy of hostname on the hosts a, b, and c.

Note, too, that --host is essentially a per-application switch. Hence, if you specify multiple applications (as in an MPMD job), --host can be specified multiple times:

shell$ mpirun -n 1 --host a hostname : -n 1 --host b uptime

This will launch hostname on host a and uptime on host b.

10.4.4. Process Slots

Slots are Open MPI’s representation of how many processes can be launched on a given host.

Open MPI maintains the number of slots for each host in a given parallel job, and — by default — will not let you launch more processes on a host than it has slots.

Important

It is common to set the number of slots on a host to be less than or equal to the number of processor cores on that host.

But it is important to realize that Open MPI’s concept of slots is actually unrelated to the number of physical processor cores on a host.

Specifically: the number of slots on a host can be less than, equal to, or more than the number of processor cores on a host.

If you wish to run more processes on a host than it has slots, see the section on oversubscription.

10.4.4.1. Calculating the number of slots

The number of slots on a host depends on a few factors:

If the host is specified by a job scheduler (e.g., Slurm, PBS/Torque, etc.), the job scheduler specifies the number of slots for that host.
If the host is specified in a hostfile:
1. If the slots parameter is specified, that value is used for the number of slots on that host.
2. Otherwise:
  1. If --map-by :HWTCPUS was specified, the number of slots defaults to the number of hardware threads on that host.
  2. Otherwise, the number of slots defaults to the number of processor cores on that host.
If the host is specified via the --host command line option:
1. If the :N suffix is specified, N is used for the number of slots on that host.
2. Otherwise, the number of slots defaults to 1.
3. If the same host name is specified multiple times, the slots value for that host is increased by N if :N is specified, or increased by 1 if :N is not specified.

Caution

The exact scheme used to determine the number of slots has varied between different major versions of Open MPI. The scheme described above is relevant for Open MPI v5.0.x.

Max slot counts, however, are rarely specified by schedulers. The max slot count for each node will default to “infinite” if it is not provided (meaning that Open MPI will oversubscribe the node if you ask it to — see more on oversubscribing in this section).

Here are some examples, all from unscheduled environments:

Use a hostfile and specify the slots parameter.

shell$ cat my-hostfile
node01.example.come slots=4
shell$ mpirun --hostfile my-hostfile hostname
node01
node01
node01
node01

This launched 4 processes because slots=4 was specified in the hostfile.

Use a hostfile and do not specify the slots parameter (assume that node01.example.com has 2 processor cores):
```
shell$ cat my-hostfile
node01.example.come
shell$ mpirun --hostfile my-hostfile hostname
node01
node01
```
This launched 2 processes because slots was not specified, and node02 has 2 processor cores.
Use --host:
```
shell$ mpirun --host node01.example.com hostname
node01
```
This launched 1 processes because --host with no :N suffix increments the slot count for that host by 1.
Use --host with a :N suffix:
```
shell$ mpirun --host node01.example.com:2 hostname
node01
node01
```
This launched 2 processes because :2 was specified on the command line.
Use --host with a :N suffix, and mention the host multiple times:
```
shell$ mpirun --host node01.example.com:2,node01.example.com hostname
node01
node01
node01
```
This launched 3 processes because :2 was specified on the command line, and then node01.example.com was specified an additional time, incrementing the slot count for that host to 3.

10.4.5. Oversubscribing nodes

Running more MPI processes than processors are available on a node is called oversubscribing the node. Open MPI can oversubscribe nodes, but it very much matters how you do it.

Specifically: it is critical that Open MPI knows that you are oversubscribing the node, or severe performance degradation can result.

Important

Here is a good general rule to follow: never specify a number of slots that is more than the available number of processors.

For example, if you want to run 4 processes on a host with 2 processor cores, then indicate that you only have 2 slots but want to run 4 processes. For example:

# In a hostfile, the number of slots will default to the number of
# processor cores on the host
shell$ cat my-hostfile
localhost
shell$ mpirun -n 4 --hostfile my-hostfile a.out

Specifically: we strongly suggest that you do NOT have a hostfile that contains slots=4 (because there are only two available processor cores).

That being said, the above command will fail, because you are trying to run 4 processes but there are only 2 slots available. You must specifically tell Open MPI that it is ok to oversubscribe via --map-by :OVERSUBSCRIBE:

shell$ cat my-hostfile
# For the purposes of this example, explicitly tell Open MPI
# that we have 2 slots on the host.
localhost slots=2
shell$ mpirun -n 4 --hostfile my-hostfile --map-by :OVERSUBSCRIBE a.out

The reason you should tell Open MPI whether you’re oversubscribing or not (i.e., never specify a slots value more than the number of processor cores available) is because Open MPI basically runs its message passing progression engine in two modes: aggressive and degraded.

Degraded: When Open MPI thinks that it is in an oversubscribed mode (i.e., more processes are running than there are processor cores available), MPI processes will automatically run in degraded mode and frequently yield the processor to its peers, thereby allowing all processes to make progress.

Note

Be sure to see this FAQ entry that describes how degraded mode affects processor and memory affinity.
Aggressive: When Open MPI thinks that it is in an exactly- or under-subscribed mode (i.e., the number of running processes is equal to or less than the number of available processor cores), MPI processes will automatically run in aggressive mode, meaning that they will never voluntarily give up the processor to other processes. With some network transports, this means that Open MPI will spin in tight loops attempting to make message passing progress, effectively causing other processes to not get any CPU cycles (and therefore never make any progress).

For example, on a node with a two processor cores:

shell$ cat my-hostfile
localhost slots=4
shell$ mpirun -n 4 --hostfile my-hostfile a.out

This would cause all 4 MPI processes to run in aggressive mode because Open MPI thinks that there are 4 available processor cores to use. This is actually a lie (there are only 2 processor core — not 4), and can cause extremely bad performance.

10.4.6. Forcing aggressive or degraded performance mode

The MCA parameter mpi_yield_when_idle controls whether an MPI process runs in Aggressive or Degraded performance mode. Setting it to 0 forces Aggressive mode; setting it to 1 forces Degraded mode (see this FAQ entry to see how to set MCA parameters).

Note that this value only affects the behavior of MPI processes when they are blocking in MPI library calls. It does not affect behavior of non-MPI processes, nor does it affect the behavior of a process that is not inside an MPI library call.

Open MPI normally sets this parameter automatically and users are cautioned against setting this parameter unless you are really, absolutely, positively sure of what you are doing.