Scheduling processes across hosts ================================= Open MPI provides many options for scheduling application processes across hosts, including oversubscribing processes to processors. This section describes how to define that mapping. Scheduling overview ------------------- If you are not oversubscribing your hosts (i.e., trying to run more processes than slots available on that host), scheduling is pretty simple and occurs either on a by-slot or by-node round robin schedule. If you're oversubscribing, the issue gets much more complicated |mdash| keep reading. The more complete answer is: Open MPI schedules processes to nodes by asking two questions from each application on the ``mpirun`` command line: #. *How many* processes should be launched? #. *Where* should those processes be launched? The "how many" question is directly answered with the ``-n`` switch to ``mpirun``. If ``-n`` is not specified on the ``mpirun`` command line, its value is the sum of the slots on all the nodes. The "where" question is a little more complicated, and depends on three factors: #. The final node list (e.g., after ``-hostname`` / ``--host`` exclusionary or inclusionary processing) #. The scheduling policy (which applies to all applications in a single job) #. The default and maximum number of slots on each host Open MPI currently supports two scheduling policies: by slot and by node: #. *By slot:* This is the default scheduling policy, but can also be explicitly requested by using either the ``--map-by slot`` option to ``mpirun`` or by setting the MCA parameter ``rmaps_default_mapping_policy`` to the string ``slot``. In this mode, Open MPI will schedule processes on a node until all of its default slots are exhausted before proceeding to the next node. In MPI terms, this means that Open MPI tries to maximize the number of adjacent ranks in ``MPI_COMM_WORLD`` on the same host without oversubscribing that host. For example: .. code-block:: shell$ cat my-hosts node0 slots=2 max_slots=20 node1 slots=2 max_slots=20 shell$ mpirun --hostfile my-hosts -n 8 --map-by slot hello | sort Hello World I am rank 0 of 8 running on node0 Hello World I am rank 1 of 8 running on node0 Hello World I am rank 2 of 8 running on node1 Hello World I am rank 3 of 8 running on node1 Hello World I am rank 4 of 8 running on node0 Hello World I am rank 5 of 8 running on node0 Hello World I am rank 6 of 8 running on node1 Hello World I am rank 7 of 8 running on node1 #. *By node:* This policy can be requested either by using the ``--map-by node`` option to ``mpirun`` or by setting the MCA parameter ``rmaps_default_mapping_policy`` to the string "node". In this mode, Open MPI will schedule a single process on each node in a round-robin fashion (looping back to the beginning of the node list as necessary) until all processes have been scheduled. Nodes are skipped once their default slot counts are exhausted. For example: .. code-block:: shell$ cat my-hosts node0 slots=2 max_slots=20 node1 slots=2 max_slots=20 shell$ mpirun --hostname my-hosts -n 8 --map-by node hello | sort Hello World I am rank 0 of 8 running on node0 Hello World I am rank 1 of 8 running on node1 Hello World I am rank 2 of 8 running on node0 Hello World I am rank 3 of 8 running on node1 Hello World I am rank 4 of 8 running on node0 Hello World I am rank 5 of 8 running on node1 Hello World I am rank 6 of 8 running on node0 Hello World I am rank 7 of 8 running on node1 In both policies, if the default slot count is exhausted on all nodes while there are still processes to be scheduled, Open MPI will trigger an oversubscription condition. If ``:OVERSUBSCRIBE`` is added as a modifier to the ``--map-by`` option (e.g., ``mpirun --map-by node:OVERSUBSCRIBE ...`` |mdash| :ref:`see this section ` for more details), Open MPI will continue to loop through the list of nodes again and try to schedule one more process to each node until all processes are scheduled. Nodes are skipped in this process if their maximum slot count is exhausted. If the maximum slot count is exhausted on all nodes while there are still processes to be scheduled, Open MPI will abort without launching any processes. If ``:OVERSUBSCRIBE`` is *not* specified and an oversubscription condition occurs, Open MPI will abort without launching any processes. .. _running-scheduling-hostfile-option-label: Scheduling with the --hostfile option ------------------------------------- The ``--hostfile`` option to ``mpirun`` takes a filename that lists hosts on which to launch MPI processes. .. important:: The hosts listed in a hostfile have *nothing* to do with which network interfaces are used for MPI communication. They are *only* used to specify on which hosts to launch MPI processes. Hostfiles are simple text files with hosts specified, one per line. Each host can also specify a default and maximum number of *slots* to be used on that host (i.e., the maximum number of processes that will be launched on that node). Comments are also supported, and blank lines are ignored. For example: .. code-block:: # This is an example hostfile. Comments begin with #. # # Since no slots are specified, the number of slots defaults to the # number of processor cores available on the machine. foo.example.com # We want to allow launching a maximum of 2 processes on this host # (e.g., potentially because it has two processor cores): bar.example.com slots=2 Slots are discussed in much more detail :ref:`in this section `. Hostfiles works in two different ways: #. *Exclusionary:* If a list of hosts to run on has been provided by another source (e.g., by a hostfile or a batch scheduler such as Slurm, PBS/Torque, SGE, etc.), the hosts provided by the hostfile must be in the already-provided host list. If the hostfile-specified nodes are *not* in the already-provided host list, ``mpirun`` will abort without launching anything. In this case, hostfiles act like an exclusionary filter |mdash| they limit the scope of where processes will be scheduled from the original list of hosts to produce a final list of hosts. For example, say that a scheduler job contains hosts ``node01`` through ``node04``. If you run: .. code-block:: shell$ cat my_hosts node03 shell$ mpirun -n 1 --hostfile my_hosts hostname This will run a single copy of ``hostname`` on the host ``node03``. However, presuming your job was allocated only to ``node03`` and you run the following: .. code-block:: shell$ cat my_hosts node17 shell$ mpirun -n 1 --hostfile my_hosts hostname This is an error (because ``node17`` is not allocated to your job), and ``mpirun`` will abort. Finally, note that in exclusionary mode, processes will *only* be executed on the hostfile-specified hosts, If this ends up causing an oversubscription situation, ``mpirun`` will abort by default. #. *Inclusionary:* If a list of hosts has *not* been provided by another source, then the hosts provided by the ``--hostfile`` option will be used as the original and final host list. In this case, ``--hostfile`` acts as an inclusionary agent; all ``--hostfile``-supplied hosts become available for scheduling processes. For example (assume that you are *not* in a scheduling environment where a list of nodes is being transparently supplied): .. code-block:: shell$ cat my_hosts node01.example.com slots=1 node02.example.com slots=1 node03.example.com slots=1 shell$ mpirun -n 3 --hostfile my_hosts hostname This will launch a single copy of ``hostname`` on the hosts ``node01.example.com``, ``node02.example.com``, and ``node03.example.com``. Note, too, that ``--hostfile`` is essentially a per-application switch. Hence, if you specify multiple applications (as in an MPMD job), ``--hostfile`` can be specified multiple times: .. code-block:: shell$ cat hostfile_1 node01.example.com shell$ cat hostfile_2 node02.example.com shell$ mpirun -n 1 --hostfile hostfile_1 hostname : -n 1 --hostfile hostfile_2 uptime node01.example.com 06:11:45 up 1 day, 2:32, 0 users, load average: 21.65, 20.85, 19.84 Notice that ``hostname`` was launched on ``node01.example.com`` and ``uptime`` was launched on ``node02.example.com``. .. _running-scheduling-host-option-label: Scheduling with the --host option --------------------------------- The ``--host`` option to ``mpirun`` takes a comma-delimited list of hosts on which to run. For example: .. code-block:: shell$ mpirun -n 3 --host a,b,c hostname Will launch *one* copy of ``hostname`` on each of hosts ``a``, ``b``, and ``c``. Specifically: each host defaults to 1 slot, unless specified by the ``:N`` suffix. For example: .. code-block:: shell$ mpirun --host a,b:2,c:3 hostname Will launch one copy of ``hostname`` on ``a``, two copies of ``hostname`` on ``b``, and three copies of ``hostname`` and ``c``. Slots are discussed in much more detail :ref:`in this section `. .. important:: The hosts specified by the ``--host`` option have *nothing* to do with which network interfaces are used for MPI communication. They are *only* used to specify on which hosts to launch MPI processes. ``--host`` works in two different ways: #. *Exclusionary:* If a list of hosts to run on has been provided by another source (e.g., by a hostfile or a batch scheduler such as Slurm, PBS/Torque, SGE, etc.), the hosts provided by the ``--host`` option must be in the already-provided host list. If the ``--host``-specified nodes are *not* in the already-provided host list, ``mpirun`` will abort without launching anything. In this case, the ``--host`` option acts like an exclusionary filter |mdash| it limits the scope of where processes will be scheduled from the original list of hosts to produce a final list of hosts. For example, say that the hostfile ``my_hosts`` contains the hosts ``node1`` through ``node4``. If you run: .. code-block:: shell$ mpirun -n 1 --hostfile my_hosts --host node3 hostname This will run a single copy of ``hostname`` on the host ``node3``. However, if you run: .. code-block:: shell$ mpirun -n 1 --hostfile my_hosts --host node17 hostname This is an error (because ``node17`` is not listed in ``my_hosts``); ``mpirun`` will abort. Finally, note that in exclusionary mode, processes will *only* be executed on the ``--host``-specified hosts. If this ends up causing an oversubscription situation, ``mpirun`` will abort by default. #. *Inclusionary:* If a list of hosts has *not* been provided by another source, then the hosts provided by the ``--host`` option will be used as the original and final host list. In this case, ``--host`` acts as an inclusionary agent; all ``--host``-supplied hosts become available for scheduling processes. For example (assume that you are *not* in a scheduling environment where a list of nodes is being transparently supplied): .. code-block:: shell$ mpirun -n 3 --host a,b,c hostname This will launch a single copy of ``hostname`` on the hosts ``a``, ``b``, and ``c``. Note, too, that ``--host`` is essentially a per-application switch. Hence, if you specify multiple applications (as in an MPMD job), ``--host`` can be specified multiple times: .. code-block:: shell$ mpirun -n 1 --host a hostname : -n 1 --host b uptime This will launch ``hostname`` on host ``a`` and ``uptime`` on host ``b``. .. _running-scheduling-slots-label: Process Slots ------------- *Slots* are Open MPI's representation of how many processes can be launched on a given host. Open MPI maintains the number of slots for each host in a given parallel job, and |mdash| by default |mdash| will not let you launch more processes on a host than it has slots. .. important:: It is common to set the number of slots on a host to be less than or equal to the number of processor cores on that host. **But it is important to realize that Open MPI's concept of slots is actually unrelated to the number of physical processor cores on a host.** Specifically: the number of slots on a host can be less than, equal to, or more than the number of processor cores on a host. If you wish to run more processes on a host than it has slots, :ref:`see the section on oversubscription `. Calculating the number of slots ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The number of slots on a host depends on a few factors: #. If the host is specified by a job scheduler (e.g., Slurm, PBS/Torque, etc.), the job scheduler specifies the number of slots for that host. #. If the host is specified in a hostfile: #. If the ``slots`` parameter is specified, that value is used for the number of slots on that host. #. Otherwise: #. If ``--map-by :HWTCPUS`` was specified, the number of slots defaults to the number of hardware threads on that host. #. Otherwise, the number of slots defaults to the number of processor cores on that host. #. If the host is specified via the ``--host`` command line option: #. If the ``:N`` suffix is specified, ``N`` is used for the number of slots on that host. #. Otherwise, the number of slots defaults to 1. #. If the same host name is specified multiple times, the slots value for that host is increased by ``N`` if ``:N`` is specified, or increased by 1 if ``:N`` is not specified. .. caution:: The exact scheme used to determine the number of slots has varied between different major versions of Open MPI. The scheme described above is relevant for Open MPI |ompi_series|. Max slot counts, however, are rarely specified by schedulers. The max slot count for each node will default to "infinite" if it is not provided (meaning that Open MPI will oversubscribe the node if you ask it to |mdash| see more on oversubscribing in :ref:`this section `). Here are some examples, all from unscheduled environments: #. Use a hostfile and specify the ``slots`` parameter. .. code-block:: sh shell$ cat my-hostfile node01.example.come slots=4 shell$ mpirun --hostfile my-hostfile hostname node01 node01 node01 node01 This launched 4 processes because ``slots=4`` was specified in the hostfile. #. Use a hostfile and do *not* specify the ``slots`` parameter (assume that ``node01.example.com`` has 2 processor cores): .. code-block:: sh shell$ cat my-hostfile node01.example.come shell$ mpirun --hostfile my-hostfile hostname node01 node01 This launched 2 processes because ``slots`` was not specified, and ``node02`` has 2 processor cores. #. Use ``--host``: .. code-block:: sh shell$ mpirun --host node01.example.com hostname node01 This launched 1 processes because ``--host`` with no ``:N`` suffix increments the slot count for that host by 1. #. Use ``--host`` with a ``:N`` suffix: .. code-block:: sh shell$ mpirun --host node01.example.com:2 hostname node01 node01 This launched 2 processes because ``:2`` was specified on the command line. #. Use ``--host`` with a ``:N`` suffix, and mention the host multiple times: .. code-block:: sh shell$ mpirun --host node01.example.com:2,node01.example.com hostname node01 node01 node01 This launched 3 processes because ``:2`` was specified on the command line, and then ``node01.example.com`` was specified an additional time, incrementing the slot count for that host to 3. .. _running-mpi-apps-oversubscribing-label: Oversubscribing nodes --------------------- Running more MPI processes than processors are available on a node is called *oversubscribing* the node. Open MPI can oversubscribe nodes, but it very much matters *how* you do it. Specifically: it is critical that Open MPI *knows* that you are oversubscribing the node, or **severe** performance degradation can result. .. important:: Here is a good general rule to follow: **never specify a number of slots that is more than the available number of processors.** For example, if you want to run 4 processes on a host with 2 processor cores, then indicate that you only have 2 slots but want to run 4 processes. For example: .. code-block:: sh # In a hostfile, the number of slots will default to the number of # processor cores on the host shell$ cat my-hostfile localhost shell$ mpirun -n 4 --hostfile my-hostfile a.out Specifically: we strongly suggest that you do **NOT** have a hostfile that contains ``slots=4`` (because there are only two available processor cores). That being said, the above command will fail, because you are trying to run 4 processes but there are only 2 slots available. You must specifically tell Open MPI that it is ok to oversubscribe via ``--map-by :OVERSUBSCRIBE``: .. code-block:: sh shell$ cat my-hostfile # For the purposes of this example, explicitly tell Open MPI # that we have 2 slots on the host. localhost slots=2 shell$ mpirun -n 4 --hostfile my-hostfile --map-by :OVERSUBSCRIBE a.out The reason you should tell Open MPI whether you're oversubscribing or not (i.e., never specify a ``slots`` value more than the number of processor cores available) is because Open MPI basically runs its message passing progression engine in two modes: *aggressive* and *degraded*. #. *Degraded:* When Open MPI thinks that it is in an oversubscribed mode (i.e., more processes are running than there are processor cores available), MPI processes will automatically run in *degraded* mode and frequently yield the processor to its peers, thereby allowing all processes to make progress. .. note:: Be sure to see :ref:`this FAQ entry ` that describes how degraded mode affects processor and memory affinity. #. *Aggressive:* When Open MPI thinks that it is in an exactly- or under-subscribed mode (i.e., the number of running processes is equal to or less than the number of available processor cores), MPI processes will automatically run in *aggressive* mode, meaning that they will never voluntarily give up the processor to other processes. With some network transports, this means that Open MPI will spin in tight loops attempting to make message passing progress, effectively causing other processes to not get any CPU cycles (and therefore never make any progress). For example, on a node with a two processor cores: .. code-block:: shell$ cat my-hostfile localhost slots=4 shell$ mpirun -n 4 --hostfile my-hostfile a.out This would cause all 4 MPI processes to run in *aggressive* mode because Open MPI thinks that there are 4 available processor cores to use. This is actually a lie (there are only 2 processor core |mdash| not 4), and can cause extremely bad performance. Forcing aggressive or degraded performance mode ----------------------------------------------- The MCA parameter ``mpi_yield_when_idle`` controls whether an MPI process runs in Aggressive or Degraded performance mode. Setting it to 0 forces Aggressive mode; setting it to 1 forces Degraded mode (see :ref:`this FAQ entry ` to see how to set MCA parameters). Note that this value *only* affects the behavior of MPI processes when they are blocking in MPI library calls. It does not affect behavior of non-MPI processes, nor does it affect the behavior of a process that is not inside an MPI library call. Open MPI normally sets this parameter automatically and users are cautioned against setting this parameter unless you are really, absolutely, positively sure of what you are doing.