4. Exceptions to the PMIx Standard

Exceptions to the base PMIx Standard are listed here. These exceptions are not indicative of any intent to stray from the Standard, but instead represent the difference between the pace of development of the library versus the normal Standard’s process. Accordingly, it is expected that the exceptions listed below will make their way into a future release of the PMIx Standard and then be removed from the list of exceptions in some future OpenPMIx release.

4.1. Extensions

OpenPMIx v5.0.3rc1 is based on the PMIx 4.2 Standard. In addition to Extensions to the Standard, this release includes the conversion of all support macros to PMIx function APIs — e.g., the PMIX_LOAD_PROCID macro is now the PMIx_Load_procid() function — in accordance with planned changes to the Standard. The macro versions have been retained as deprecated (without warnings) for backward compatibility.

4.2. Qualified Values

OpenPMIx has introduced the concept of qualified values to allow users to specify a value combined with one or more qualifiers.

4.3. Scheduler Integration APIs

Allow a scheduler to direct the resource manager to execute a session-related operation, or allow the resource manager to report a session-related action (e.g., session terminated) to the scheduler:

pmix_status_t PMIx_Session_control(uint32_t sessionID,
                                   const pmix_info_t *directives, size_t ndirs,
                                   pmix_info_cbfunc_t cbfunc, void *cbdata);

4.4. Tool APIs

Check if the tool is connected to a PMIx server:
```
bool PMIx_tool_is_connected(void);
```
Allow the tool to register a server function pointer module so it can service client requests:
```
pmix_status_t PMIx_tool_set_server_module(pmix_server_module_t *mod);
```

4.5. Utility APIs

Load a key:

void PMIx_Load_key(pmix_key_t key, const char *src);

Check a key:

bool PMIx_Check_key(const char *key, const char *str);

Check to see if a key is a “reserved” key:

bool PMIx_Check_reserved_key(const char *key);

Load a string into a pmix_nspace_t struct:

void PMIx_Load_nspace(pmix_nspace_t nspace, const char *str);

Check two nspace structs for equality:

bool PMIx_Check_nspace(const char *key1, const char *key2);

Check if a namespace is invalid:

bool PMIx_Nspace_invalid(const char *nspace);

Load a process ID struct:

void PMIx_Load_procid(pmix_proc_t *p,
                      const char *ns,
                      pmix_rank_t rk);

Transfer a process ID struct (non-destructive):

void PMIx_Xfer_procid(pmix_proc_t *dst,
                      const pmix_proc_t *src);

Check two proc IDs for equality:

bool PMIx_Check_procid(const pmix_proc_t *a,
                       const pmix_proc_t *b);

Check two ranks for equality:

bool PMIx_Check_rank(pmix_rank_t a,
                     pmix_rank_t b);

Check if proc ID is invalid:

bool PMIx_Procid_invalid(const pmix_proc_t *p);

4.6. Argv Handling

Functions for handling of argv arrays (NULL-terminated array of strings)

Count the number of entries

int PMIx_Argv_count(char **a);

Append a string to the array

pmix_status_t PMIx_Argv_append_nosize(char ***argv, const char *arg);

Prepend a string to the array

pmix_status_t PMIx_Argv_prepend_nosize(char ***argv, const char *arg);

Append a string to the array, but only if it doesn’t already appear on the array (ignore if it does)

pmix_status_t PMIx_Argv_append_unique_nosize(char ***argv, const char *arg);

Free an array, including each string on the array

void PMIx_Argv_free(char **argv);

Split a string into an argv array, dividing the string on each occurrence of the specified delimiter character. Retain empty entries in the array when more than one copy of the delimiter character appears in a sequence.

char **PMIx_Argv_split_inter(const char *src_string,
                             int delimiter,
                             bool include_empty);

Split a string into an argv array, dividing the string on each occurrence of the specified delimiter character. Retain empty entries in the array when more than one copy of the delimiter character appears in a sequence. Acts as a wrapper to PMIx_Argv_split_inter with include_empty set to true

char **PMIx_Argv_split_with_empty(const char *src_string, int delimiter);

Split a string into an argv array, dividing the string on each occurrence of the specified delimiter character. Discard empty entries in the array when more than one copy of the delimiter character appears in a sequence. Acts as a wrapper to PMIx_Argv_split_inter with include_empty set to false

char **PMIx_Argv_split(const char *src_string, int delimiter);

Join all the elements of an argv array into a single newly-allocated string, with the specified delimiter character at the join points.

char *PMIx_Argv_join(char **argv, int delimiter);

Copy a NULL-terminated argv array.

char **PMIx_Argv_copy(char **argv);

Set environment variable:

pmix_status_t PMIx_Setenv(const char *name,
                          const char *value,
                          bool overwrite,
                          char ***env);

4.7. Value Struct Functions

Initialize a value struct:

void PMIx_Value_construct(pmix_value_t *val);

Free memory stored inside a value struct:

void PMIx_Value_destruct(pmix_value_t *val);

Create and initialize an array of value structs:
```
pmix_value_t* PMIx_Value_create(size_t n);
```
Free memory stored inside an array of coord structs (does not free the struct memory itself):
```
void PMIx_Value_free(pmix_value_t *v, size_t n);
```
Check the given value struct to determine if it includes a boolean value (includes strings for true and false, including abbreviations such as t or f), and if so, then its value. A value type of PMIX_UNDEF is taken to imply a boolean true.
```
pmix_boolean_t PMIx_Value_true(const pmix_value_t *v);
```

Compare the contents of two pmix_value_t structures:

pmix_value_cmp_t PMIx_Value_compare(pmix_value_t *v1,
                                    pmix_value_t *v2);

Get the size of the contents of a pmix_value_t structure:

pmix_status_t PMIx_Value_get_size(const pmix_value_t *val,
                                  size_t *size);

4.8. Data Array Functions

Construct a data array object, allocating the memory for the indicated number of the specified data type. Memory for the provided data array object must have previously been allocated or statically declared:
```
void PMIx_Data_array_construct(pmix_data_array_t *p,
                               size_t num, pmix_data_type_t type);
```

Initialize the fields of a data array object without allocating any memory for the included array:

void PMIx_Data_array_init(pmix_data_array_t *p,
                          pmix_data_type_t type);

Destroy a data array object, releasing all memory included in it:
```
void PMIx_Data_array_destruct(pmix_data_array_t *d);
```
Create and initialize a pmix_data_array_t structure, allocating the memory for the indicated number of the specified data type as well as the pmix_data_array_t object itself:
```
pmix_data_array_t* PMIx_Data_array_create(size_t n, pmix_data_type_t type);
```
Free memory stored inside a pmix_data_array_t structure (does not free the provided pmix_data_array_t object itself):
```
void PMIx_Data_array_free(pmix_data_array_t *p);
```

4.9. Info Struct Functions

Initialize an info struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Info_construct(pmix_info_t *p);
```

Free memory stored inside an info struct:

void PMIx_Info_destruct(pmix_info_t *p);

Create and initialize an array of info structs:
```
pmix_info_t* PMIx_Info_create(size_t n);
```
Free memory stored inside an array of coord structs (does not free the struct memory itself):
```
void PMIx_Info_free(pmix_info_t *p, size_t n);
```
Check the given info struct to determine if it includes a boolean value (includes strings for true and false, including abbreviations such as t or f), and if so, then its value. A value type of PMIX_UNDEF is taken to imply a boolean true as the presence of the key defaults to indicating true.
```
pmix_boolean_t PMIx_Info_true(const pmix_info_t *p);
```

Mark the info struct as required:

void PMIx_Info_required(pmix_info_t *p);

Mark the info struct as optional:

void PMIx_Info_optional(pmix_info_t *p);

Check if the info struct is required:

bool PMIx_Info_is_required(const pmix_info_t *p);

Check if the info struct is optional:

bool PMIx_Info_is_optional(const pmix_info_t *p);

Mark the info struct as processed:

void PMIx_Info_processed(pmix_info_t *p);

Check if the info struct has been processed:

bool PMIx_Info_was_processed(const pmix_info_t *p);

Mark the info struct as the end of an array:
```
void PMIx_Info_set_end(pmix_info_t *p);
```

Check if the info struct is the end of an array:

bool PMIx_Info_is_end(const pmix_info_t *p);

Mark the info as a qualifier:

void PMIx_Info_qualifier(pmix_info_t *p);

Check if the info struct is a qualifier:

bool PMIx_Info_is_qualifier(const pmix_info_t *p);

Mark the info struct as persistent — do not release its contents:
```
void PMIx_Info_persistent(pmix_info_t *p);
```

Check if the info struct is persistent:

bool PMIx_Info_is_persistent(const pmix_info_t *p);

Get the size of a pmix_info_t structure:

pmix_status_t PMIx_Info_get_size(const pmix_info_t *val,
                                 size_t *size);

4.10. Coordinate Struct Functions

Initialize a coord struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Coord_construct(pmix_coord_t *m);
```

Free memory stored inside a coord struct:

void PMIx_Coord_destruct(pmix_coord_t *m);

Create and initialize an array of coord structs:

pmix_coord_t* PMIx_Coord_create(size_t dims,
                                size_t number);

Free memory stored inside an array of coord structs (does not free the struct memory itself):
```
void PMIx_Coord_free(pmix_coord_t *m, size_t number);
```

4.11. Topology Functions

Initialize a topology struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Topology_construct(pmix_topology_t *t);
```

Create and initialize an array of topology structs:

pmix_topology_t* PMIx_Topology_create(size_t n);

Free memory stored inside an array of topology structs (does not free the struct memory itself):
```
void PMIx_Topology_free(pmix_topology_t *t, size_t n);
```

4.12. Cpuset Functions

Initialize a cpuset struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Cpuset_construct(pmix_cpuset_t *cpuset);
```

Free memory stored inside a cpuset struct:

void PMIx_Cpuset_destruct(pmix_cpuset_t *cpuset);

Create and initialize an array of cpuset structs:
```
pmix_cpuset_t* PMIx_Cpuset_create(size_t n);
```
Free memory stored inside an array of cpuset structs (does not free the struct memory itself):
```
void PMIx_Cpuset_free(pmix_cpuset_t *c, size_t n);
```

4.13. Geometry Functions

Initialize a geometry struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Geometry_construct(pmix_geometry_t *g);
```

Free memory stored inside a cpuset struct:

void PMIx_Geometry_destruct(pmix_geometry_t *g);

Create and initialize an array of cpuset structs:

pmix_geometry_t* PMIx_Geometry_create(size_t n);

Free memory stored inside an array of cpuset structs (does not free the struct memory itself):
```
void PMIx_Geometry_free(pmix_geometry_t *g, size_t n);
```

4.14. Device Distance Functions

Initialize a device distance struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Device_distance_construct(pmix_device_distance_t *d);
```

Free memory stored inside a device distance struct:

void PMIx_Device_distance_destruct(pmix_device_distance_t *d);

Create and initialize an array of device distance structs:

pmix_device_distance_t* PMIx_Device_distance_create(size_t n);

Free memory stored inside an array of device distance structs (does not free the struct memory itself):
```
void PMIx_Device_distance_free(pmix_device_distance_t *d, size_t n);
```

4.15. Byte Object Functions

Initialize a byte object struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Byte_object_construct(pmix_byte_object_t *b);
```

Free memory stored inside a byte object struct:

void PMIx_Byte_object_destruct(pmix_byte_object_t *g);

Create and initialize an array of byte object structs:

pmix_byte_object_t* PMIx_Byte_object_create(size_t n);

Free memory stored inside an array of byte object structs (does not free the struct memory itself):
```
void PMIx_Byte_object_free(pmix_byte_object_t *g, size_t n);
```

Load a byte object:

void PMIx_Byte_object_load(pmix_byte_object_t *b,
                           char *d, size_t sz);

4.16. Endpoint Functions

Initialize an endpoint struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Endpoint_construct(pmix_endpoint_t *e);
```

Free memory stored inside an endpoint struct:

void PMIx_Endpoint_destruct(pmix_endpoint_t *e);

Create and initialize an array of endpoint structs:

pmix_endpoint_t* PMIx_Endpoint_create(size_t n);

Free memory stored inside an array of endpoint structs (does not free the struct memory itself):
```
void PMIx_Endpoint_free(pmix_endpoint_t *e, size_t n);
```

4.17. Envar Functions

Initialize an envar struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Envar_construct(pmix_envar_t *e);
```

Free memory stored inside an envar struct:

void PMIx_Envar_destruct(pmix_envar_t *e);

Create and initialize an array of envar structs:
```
pmix_envar_t* PMIx_Envar_create(size_t n);
```
Free memory stored inside an array of envar structs (does not free the struct memory itself):
```
void PMIx_Envar_free(pmix_envar_t *e, size_t n);
```

Load an envar struct:

void PMIx_Envar_load(pmix_envar_t *e,
                     char *var,
                     char *value,
                     char separator);

4.18. Data Buffer Functions

Initialize a data buffer struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Data_buffer_construct(pmix_data_buffer_t *b);
```

Free memory stored inside a data buffer struct:

void PMIx_Data_buffer_destruct(pmix_data_buffer_t *b);

Create a data buffer struct:

pmix_data_buffer_t* PMIx_Data_buffer_create(void);

Free memory stored inside a data buffer struct:

void PMIx_Data_buffer_release(pmix_data_buffer_t *b);

Load a data buffer struct:

void PMIx_Data_buffer_load(pmix_data_buffer_t *b,
                           char *bytes, size_t sz);

Unload a data buffer struct:

void PMIx_Data_buffer_unload(pmix_data_buffer_t *b,
                             char **bytes, size_t *sz);

4.19. Proc Struct Functions

Initialize a proc struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Proc_construct(pmix_proc_t *p);
```

Clear memory inside a proc struct:

void PMIx_Proc_destruct(pmix_proc_t *p);

Create and initialize an array of proc structs:
```
pmix_proc_t* PMIx_Proc_create(size_t n);
```
Free memory stored inside an array of proc structs (does not free the struct memory itself):
```
void PMIx_Proc_free(pmix_proc_t *p, size_t n);
```

Load a proc struct:

void PMIx_Proc_load(pmix_proc_t *p,
                    char *nspace, pmix_rank_t rank);

Construct a multicluster nspace struct from cluster and nspace values:

void PMIx_Multicluster_nspace_construct(pmix_nspace_t target,
                                        pmix_nspace_t cluster,
                                        pmix_nspace_t nspace);

Parse a multicluster nspace struct to separate out the cluster and nspace portions:

void PMIx_Multicluster_nspace_parse(pmix_nspace_t target,
                                    pmix_nspace_t cluster,
                                    pmix_nspace_t nspace);

4.20. Proc Info Functions

Initialize a proc info struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Proc_info_construct(pmix_proc_info_t *p);
```

Clear memory inside a proc info struct:

void PMIx_Proc_info_destruct(pmix_proc_info_t *p);

Create and initialize an array of proc info structs:

pmix_proc_info_t* PMIx_Proc_info_create(size_t n);

Free memory stored inside an array of proc info structs (does not free the struct memory itself):
```
void PMIx_Proc_info_free(pmix_proc_info_t *p, size_t n);
```

4.21. Proc Stats Functions

Initialize a proc stats struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Proc_stats_construct(pmix_proc_stats_t *p);
```

Clear memory inside a proc stats struct:

void PMIx_Proc_stats_destruct(pmix_proc_stats_t *p);

Create and initialize an array of proc stats structs:

pmix_proc_stats_t* PMIx_Proc_stats_create(size_t n);

Free memory stored inside an array of proc stats structs (does not free the struct memory itself):
```
void PMIx_Proc_stats_free(pmix_proc_stats_t *p, size_t n);
```

4.22. Disk Stats Functions

Initialize a disk stats struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Disk_stats_construct(pmix_disk_stats_t *p);
```

Clear memory inside a disk stats struct:

void PMIx_Disk_stats_destruct(pmix_disk_stats_t *p);

Create and initialize an array of disk stats structs:

pmix_disk_stats_t* PMIx_Disk_stats_create(size_t n);

Free memory stored inside an array of disk stats structs (does not free the struct memory itself):
```
void PMIx_Disk_stats_free(pmix_disk_stats_t *p, size_t n);
```

4.23. Net Stats Functions

Initialize a net stats struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Net_stats_construct(pmix_net_stats_t *p);
```

Clear memory inside a net stats struct:

void PMIx_Net_stats_destruct(pmix_net_stats_t *p);

Create and initialize an array of net stats structs:

pmix_net_stats_t* PMIx_Net_stats_create(size_t n);

Free memory stored inside an array of net stats structs (does not free the struct memory itself):
```
void PMIx_Net_stats_free(pmix_net_stats_t *p, size_t n);
```

4.24. Process Data Functions

Initialize a pdata struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_Pdata_construct(pmix_pdata_t *p);
```

Clear memory inside a pdata struct:

void PMIx_Pdata_destruct(pmix_pdata_t *p);

Create and initialize an array of pdata structs:
```
pmix_pdata_t* PMIx_Pdata_create(size_t n);
```
Free memory stored inside an array of pdata structs (does not free the struct memory itself):
```
void PMIx_Pdata_free(pmix_pdata_t *p, size_t n);
```

4.25. App Struct Functions

Initialize a pmix_app_t struct. Memory for the provided object must have previously been allocated or statically declared:
```
void PMIx_App_construct(pmix_app_t *p);
```
Clear memory inside an app struct:
```
void PMIx_App_destruct(pmix_app_t *p);
```
Create and initialize an array of app structs:
```
pmix_app_t* PMIx_App_create(size_t n);
```
Create and initialize an array of pmix_info_t structs in the provided pmix_app_t object:
```
void PMIx_App_info_create(pmix_app_t *p, size_t n);
```
Free memory stored inside an array of app structs (does not free the struct memory itself):
```
void PMIx_App_free(pmix_app_t *p, size_t n);
```
Free memory stored inside a pmix_app_t object
```
void PMIx_App_release(pmix_app_t *p);
```

4.26. PMIx Info List Functions

Constructing arrays of pmix_info_t for passing to an API can be tedious since the pmix_info_t itself is not a “list object”. Since this is a very frequent operation, a set of APIs has been provided that opaquely manipulates internal PMIx list structures for this purpose. The user only need provide a void* pointer to act as the caddy for the internal list object. The base functions for these operations are in the Standard, but the following functions have been added here:

Retrieve the next pmix_info_t from the provided list, given the current pointer. Passing a NULL to the prev parameter will return the first object on the list. A NULL is returned upon reaching the end of the list:
pmix_info_t* PMIx_Info_list_get_info(void *ptr, void *prev, void **next);
Insert a pmix_info_t` struct into the provided list. This directly copies the contents of the provided pmix_info_t struct, preserving any included pointers. The object on the list is subsequently marked as persistent to avoid free’ing any objects pointed to in the struct:
pmix_status_t PMIx_Info_list_insert(void *ptr, pmix_info_t *info);

Prepend a value onto the provided list:

pmix_status_t PMIx_Info_list_prepend(void *ptr,
                                     const char *key,
                                     const void *value,
                                     pmix_data_type_t type);

4.27. Pretty-Print Functions

The following pretty-print support APIs have been added:

Print a pmix_value_cmp_t value

const char* PMIx_Value_comparison_string(pmix_value_cmp_t cmp);

Print the contents of a pmix_app_t struct. Note that the returned string must be free’d by the caller:
char* PMIx_App_string(const pmix_app_t *app);

The following pretty-print support APIs have been slightly modified to add a const qualifier to their input parameter:

const char* PMIx_Get_attribute_string(const char *attribute);
const char* PMIx_Get_attribute_name(const char *attrstring);
char* PMIx_Info_string(const pmix_info_t *info);
char* PMIx_Value_string(const pmix_value_t *value);

This is not expected to cause any issues for users.

The following function has been added to return the pmix_status_t corresponding to the string name of the constant:

pmix_status_t PMIx_Error_code(const char *errname);

4.28. Constants

PMIX_DATA_BUFFER: data type for packing/unpacking of pmix_data_buffer_t objects
PMIX_DISK_STATS: data type for packing/unpacking of pmix_disk_stats_t objects
PMIX_NET_STATS: data type for packing/unpacking of pmix_net_stats_t objects
PMIX_NODE_STATS: data type for packing/unpacking of pmix_node_stats_t objects
PMIX_PROC_STATS: data type for packing/unpacking of pmix_proc_stats_t objects
PMIX_ERR_JOB_EXE_NOT_FOUND: specified executable not found
PMIX_ERR_JOB_INSUFFICIENT_RESOURCES: insufficient resources to spawn job
PMIX_ERR_JOB_SYS_OP_FAILED: system library operation failed
PMIX_ERR_JOB_WDIR_NOT_FOUND: specified working directory not found
PMIX_READY_FOR_DEBUG: event indicating job/proc is ready for debug (accompanied by PMIX_BREAKPOINT indicating where proc is waiting)
PMIX_ERR_PROC_REQUESTED_ABORT: process called PMIx_Abort
PMIX_ERR_PROC_KILLED_BY_CMD: process was terminated by RTE command
PMIX_ERR_PROC_FAILED_TO_START: process failed to start
PMIX_ERR_PROC_ABORTED_BY_SIG: process aborted by signal (e.g., segmentation fault)
PMIX_ERR_PROC_SENSOR_BOUND_EXCEEDED: process terminated due to exceeding a sensor boundary
PMIX_ERR_EXIT_NONZERO_TERM: process exited normally, but with a non-zero status
PMIX_INFO_QUALIFIER (value: 0x00000008): Info is a qualifier to the primary value
PMIX_INFO_PERSISTENT (value: 0x00000010): Do not release included value

Note

OpenPMIx version v5.0.3rc1 renamed the PMIX_DEBUG_WAIT_FOR_NOTIFY to PMIX_READY_FOR_DEBUG. The prior name is retained as deprecated for backward compatibility.

4.29. Attributes

Attribute	Type	Description
`PMIX_EXTERNAL_AUX_EVENT_BASE` `"pmix.evaux"`	`(void*)`	event base to be used for auxiliary functions (e.g., capturing signals) that would otherwise interfere with the host
`PMIX_CONNECT_TO_SCHEDULER` `"pmix.cnct.sched"`	`(bool)`	Connect to the system scheduler
`PMIX_BIND_PROGRESS_THREAD` `"pmix.bind.pt"`	`(char*)`	Comma-delimited ranges of CPUs that the internal PMIx progress thread shall be bound to
`PMIX_BIND_REQUIRED` `"pmix.bind.reqd"`	`(bool)`	Return error if the internal PMIx progress thread cannot be bound
`PMIX_COLOCATE_PROCS` `"pmix.colproc"`	`(pmix_data_array_t*)`	Array of `pmix_proc_t` identifying the procs with which the new job’s procs are to be colocated
`PMIX_COLOCATE_NPERPROC` `"pmix.colnum.proc"`	`(uint16_t)`	Number of procs to colocate with each identified proc
`PMIX_COLOCATE_NPERNODE` `"pmix.colnum.node"`	`(uint16_t)`	Number of procs to colocate on the node of each identified proc
`PMIX_EVENT_ONESHOT` `pmix.evone`	`(bool)`	when registering, indicate that this event handler is to be deleted after being invoked
`PMIX_GROUP_ADD_MEMBERS` `pmix.grp.add`	`(pmix_data_array_t*)`	Array of `pmix_proc_t` identifying procs that are not included in the membership specified in the procs array passed to the `PMIx_Group_construct[_nb]()` call, but are to be included in the final group. The identified procs will be sent an invitation to join the group during the construction procedure. This is used when some members of the proposed group do not know the full membership and therefore cannot include all members in the call to construct.
`PMIX_GROUP_LOCAL_CID` `pmix.grp.lclid`	`(size_t)`	Local context ID for the specified process member of a group
`PMIX_GROUP_INFO` `pmix.grp.info`	`pmix_data_array_t`	Array of pmix_info_t containing data that is to be shared across all members of a group during group construction
`PMIX_IOF_TAG_DETAILED_OUTPUT` `pmix.iof.tagdet`	`(bool)`	Tag output with the [local jobid,rank][hostname:pid] and channel it comes from
`PMIX_IOF_TAG_FULLNAME_OUTPUT` `pmix.iof.tagfull`	`(bool)`	Tag output with the [nspace,rank] and channel it comes from
`PMIX_LOG_AGG` `pmix.log.agg`	`(bool)`	Whether to aggregate and prevent duplicate logging messages based on key value pairs.
`PMIX_LOG_KEY` `pmix.log.key`	`(char*)`	key to a logging message
`PMIX_LOG_VAL` `pmix.log.val`	`(char*)`	value to a logging message
`PMIX_MYSERVER_URI` `pmix.mysrvr.uri`	`(char*)`	URI of this proc’s listener socket
`PMIX_QUALIFIED_VALUE` `pmix.qual.val`	`(pmix_data_array_t*)`	Value being provided consists of the primary key-value pair in first position, followed by one or more key-value qualifiers to be used when subsequently retrieving the primary value
`PMIX_WDIR_USER_SPECIFIED` `pmix.wdir.user`	`(bool)`	User specified the working directory
`PMIX_RUNTIME_OPTIONS` `pmix.runopt`	`(char*)`	Environment-specific runtime directives that control job behavior
`PMIX_ABORT_NON_ZERO_TERM` `pmix.abnz`	`(bool)`	Abort the spawned job if any process terminates with non-zero status
`PMIX_DO_NOT_LAUNCH` `pmix.dnl`	`(bool)`	Execute all procedures to prepare the requested job for launch, but do not launch it. Typically combined with the PMIX_DISPLAY_MAP or PMIX_DISPLAY_MAP_DETAILED for debugging purposes.
`PMIX_SHOW_LAUNCH_PROGRESS` `pmix.showprog`	`(bool)`	Provide periodic progress reports on job launch procedure (e.g., after every 100 processes have been spawned)
`PMIX_AGGREGATE_HELP` `pmix.agg.help`	`(bool)`	Aggregate help messages, reporting each unique help message once accompanied by the number of processes that reported it
`PMIX_REPORT_CHILD_SEP` `pmix.rptchildsep`	`(bool)`	Report the exit status of any child jobs spawned by the primary job separately. If false, then the final exit status reported will be zero if the primary job and all spawned jobs exit normally, or the first non-zero status returned by either primary or child jobs.
`PMIX_DISPLAY_MAP_DETAILED` `pmix.dispmapdet`	`(bool)`	display a highly detailed placement map upon spawn
`PMIX_DISPLAY_ALLOCATION` `pmix.dispalloc`	`(bool)`	display the resource allocation
`PMIX_DISPLAY_TOPOLOGY` `pmix.disptopo`	`(char*)`	comma-delimited list of hosts whose topology is to be displayed
`PMIX_DISPLAY_PROCESSORS` `pmix.dispcpus`	`(char*)`	comma-delimited list of hosts whose available CPUs are to be displayed
`PMIX_DISPLAY_PARSEABLE_OUTPUT` `pmix.dispparse`	`(bool)`	display requested info in a format more amenable to machine parsing
`PMIX_SORTED_PROC_ARRAY` `pmix.sorted.parr`	`(bool)`	Proc array being passed has been sorted
`PMIX_QUERY_PROVISIONAL_ABI_VERSION` `pmix.qry.prabiver`	`(char*)`	The PMIx Standard Provisional ABI version(s) supported, returned in the form of a comma separated list of “MAJOR.MINOR” pairs
`PMIX_QUERY_STABLE_ABI_VERSION` `pmix.qry.stabiver`	`(char*)`	The PMIx Standard Stable ABI version(s) supported, returned in the form of a comma separated list of “MAJOR.MINOR” pairs

Note

The attribute PMIX_DEBUG_STOP_IN_APP has been modified to only support a PMIX_BOOL value instead of an optional array of ranks due to questions over the use-case calling for stopping a subset of a job’s processes while allowing others to run “free”.

4.30. Datatypes

pmix_value_cmp_t: an enum indicating the relative value of two pmix_value_t objects. Values include:
- PMIX_EQUAL
- PMIX_VALUE1_GREATER
- PMIX_VALUE2_GREATER
- PMIX_VALUE_TYPE_DIFFERENT
- PMIX_VALUE_INCOMPATIBLE_OBJECTS
- PMIX_VALUE_COMPARISON_NOT_AVAIL
pmix_boolean_t: an enum indicating boolean state of a pmix_value_t (possibly contained in a pmix_info_t object):
- PMIX_BOOL_TRUE
- PMIX_BOOL_FALSE
- PMIX_NON_BOOL
pmix_disk_stats_t: contains statistics on disk read/write operations
pmix_net_stats_t: contains statistics on network activity
pmix_node_stats_t: contains statistics on node resource usage
pmix_proc_stats_t: contains statistics on process resource usage

4.30.1. Datatype static initializers

Static initializers were added for each complex data type (i.e., a data type defined as a struct). Most are contained in the Standard, but the following extensions have been provided:

PMIX_PROC_STATS_STATIC_INIT
PMIX_DISK_STATS_STATIC_INIT
PMIX_NET_STATS_STATIC_INIT
PMIX_NODE_STATS_STATIC_INIT

4.31. Macros

Although the convenience macros have been deprecated, several were added (in deprecated form) that previously were missing. These are added for symmetry to support those who continue to use the macros, and include:

PMIX_XFER_PROCID: transfer a pmix_proc_t to another one (non-destructive copy)
PMIX_INFO_SET_END: mark this pmix_info_t as being at the end of an array
PMIX_INFO_SET_PERSISTENT: mark that the data in this pmix_info_t is not to be released by PMIX_Info_destruct() (or its macro form)
PMIX_INFO_SET_QUALIFIER: mark this pmix_info_t as a qualifier to the primary key
PMIX_INFO_IS_PERSISTENT: test if this pmix_info_t has been marked as persistent
PMIX_INFO_IS_QUALIFIER: test if this pmix_info_t has been marked as a qualifier
PMIX_DATA_ARRAY_INIT: initialize a pmix_data_array_t
PMIX_CHECK_TRUE: check if a pmix_value_t is boolean true (supports string as well as traditional boolean values)
PMIX_CHECK_BOOL: check if a pmix_value_t is a boolean value (supports string as well as traditional boolean values)

Macros supporting pmix_disk_stats_t objects:

PMIX_DISK_STATS_CONSTRUCT
PMIX_DISK_STATS_CREATE
PMIX_DISK_STATS_DESTRUCT
PMIX_DISK_STATS_FREE
PMIX_DISK_STATS_RELEASE

Macros supporting pmix_net_stats_t objects:

PMIX_NET_STATS_CONSTRUCT
PMIX_NET_STATS_CREATE
PMIX_NET_STATS_DESTRUCT
PMIX_NET_STATS_FREE
PMIX_NET_STATS_RELEASE

Macros supporting pmix_node_stats_t objects:

PMIX_NODE_STATS_CONSTRUCT
PMIX_NODE_STATS_CREATE
PMIX_NODE_STATS_DESTRUCT
PMIX_NODE_STATS_RELEASE

Macros supporting pmix_proc_stats_t objects:

PMIX_PROC_STATS_CONSTRUCT
PMIX_PROC_STATS_CREATE
PMIX_PROC_STATS_DESTRUCT
PMIX_PROC_STATS_FREE
PMIX_PROC_STATS_RELEASE

4.32. Scheduler Integration

OpenPMIx has taken some initial steps towards supporting the integration of schedulers to runtime environments (RTEs) using PMIx as the middleware. Supporting definitions will continue to be added going forward. This section describes the current state of those definitions.

4.32.1. Session Control Function

Used by the scheduler to request a session control action by the RTE - e.g., setup a session (allocate the specified nodes to the new session, provision the nodes with the specified image, setup a user-level DVM across those nodes, and startup the given application under control of that DVM). In addition to setting up a new session, the function can be called to direct that a currently executing session be preempted or terminated. The sessionID identifies the session to which the specified control action is to be applied. A UINT32_MAX value can be used to indicate all sessions under the caller’s control.

Also used by the RTE to report a change in session state - e.g., that the session has completed The directives are provided as pmix_info_t structs in the directives array. The callback function provides a status to indicate whether or not the request was granted, and to provide some information as to the reason for any denial in the pmix_info_cbfunc_t array of pmix_info_t structures. If non-NULL, then the specified release_fn must be called when the callback function completes — this will be used to release any provided pmix_info_t array.

Passing NULL as the cbfunc to this call indicates that it shall be treated as a blocking operation, with the return status indicative of the overall operation’s completion.

pmix_status_t PMIx_Session_control(uint32_t sessionID,
                                   const pmix_info_t directives[], size_t ndirs,
                                   pmix_info_cbfunc_t cbfunc, void *cbdata);

4.32.2. Session Control Attributes

Schedulers calling to create a session are required to provide:

the effective userID and groupID that the session should have when instantiated.
description of the resources that are to be included in the session
if applicable, the image that should be provisioned on nodes included in the session
an array of applications (if any) that are to be started in the session once instantiated

Attributes supported by this API when called by the scheduler include:

Attribute	Type	Description
`PMIX_SESSION_APP` `pmix.ssn.app`	`(pmix_data_array_t*)`	Array of `pmix_app_t` to be executed in the assigned session upon session instantiation
`PMIX_SESSION_PROVISION` `pmix.ssn.pvn`	`(pmix_data_array_t*)`	description of nodes to be provisioned with specified image
`PMIX_SESSION_PROVISION_NODES` `pmix.ssn.pvnnds`	`(char*)`	regex identifying nodes that are to be provisioned
`PMIX_SESSION_PROVISION_IMAGE` `pmix.ssn.pvnimg`	`(char*)`	name of the image that is to be provisioned
`PMIX_SESSION_PAUSE` `pmix.ssn.pause`	`(bool)`	pause all jobs in the specified session
`PMIX_SESSION_RESUME` `pmix.ssn.resume`	`(bool)`	“un-pause” all jobs in the specified session
`PMIX_SESSION_TERMINATE` `pmix.ssn.terminate`	`(bool)`	terminate all jobs in the specified session and recover all resources included in the session.
`PMIX_SESSION_PREEMPT` `pmix.ssn.preempt`	`(bool)`	preempt indicated jobs (given in accompanying `pmix_info_t` via the `PMIX_NSPACE` attribute) in the specified session and recover all their resources. If no `PMIX_NSPACE` is specified, then preempt all jobs in the session.
`PMIX_SESSION_RESTORE` `pmix.ssn.restore`	`(bool)`	restore indicated jobs (given in accompanying `pmix_info_t` via the `PMIX_NSPACE` attribute) in the specified session, including all their resources. If no `PMIX_NSPACE` is specified, then restore all jobs in the session.
`PMIX_SESSION_SIGNAL` `pmix.ssn.sig`	`(int)`	send given signal to all processes of every job in the session

Attributes supported by this API when called by the RTE include:

Attribute	Type	Description
`PMIX_SESSION_COMPLETE` `pmix.ssn.complete`	`(bool)`	specified session has completed, all resources have been recovered and are available for scheduling. Must include `pmix_info_t` indicating ID and returned status of any jobs executing in the session.

4.32.3. Server module function pointers

The PMIx server module was extended to include the following interface that is used by the PMIx server to pass control requests received by the RTE from the scheduler. These include requests to establish a newly allocated session, preempt jobs, etc. A UINT32_MAX value for the sessionID indicates that the specified action shall be applied to all currently existing sessions.

Provide a session control operation request

typedef pmix_status_t (*pmix_server_session_control_fn_t)(
                              const pmix_proc_t *requestor,
                              uint32_t sessionID,
                              const pmix_info_t directives[], size_t ndirs,
                              pmix_info_cbfunc_t cbfunc, void *cbdata);

4.32.4. Server module attributes

A number of allocation related attributes have already been defined in the Standard. These can be used to describe the request (e.g., the resources to be included in the session). The following attribute has been added to that list:

Attribute	Type	Description
`PMIX_SESSION_CTRL_ID` `pmix.ssnctrl.id`	`(char*)`	provide a string identifier for this request. This identifier shall be included in all subsequent interactions relating to the request.

4.32.5. Scheduler query attributes

The scheduler typically discovers its available resources by querying the RTE for a list of them. The following attributes augment those already in the Standard to support the query:

Attribute	Type	Description
`PMIX_QUERY_ALLOCATION` `pmix.query.allc`	`(pmix_data_array_t*)`	returns an array of `pmix_info_t` describing the nodes known to the server. Each array element will consist of the `PMIX_NODE_INFO` key containing a `pmix_data_array_t` of `pmix_info_t`. The first element of the array must be the hostname of that node, with additional info on the node in subsequent entries. SUPPORTED_QUALIFIER: a `PMIX_ALLOC_ID` qualifier indicating the specific allocation of interest
`PMIX_TOPOLOGY_INDEX` `pmix.topo.index`	`(int)`	index of a topology in a storage array. Used when returning an allocation to avoid duplicate topology information - the RTE can return an array of topologies and then indicate the index to the topology as part of each node entry.

4.32.6. Allocation attributes

A number of allocation related attributes have already been defined in the Standard. These can be used to describe the request (e.g., the number and type of resources being requested). The following attribute has been added to that list:

Attribute	Type	Description
`PMIX_ALLOC_PREEMPTIBLE` `pmix.alloc.preempt`	`(bool)`	by default, all jobs in the resulting allocation are to be considered preemptible (can be overridden at per-job level)

4.32.7. Allocation directive values

The PMIx_Allocation_request API includes a directive parameter to specify the operation being requested. These values were extended to include:

PMIX_ALLOC_REQ_CANCEL (value: 5): Cancel the indicated allocation request