OpenMPI
0.1.1
|
This interface is designed to hide the back-end details of how IB RC connections are made from the rest of the wv BTL. More...
Go to the source code of this file.
Data Structures | |
struct | ompi_btl_wv_connect_base_component_t |
CPC component struct. More... | |
struct | ompi_btl_wv_connect_base_module_data_t |
Meta data about a CPC module. More... | |
struct | ompi_btl_wv_connect_base_module_t |
Struct for holding CPC module and associated meta data. More... | |
Macros | |
#define | BCF_MAX_NAME 64 |
Typedefs | |
typedef void(* | ompi_btl_wv_connect_base_component_register_fn_t )(void) |
Function to register MCA params in the connect functions. More... | |
typedef int(* | ompi_btl_wv_connect_base_component_init_fn_t )(void) |
This function is invoked once by the wv BTL component during startup. More... | |
typedef int(* | ompi_btl_wv_connect_base_func_component_query_t )(struct mca_btl_wv_module_t *btl, struct ompi_btl_wv_connect_base_module_t **cpc) |
Query the CPC to see if it wants to run on a specific port (i.e., a specific BTL module). More... | |
typedef int(* | ompi_btl_wv_connect_base_component_finalize_fn_t )(void) |
This function is invoked once by the wv BTL component during shutdown. More... | |
typedef struct ompi_btl_wv_connect_base_component_t | ompi_btl_wv_connect_base_component_t |
Convenience typedef. | |
typedef int(* | ompi_btl_wv_connect_base_module_endpoint_init_fn_t )(struct mca_btl_base_endpoint_t *endpoint) |
Function called when an endpoint has been created and has been associated with a CPC. | |
typedef int(* | ompi_btl_wv_connect_base_module_start_connect_fn_t )(struct ompi_btl_wv_connect_base_module_t *cpc, struct mca_btl_base_endpoint_t *endpoint) |
Function to initiate a connection to a remote process. | |
typedef int(* | ompi_btl_wv_connect_base_module_endpoint_finalize_fn_t )(struct mca_btl_base_endpoint_t *endpoint) |
Function called when an endpoint is being destroyed. | |
typedef int(* | ompi_btl_wv_connect_base_module_finalize_fn_t )(struct mca_btl_wv_module_t *btl, struct ompi_btl_wv_connect_base_module_t *cpc) |
Function to finalize the CPC module. More... | |
typedef struct ompi_btl_wv_connect_base_module_data_t | ompi_btl_wv_connect_base_module_data_t |
Meta data about a CPC module. More... | |
typedef struct ompi_btl_wv_connect_base_module_t | ompi_btl_wv_connect_base_module_t |
Struct for holding CPC module and associated meta data. | |
This interface is designed to hide the back-end details of how IB RC connections are made from the rest of the wv BTL.
There are module-like instances of the implemented functionality (dlopen and friends are not used, but all the functionality is accessed through struct's of function pointers, so you can swap between multiple different implementations at run time, just like real components). Hence, these entities are referred to as "Connect Pseudo-Components" (CPCs).
The CPCs are referenced by their names (e.g., "oob", "rdma_cm").
CPCs are split into components and modules, similar to all other MCA frameworks in this code base.
Before diving into the CPC interface, let's discuss some terminology and mappings of data structures:
Component interface:
NOTE: The connect_base_register() function will process the btl_wv_cpc_include and btl_wv_cpc_exclude MCA parameters and automatically include/exclude CPCs as relevant. If a CPC is excluded, none of its other interface functions will be invoked for the duration of the process.
component_finalize(): The wv BTL's component_close() function calls connect_base_finalize(), which, in turn, calls the component_finalize() function on all available CPCs. Note that all CPC modules will have been finalized by this point; the CPC component_finalize() function is a chance for the CPC to clean up any component-specific resources.
Module interface:
cbm_component member: A pointer pointing to the single, global instance of the CPC component. This member is used for creating a unique index representing the modules' component so that it can be shared with remote peer processes.
cbm_priority member: An integer between 0 and 100, inclusive, representing the priority of this CPC.
cbm_modex_message member: A pointer to a blob buffer that will be included in the modex message for this port for this CPC (it is assumed that this blob is a) only understandable by the corresponding CPC in the peer process, and b) contains specific addressing/contact information for this port's CPC module).
cbm_modex_message_len member: The length of the cbm_modex_message blob, in bytes.
cbm_endpoint_init(): Called during endpoint creation, allowing a CPC module to cache information on the endpoint. A pointer to the endpoint's CPC module is already cached on the endpoint.
cbm_start_connect(): initiate a connection to a remote peer. The CPC is responsible for setting itself up for asyncronous operation for progressing the outgoing connection request.
cbm_endpoint_finalize(): Called during the endpoint destrouction, allowing the CPC module to destroy anything that it cached on the endpoint.
cbm_finalize(): shut down all asynchronous handling and clean up any state that was setup for this CPC module/BTL. Some CPCs setup asynchronous support on a per-HCA/NIC basis (vs. per-port/LID). It is the reponsibility of the CPC to figure out such issues (e.g., via reference counting) – there is no notification from the upper-level BTL about when an entire HCA/NIC is no longer being used. There is only this function, which tells when a specific CPC/BTL module is no longer being used.
cbm_uses_cts: a bool that indicates whether the CPC will use the CTS protocol or not.
There are two functions in the main wv BTL that the CPC may call:
typedef int(* ompi_btl_wv_connect_base_component_finalize_fn_t)(void) |
This function is invoked once by the wv BTL component during shutdown.
It is intended to have CPC component-wide shutdown.
typedef int(* ompi_btl_wv_connect_base_component_init_fn_t)(void) |
This function is invoked once by the wv BTL component during startup.
It is intended to have CPC component-wide startup.
Return value:
typedef void(* ompi_btl_wv_connect_base_component_register_fn_t)(void) |
Function to register MCA params in the connect functions.
It returns no value, so it cannot fail.
typedef int(* ompi_btl_wv_connect_base_func_component_query_t)(struct mca_btl_wv_module_t *btl, struct ompi_btl_wv_connect_base_module_t **cpc) |
Query the CPC to see if it wants to run on a specific port (i.e., a specific BTL module).
If the component init function previously returned OMPI_SUCCESS, this function is invoked once per BTL module creation (i.e., for each port found by an MPI process). If this CPC wants to be used on this BTL module, it returns a CPC module that is specific to this BTL module.
The BTL module in question is passed to the function; all of its attributes can be used to query to see if it's eligible for this CPC.
If it is eligible, the CPC is responsible for creating a corresponding CPC module, filling in all the relevant fields on the modules, and for setting itself up to run (per above) and returning a CPC module (this is effectively the "module_init" function). Note that the module priority must be between 0 and 100 (inclusive). When multiple CPCs are eligible for a single module, the CPC with the highest priority will be used.
Return value:
Meta data about a CPC module.
This is in a standalone struct because it is used in both the CPC module struct and the wv_btl_proc_t struct to hold information received from the modex.
typedef int(* ompi_btl_wv_connect_base_module_finalize_fn_t)(struct mca_btl_wv_module_t *btl, struct ompi_btl_wv_connect_base_module_t *cpc) |
Function to finalize the CPC module.
It is called once when the CPC module's corresponding wv BTL module is being finalized.