OpenMPI  0.1.1
coll_sm_barrier.c File Reference
#include "ompi_config.h"
#include "ompi/constants.h"
#include "ompi/communicator/communicator.h"
#include "ompi/mca/coll/coll.h"
#include "opal/sys/atomic.h"
#include "coll_sm.h"

Functions

int mca_coll_sm_barrier_intra (struct ompi_communicator_t *comm, mca_coll_base_module_t *module)
 Shared memory barrier. More...
 

Function Documentation

int mca_coll_sm_barrier_intra ( struct ompi_communicator_t comm,
mca_coll_base_module_t module 
)

Shared memory barrier.

Tree-based algorithm for a barrier: a fan in to rank 0 followed by a fan out using the barrier segments in the shared memory area.

There are 2 sets of barrier buffers – since there can only be, at most, 2 outstanding barriers at any time, there is no need for more than this. The generalized in-use flags, control, and data segments are not used.

The general algorithm is for a given process to wait for its N children to fan in by monitoring a uint32_t in its barrier "in" buffer. When this value reaches N (i.e., each of the children have atomically incremented the value), then the process atomically increases the uint32_t in its parent's "in" buffer. Then the process waits for the parent to set a "1" in the process' "out" buffer. Once this happens, the process writes a "1" in each of its children's "out" buffers, and returns.

There's corner cases, of course, such as the root that has no parent, and the leaves that have no children. But that's the general idea.

References mca_coll_sm_component, mca_coll_sm_comm_t::mcb_barrier_control_children, mca_coll_sm_comm_t::mcb_barrier_control_me, mca_coll_sm_comm_t::mcb_barrier_control_parent, mca_coll_sm_comm_t::mcb_barrier_count, mca_coll_sm_comm_t::mcb_tree, mca_coll_sm_tree_node_t::mcstn_num_children, opal_atomic_add, and mca_coll_sm_component_t::sm_control_size.