\hypertarget{btl__openib__failover_8h}{\section{ompi/mca/btl/openib/btl\+\_\+openib\+\_\+failover.h File Reference}
\label{btl__openib__failover_8h}\index{ompi/mca/btl/openib/btl\+\_\+openib\+\_\+failover.\+h@{ompi/mca/btl/openib/btl\+\_\+openib\+\_\+failover.\+h}}
}


Functions called by B\+T\+L to handle error events.  


\subsection*{Functions}
\begin{DoxyCompactItemize}
\item 
B\+E\+G\+I\+N\+\_\+\+C\+\_\+\+D\+E\+C\+L\+S void \hyperlink{btl__openib__failover_8h_abc4d44d681586c7e47339740020abd0a}{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+endpoint\+\_\+error} (\hyperlink{structmca__btl__openib__module__t}{mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t} $\ast$openib\+\_\+btl, \hyperlink{structmca__btl__base__descriptor__t}{mca\+\_\+btl\+\_\+base\+\_\+descriptor\+\_\+t} $\ast$des, int qp, \hyperlink{structompi__proc__t}{ompi\+\_\+proc\+\_\+t} $\ast$remote\+\_\+proc, \hyperlink{structmca__btl__base__endpoint__t}{mca\+\_\+btl\+\_\+openib\+\_\+endpoint\+\_\+t} $\ast$endpoint)
\begin{DoxyCompactList}\small\item\em This function is called when we get an error on the completion event of a fragment. \end{DoxyCompactList}\item 
void \hyperlink{btl__openib__failover_8h_a4027610aedc70570559b30d125592eac}{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+btl\+\_\+error} (\hyperlink{structmca__btl__openib__module__t}{mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t} $\ast$openib\+\_\+btl)
\begin{DoxyCompactList}\small\item\em This functions allows an error to map out the entire B\+T\+L. \end{DoxyCompactList}\item 
void \hyperlink{btl__openib__failover_8h_ac7e75cc42802b695c3451f2a0ba58183}{btl\+\_\+openib\+\_\+handle\+\_\+failover\+\_\+control\+\_\+messages} (\hyperlink{structmca__btl__openib__control__header__t}{mca\+\_\+btl\+\_\+openib\+\_\+control\+\_\+header\+\_\+t} $\ast$ctl\+\_\+hdr, \hyperlink{structmca__btl__base__endpoint__t}{mca\+\_\+btl\+\_\+openib\+\_\+endpoint\+\_\+t} $\ast$ep)
\begin{DoxyCompactList}\small\item\em This function gets called when a control message is received that is one of the following types\+: M\+C\+A\+\_\+\+B\+T\+L\+\_\+\+O\+P\+E\+N\+I\+B\+\_\+\+C\+O\+N\+T\+R\+O\+L\+\_\+\+E\+P\+\_\+\+B\+R\+O\+K\+E\+N M\+C\+A\+\_\+\+B\+T\+L\+\_\+\+O\+P\+E\+N\+I\+B\+\_\+\+C\+O\+N\+T\+R\+O\+L\+\_\+\+E\+P\+\_\+\+E\+A\+G\+E\+R\+\_\+\+R\+D\+M\+A\+\_\+\+E\+R\+R\+O\+R message Note that we are using the working connection to send information about the broken connection. \end{DoxyCompactList}\end{DoxyCompactItemize}


\subsection{Detailed Description}
Functions called by B\+T\+L to handle error events. 



\subsection{Function Documentation}
\hypertarget{btl__openib__failover_8h_ac7e75cc42802b695c3451f2a0ba58183}{\index{btl\+\_\+openib\+\_\+failover.\+h@{btl\+\_\+openib\+\_\+failover.\+h}!btl\+\_\+openib\+\_\+handle\+\_\+failover\+\_\+control\+\_\+messages@{btl\+\_\+openib\+\_\+handle\+\_\+failover\+\_\+control\+\_\+messages}}
\index{btl\+\_\+openib\+\_\+handle\+\_\+failover\+\_\+control\+\_\+messages@{btl\+\_\+openib\+\_\+handle\+\_\+failover\+\_\+control\+\_\+messages}!btl\+\_\+openib\+\_\+failover.\+h@{btl\+\_\+openib\+\_\+failover.\+h}}
\subsubsection[{btl\+\_\+openib\+\_\+handle\+\_\+failover\+\_\+control\+\_\+messages}]{\setlength{\rightskip}{0pt plus 5cm}void btl\+\_\+openib\+\_\+handle\+\_\+failover\+\_\+control\+\_\+messages (
\begin{DoxyParamCaption}
\item[{{\bf mca\+\_\+btl\+\_\+openib\+\_\+control\+\_\+header\+\_\+t} $\ast$}]{ctl\+\_\+hdr, }
\item[{{\bf mca\+\_\+btl\+\_\+openib\+\_\+endpoint\+\_\+t} $\ast$}]{ep}
\end{DoxyParamCaption}
)}}\label{btl__openib__failover_8h_ac7e75cc42802b695c3451f2a0ba58183}


This function gets called when a control message is received that is one of the following types\+: M\+C\+A\+\_\+\+B\+T\+L\+\_\+\+O\+P\+E\+N\+I\+B\+\_\+\+C\+O\+N\+T\+R\+O\+L\+\_\+\+E\+P\+\_\+\+B\+R\+O\+K\+E\+N M\+C\+A\+\_\+\+B\+T\+L\+\_\+\+O\+P\+E\+N\+I\+B\+\_\+\+C\+O\+N\+T\+R\+O\+L\+\_\+\+E\+P\+\_\+\+E\+A\+G\+E\+R\+\_\+\+R\+D\+M\+A\+\_\+\+E\+R\+R\+O\+R message Note that we are using the working connection to send information about the broken connection. 

That is why we have to look at the various information in the control message to figure out which endpoint is broken. It is (obviously) not the one the message was received on, because we would not have received the message in that case. In the case of the B\+R\+O\+K\+E\+N message, that means the remote side is notifying us that it has brought down its half of the connection. Therefore, we need to bring out half down. This is done because it has been observed that there are cases where only one side of the connection actually sees the error. This means we can be left in a state where one side believes it has two B\+T\+Ls, but the other side believes it only has one. This can cause problems. In the case of the E\+A\+G\+E\+R\+\_\+\+R\+D\+M\+A\+\_\+\+E\+R\+R\+O\+R, see elsewhere in the code what we are doing. 
\begin{DoxyParams}{Parameters}
{\em ctl\+\_\+hdr} & Pointer control header that was received \\
\hline
\end{DoxyParams}


References mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::eager\+\_\+rdma\+\_\+local, mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::endpoint\+\_\+proc, mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::endpoint\+\_\+state, mca\+\_\+btl\+\_\+openib\+\_\+device\+\_\+t\+::endpoints, mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t\+::error\+\_\+cb, error\+\_\+out\+\_\+all\+\_\+pending\+\_\+frags(), mca\+\_\+btl\+\_\+openib\+\_\+eager\+\_\+rdma\+\_\+local\+\_\+t\+::head, mca\+\_\+btl\+\_\+openib\+\_\+component\+\_\+t\+::ib\+\_\+num\+\_\+btls, mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t\+::lid, mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::nbo, opal\+\_\+output\+\_\+verbose(), opal\+\_\+pointer\+\_\+array\+\_\+get\+\_\+item(), opal\+\_\+pointer\+\_\+array\+\_\+get\+\_\+size(), mca\+\_\+btl\+\_\+openib\+\_\+component\+\_\+t\+::openib\+\_\+btls, O\+R\+T\+E\+\_\+\+P\+R\+O\+C\+\_\+\+M\+Y\+\_\+\+N\+A\+M\+E, ompi\+\_\+proc\+\_\+t\+::proc\+\_\+name, mca\+\_\+btl\+\_\+elan\+\_\+proc\+\_\+t\+::proc\+\_\+ompi, mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::rem\+\_\+info, and orte\+\_\+process\+\_\+name\+\_\+t\+::vpid.

\hypertarget{btl__openib__failover_8h_a4027610aedc70570559b30d125592eac}{\index{btl\+\_\+openib\+\_\+failover.\+h@{btl\+\_\+openib\+\_\+failover.\+h}!mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+btl\+\_\+error@{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+btl\+\_\+error}}
\index{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+btl\+\_\+error@{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+btl\+\_\+error}!btl\+\_\+openib\+\_\+failover.\+h@{btl\+\_\+openib\+\_\+failover.\+h}}
\subsubsection[{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+btl\+\_\+error}]{\setlength{\rightskip}{0pt plus 5cm}void mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+btl\+\_\+error (
\begin{DoxyParamCaption}
\item[{{\bf mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t} $\ast$}]{openib\+\_\+btl}
\end{DoxyParamCaption}
)}}\label{btl__openib__failover_8h_a4027610aedc70570559b30d125592eac}


This functions allows an error to map out the entire B\+T\+L. 

First a call is made up to the P\+M\+L to map out all connections from this B\+T\+L. Then a message is sent to all the endpoints connected to this B\+T\+L. This function is enabled by the btl\+\_\+openib\+\_\+port\+\_\+error\+\_\+failover M\+C\+A parameter. If that parameter is not set, then this function does not do anything. 
\begin{DoxyParams}{Parameters}
{\em openib\+\_\+btl} & Pointer to B\+T\+L that had the error \\
\hline
\end{DoxyParams}


References mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::endpoint\+\_\+state, mca\+\_\+btl\+\_\+openib\+\_\+device\+\_\+t\+::endpoints, mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t\+::error\+\_\+cb, error\+\_\+out\+\_\+all\+\_\+pending\+\_\+frags(), mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t\+::lid, mca\+\_\+btl\+\_\+openib\+\_\+endpoint\+\_\+notify(), opal\+\_\+pointer\+\_\+array\+\_\+get\+\_\+item(), and opal\+\_\+pointer\+\_\+array\+\_\+get\+\_\+size().

\hypertarget{btl__openib__failover_8h_abc4d44d681586c7e47339740020abd0a}{\index{btl\+\_\+openib\+\_\+failover.\+h@{btl\+\_\+openib\+\_\+failover.\+h}!mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+endpoint\+\_\+error@{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+endpoint\+\_\+error}}
\index{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+endpoint\+\_\+error@{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+endpoint\+\_\+error}!btl\+\_\+openib\+\_\+failover.\+h@{btl\+\_\+openib\+\_\+failover.\+h}}
\subsubsection[{mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+endpoint\+\_\+error}]{\setlength{\rightskip}{0pt plus 5cm}B\+E\+G\+I\+N\+\_\+\+C\+\_\+\+D\+E\+C\+L\+S void mca\+\_\+btl\+\_\+openib\+\_\+handle\+\_\+endpoint\+\_\+error (
\begin{DoxyParamCaption}
\item[{{\bf mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t} $\ast$}]{openib\+\_\+btl, }
\item[{{\bf mca\+\_\+btl\+\_\+base\+\_\+descriptor\+\_\+t} $\ast$}]{des, }
\item[{int}]{qp, }
\item[{{\bf ompi\+\_\+proc\+\_\+t} $\ast$}]{remote\+\_\+proc, }
\item[{{\bf mca\+\_\+btl\+\_\+openib\+\_\+endpoint\+\_\+t} $\ast$}]{endpoint}
\end{DoxyParamCaption}
)}}\label{btl__openib__failover_8h_abc4d44d681586c7e47339740020abd0a}


This function is called when we get an error on the completion event of a fragment. 

We check to see what type of fragment it is and act accordingly. In most cases, we first call up into the P\+M\+L and have it map out this connection for any future communication. In addition, this function will possibly send some control messages over the other openib B\+T\+L. The first control message will tell the remote side to also map out this connection. The second control message makes sure the eager R\+D\+M\+A connection remains in a sane state. See that function for more details. 
\begin{DoxyParams}{Parameters}
{\em openib\+\_\+btl} & Pointer to B\+T\+L that had the error \\
\hline
{\em des} & Pointer to descriptor that had the error \\
\hline
{\em qp} & Queue pair that had the error \\
\hline
{\em remote\+\_\+proc} & Pointer to process that had the error \\
\hline
{\em endpoint} & Pointer to endpoint that had the error \\
\hline
\end{DoxyParams}


References mca\+\_\+btl\+\_\+base\+\_\+descriptor\+\_\+t\+::des\+\_\+cbfunc, mca\+\_\+btl\+\_\+base\+\_\+descriptor\+\_\+t\+::des\+\_\+flags, mca\+\_\+btl\+\_\+base\+\_\+descriptor\+\_\+t\+::des\+\_\+src, mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::endpoint\+\_\+state, mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t\+::error\+\_\+cb, error\+\_\+out\+\_\+all\+\_\+pending\+\_\+frags(), mca\+\_\+btl\+\_\+base\+\_\+endpoint\+\_\+t\+::get\+\_\+tokens, mca\+\_\+btl\+\_\+openib\+\_\+module\+\_\+t\+::lid, mca\+\_\+btl\+\_\+openib\+\_\+endpoint\+\_\+notify(), opal\+\_\+list\+\_\+remove\+\_\+first(), opal\+\_\+output\+\_\+verbose(), and O\+P\+A\+L\+\_\+\+T\+H\+R\+E\+A\+D\+\_\+\+A\+D\+D32.

