MPF is available via the open source license described at the end of this document
MPF is the successor to HIDEOS which was written with the following goals:
When other IP carriers became available, it was decided that that it was too time consuming to support processors besides the MV162. Thus MPF was developed. It supports the same type of client server relationship as Hideos but support two types of communication: 1) If the client and server reside on the same processor, messages are passed in memory, and 2) If the client and server reside on different processors, message are passed via TCP. The original MPF was only supported on vxWorks. When EPICS base evolved to be Operating System Independent (OSI), the OSI version of MPF was devloped. When this document refers to MPF it means the OSI version. When it refers to EPICS base it means the OSI version of base.
MPF uses the OSI features of EPICS base for threads, mutexes, etc. It should run on any platform supported by EPICS base.
This document describes two main components:
Message Passing Facility
MPF provides client server message passing. The messages are handled by message routers. Both local and remote message routers are provided.Epics Device Support for the MPF
A base class DevMpf is provided for writing epics device support that communicates with MPF.
All IP support has been unbundled from MPF. Thus it is described elsewhere
All code is written in C++. It is written to minimize dependencies between various components. For example the IP support can be used without MPF, MPF without the IP support, etc. The current implementation uses the standard C library but not the standard C++ library.
When MPF is used between processors, communication is via a tcp connection. Thus the two processors can reside anywhere as long as it is possible for them to communicate via tcp.
.../support/mpfosi/
gunzip <file>.tar.gz tar xvf <file>.tar
.../support/mpfosi/mpfosiX-Ywhere X-Y is the release. For example.
.../support/mpfosi/mpf2-1
The inet address must be replaced by the address of the processor that will be running the message servers.iocepicsclient/st.cmd iocepicsserver/st.cmd iocmpfclient/st.cmd iocmpfserver/startServers
Now issue the commands:
cd X-Y make
Now proceed to the next two sections.
The iocBoot directory has a number of sub directories for running mpf tests and/or examples. The best example is a test which connects an epics database to a mpf server. The test can be run in either local (client and server on the same processor) or remote (client and server on different processors)To start the test:
cd iocBoot/iocepicslocal ../../bin/<arch>/test st.cmd
cd iocBoot/iocepicsserver Check the inet address given to tcpMessageRouterServerStart in st.cmd ../../bin/<arch>test st.cmd
cd iocBoot/iocepicsclient check the inet addresses given to tcpMessageRouterClientStart in st.cmd ../../bin/<arch>test st.cmd
testApp/src/DevMpfInt32Test.cpp | This is an example of device support. |
testApp/src/serverInt32.cpp | The echo server for Int32 messages. |
testApp/src/Makefile | The subdirectory shows how to build clients and server applications. |
testApp/Db/*.db | The databases for the example. |
iocBoot/iocepicslocal/st.cmd | The startup file for the local test. |
iocBoot/iocepicsclient/st.cmd iocBoot/iocepicsserver/st.cmd | The startup files for the remote test. |
Studying the above will show how to write servers, device support, build servers and client, and boot servers and clients.
It is a good idea to keep mpf separate from applications which use it. An application can access mpf components by making the following changes:
MPF=<full path name>
<prod>_LIBS += mpfSerial mpfEpics mpf <prod>_LIBS += $(EPICS_BASE_IOC_LIBS)
MPF does not include support for specific ip modules. Such support must be obtained elsewhere.
Yiur xxxInclude.dbd file must include:
include "mpf.dbd"
Your xxxInclude.dbd file can include the following:
include "devStringMpf.dbd" include "mpfSerial.dbd" include "mpfSerialVx.dbd"
NOTE:
The st.cmd file must contain commands to start MPF routers and MPF servers. Look at the iocBoot/iocxxx/st.cmd files provided with MPF for examples.
Messages are sent through a router. Two routers are provided: localRouter and RMR (Remote Message Router). The localRouter passes messages between a client and server residing on the same processor. RMR sends messages between a client and server on different processors via a tcp connection.
typedef void (*clientCallback)(Message *message,void *clientPvt); class MessageClient { public: MessageClient(clientCallback,void *clientPvt); int bind(char *server, int location); int send(Message *message); void *getClientPvt(); private: ... }A client of the message passing system must:
class MessageServer { public: MessageServer(const char* name); void waitForMessage(); Message *receive(); Message *allocReplyMessage(Message *clientMessage,messageType type); int reply(Message *); void setQueueSize(int size); const char *getName() const; void report() const; private: ... }A message server must
Message instances are kept on a free list. Thus malloc is called only if the free list is empty. Free is never called. Each message type has it's own free list.
whereINP("C1 S0 @Int32")
For servers it provides a report like the following:
iocrstserver> mrr clientRouterList serverRouterList 1 RMRServer stateConnected queueSize 100 inQueue 0 replyQueueFull 0 sendPerSec 240 receivePerSec 240 tcpSendPerSec 10 tcpReceivePerSec 10It provides a report for each server router. It shows the connection state. The meaning of the other fields are:
For clients it produces the following type of report
iocrstclient> mrr clientRouterList 1 RMRClient stateConnected queueSize 100 inQueue 0 sendQueueFull 0 Server Int32 has 4 clients. bindState connected sendPerSec 40 receivePerSec 40 tcpSendPerSec 10 tcpReceivePerSec 10 serverRouterListIt produces a report for each client router. It shows if the router is connected. The queueSize, inQueue, and sendQueueFull provide statistics about the send queue. It also shows a list of each server attached to the router.
iocrstserver> msr "Int32" Int32 queueSize 10 inQueue 0 queueRequests 132171 queueFullResponses 0 replyRequests 132171The meaning of the fields are:
Each processor that is a member of a set of MPF clients/servers is assigned a unique location, which is just an integer value. For example if the set is an mv167/mv162 the mv167 could be assigned location 0 and the mv162 location 1. All that matters is that each cpu is assigned a unique location and all cpus agree on the assignments. If local communication is desired then the following command must appear:
localMessageRouterStart(location)The parameter is the location of the local server. Note that the client specifies a location when it binds to a server. Each cpu can start a local router.
If tcp communication is desired then for each remote system the client must have the following command:
tcpMessageRouterClientStart(location,port,"address",bufSize,queueSize)
Each system supporting remote servers must have the following command for each client machine.
tcpMessageRouterServerStart(location,port,"address",bufSize,queueSize)
NOTE CAREFULLY:Each server router can only handle a single client router. A particular client/server pair is established by the client and server having EXACTLY the same (location,port,addresss) tuple. Also each server/client pair must use a unique port number.
<message class> *pmessage = new <message class>;for example an Int32 message can be allocated as follows:
Int32Message *pmessage = new Int32Message;
Int32Message *pmessage =(Int32Message*)pMessageServer-> allocReplyMessage(preceive,messageTypeInt32);where preceive must be a message received from the client. The reason is that the receive message contains information describing where the client is located.
delete pmessage;Thus the server must delete any message received from a client and a client must delete reply messages from the server.
The definition of class Message is:
class Message { public: messageType getType(); int32 getClientType(); void setClientType(int32 type); int32 getClientExtra(); void setClientExtra(int32 extra); virtual ~Message(); virtual int toBuffer(char **buffer); virtual int fromBuffer(const char **buffer); virtual int fromBufferSwitch(const char **buffer); virtual void print() const; static Message *allocate(messageType type); protected: Message(messageType type); .... }
Client and server code normally only call new, delete, the methods described in the following table, and message specific methods. The other methods are called by MPF itself. A message is allocated via a call to new and released via a call to delete. When delete is called the message is placed on a free list. Only one of the message types derived from Message can be created, i.e. it is not possible to create an object of type Message.
The virtual methods must be implement by any class that derives from Message. The toBuffer and fromBuffer methods are called to put and take messages from a buffer that passes over a communication link.
method | Recommended Meaning |
getType | Returns the message type |
getClientType setClientType |
These methods are for use by a client to access field clientType. When the server allocates a reply message via a call to allocateReplyMessage, the clientType from the clients message is copied to the reply message. In general a server should not use these methods. They are intended for clients that send multiple messages to a server and need to match reply messages with messages sent. Classes derived from DevMpf MUST NOT use these because DevMpf itself uses them. Use clientExtra instead. |
getClientExtra setClientExtra |
These access a field clientExtra which is similar to clientType. They can be used by classes derived from DevMpf. |
MPF provides a header file "mpfType.h" which defines typedefs for int16, uint16, int32, uint32, float32, and float64. MPF provides static methods to transfer the following to/from network buffers:
Field | Recommended Meaning |
timeoutUnits | timeoutUnitsSeconds or timeoutUnitsMilliseconds. Default is seconds |
timeout | The timeout the server can use it it has to wait for a device. The units are determined by timeoutUnits.. |
cmd | A command. Servers can supply a header file defining an enum for the commands. |
status | A status value. |
address | For anything that is an int32 and can be interpreted as an address. |
extra | An extra int32 word that client/server can use for anything they want as long as they agree on the meaning. |
Field | Recommended Meaning |
value | The address of a char8 array. The characters are not interpreted or translated by the message class, i.e. they are just a stream of octet values. The standard C library routines such as ::strcpy and ::memcpy can be used on the value field. |
numberRetrys | A useful field for servers. It should be the number of retrys if a command fails. |
eomLen | A useful field for servers. It should be the length of an end of message string. |
eomString | A useful field for servers. It should be a 1 or 2 character end of message string. |
Char8ArrayMessage provides the following methods for allocating and
freeing the string value.
method | usage |
allocValue | This is a static method which looks for the smallest freelist that can provide the needed space. If the requested size is larger than the size for the largest freelist, it calls new to allocate space. |
setSize | Sets the current size for the array. This size MUST be less than that allocated by allocValue. |
getSize | Gets the current array size. |
getMaxSize | This gets the size allocated by the call to allocValue |
freeValue | This puts storage obtained by allocValue back on the appropriate free list or calls delete if the storage was obtained via a call to new. This method is called automatically when a Char8Array message is deleted. |
Field | Recommended Meaning |
value | The Int32 value being passed. |
method | usage |
allocValue | This is a static method which looks for the smallest freelist that can provide the needed space. If the requested size is larger than the size for the largest freelist, it calls new to allocate space. |
setLength | Sets the current length for the array. This length MUST be less than that allocated by allocValue. |
getLength | Gets the current array length. |
getMaxLength | This gets the length allocated by the call to allocValue |
freeValue | This puts storage obtained by allocValue back on the appropriate free list or calls delete if the storage was obtained via a call to new. This method is called automatically when an Int32Array message is deleted. |
Field | Recommended Meaning |
value | The Float64 value being passed |
method | usage |
allocValue | This is a static method which looks for the smallest freelist that can provide the needed space. If the requested size is larger than the size for the largest freelist, it calls new to allocate space. |
setLength | Sets the current length for the array. This length MUST be less than that allocated by allocValue. |
getLength | Gets the current array length. |
getMaxLength | This gets the length allocated by the call to allocValue |
freeValue | This puts storage obtained by allocValue back on the appropriate free list or calls delete if the storage was obtained via a call to new. This method is called automatically when an Float64Array message is deleted. |
field | usage |
baud | Baud rate. 0 means don't change |
stopBits | Stop bits. Normally 1 or 2. 0 means don't change. |
bitsPerChar | Bits per character. Normally 5,6,7, or 8. 0 means don't change. |
parity | Single character. Normally 'E', 'O', or 'N'. 0 means don't change. |
flowControl | Single character. Normally 'H' or 'N'. 0 means don't change. |
DevMpf is an abstract base class for implementing EPICS device support.
// return codes for startIO and completeIO. #define MPF_OK 0 #define MPF_NoConvert 2 enum replyType {replyTypeNone, replyTypeCompleteIO, replyTypeReceiveReply};
class DevMpf { public: DevMpf(dbCommon*,link*,bool iointValid); // Following must be implemented by device support modules virtual long startIO(dbCommon*)=0; // start async IO virtual long completeIO(dbCommon*,Message *)=0; // end async IO // the following can be implemented by device support modules virtual void receiveReply(dbCommon*,Message *); virtual void connectIO(dbCommon*,Message *); // connection message virtual void outOfBandIO(dbCommon*,OutOfBandMessage *);// outOfBand virtual long convert(dbCommon*,int pass); // do linear conversion // The following must be called after object is constructed void bind(); // The following are usefull during record initilization bool connectWait() {return(connectWait(5.0));} bool connectWait(double timeout); // send a message to MPF server with no reply expected int send(Message*); // send a message to MPF server and wait for reply via completeIO int sendReply(Message*); // send a message stating the type of reply int send(Message*,replyType); int sendWait(Message *pmessage,replyType type) {return(sendWait(pmessage,type,5.0));} int sendWait(Message*,replyType,double timeout); // This routine gets a pointer to the user portion of the parm field long getStatus() const { return(status);} // Following are DSET routines static long read_write(void*); // generic DSET read/write routine static long ioint(int cmd,dbCommon*,IOSCANPVT* iopvt); // DSET i/o intr static long linconv(void*,int); // calls convert IOSCANPVT ioscanpvt; bool iointValid; bool isConnected() { return((connectState==connectYes) ? true : false);} private: ...
}Before describing the methods a few comments may be helpful.
If the support is for a simple device, i.e. a device that can be supported via the following:
For devices that are not simple, e.g. devices that must take special action when connecting to the server, additional methods are available: In particular:
ioscanpvt is for provided for convenience. If a derived class
wishes to support io interrupt processing it can request that the base class
perform the necessary initialization. The derived class must, however, call
scanIoRequest.
method | Usage | meaning |
DevMpf | Constructor | The constructor must be given the address of the record and link (INP or OUT). iointValid specifies if the base class should initialize ioscanpvt. Normally this is only invoked via the constructor of a class derived from class DevMpf. After an object of the derived class is completely constructed, the bind method MUST be called. |
startIO | Called by DevMpf | When a request is made to process a record this is called only when
the following conditions are all true: The server is connected, PACT
is false, and no reply message from the server is available. This
MUST be implemented by any class derived from DevMpf.
This method is always called as a result of a record being processed and the record is not active. If startIO issues a send(message,replyTypeCompleteIO) then PACT is set TRUE, i.e. the record will not complete processing until the reply message is received and completeIO is called. |
completeIO | Called by DevMpf | DevMpf calls this when the reply to a
send(message,replyTypeCompleteIO) is received. This MUST be
implemented by any class that derives from DevMpf.
This method is always called as a result of a record being processed |
receiveReply | Called by DevMpf | DevMpf calls this when the reply to a
send(message,replyTypeReceiveReply) is received. This call does not
involve record processing.
The default method just prints a message and deletes the message it received. |
connectIO | Called by DevMpf Default supplied by DevMpf. |
DevMpf calls this when it receives a ConnectMessage. DevMpf
provides a default implementation.
If device support implements this method it should call DevMpf::connectIO (just before returning) to ensure correct behavior. |
outOfBandIO | Called by DevMpf Default supplied by DevMpf. |
DevMpf calls this when it receives an OutOfBand message. For
example if server receives a send (no reply) message that can not be
processed, the server can send an OutOfBand message to the client.
The default method just prints a message and deletes the message it received. |
convert | Called by DevMpf, Default supplied by DevMpf | This is the DSET convert routine for ai, ao type records. The default version does nothing. Device support derived from base can provide it's own version. |
bind | Called immediately after any object derived from class DevMpf is created. | This binds to the MPF server. Since binding may result in virtual class methods being called, binding MUST be done after construction. Thus it is not possible to bind in the constructor for DevMpf. |
connectWait | Called by derived class | Wait for connection to server. It can be called by the constructor of the derived class or by a private thread belonging to the derived class. Do NOT call this as part of record processing. |
send(Message*) | Equivalent to send(message,replyTypeNone) | |
sendReply | Equivalent to send(message,replyTypeCompleteIO) | |
send(Message*,replyType) | Called by derived class | Send a message to the server. SetClientType is called to specify
the replyType. If the message is successfully sent and replyType is
replyTypeCompleteIO pact is set true.
NOTE: Only one send(message,replyTypeCompleteIO) should be outstanding, i.e. after calling send(message,replyTypeCompleteIO) wait for completeIO to be called before issuing another send. |
sendWait | Called by derived class | Send a message and wait for the reply. It can be called by the constructor of the derived class or by a private thread belonging to the derived class. Do NOT call this as part of record processing. |
getUserParm | Called by derived class | Get the portion of the parm of the INP or OUT link that follows the server name. |
getStatus | Called by derived class | Get the status. A 0 value is success. Any other value is device dependent. |
read_write | Called by record support | This is the read or write DSET routine. Handled automatically by DevMpf |
ioint | Called by record support | This is DSET get_io_intr routine. Handled automatically by DevMpf |
linconv | Called by record support | This is DSET linr_conv routine. It calls convert. |
The INP or OUT field MUST have the format:
field(INP or OUT, "#C<location> S<signal> @<server>,<deviceSpecific>")
parameter | meaning |
location | Location of the server. Must be integer which determines which message router handles messages. |
signal | For optional use by device support derived from DevMpf |
server | Name of the server which receives messages. |
deviceSpecific | For optional use by device support derived from DevMpf. Either a blank or comma can separate the server name from the device specific information. |
The first is used for record types that support linear conversions, e.g. ai and ao, and the second for other record types.MAKE_LINCONV_DSET(<dset name>,<dev init>) MAKE_DSET(<dset name>,<dev init>)
<top>/ configure/ RELEASE mpfApp/ coreSrc/ messageSrc/ registerSrc/ testSrc/ utilSrc/ epicsApp/ devMpf.h devMpf.cpp testApp/ adl/ Db/ src/ iocBoot/ iocepicsclient/ iocepicslocal/ iocepicsserver/ iocmpfclient/ iocmpflocal iocmpfserver/ iocmpflocal/ ioctcp
configure | This is almost exactly like <top>configure described in the 3.14 version of makeBaseApp |
RELEASE | EPICS_BASE must be defined correctly. |
utilSrc | DLList, DataFreeList, FreeList, Reboot and WatchDog. |
coreSrc | Contains the MPF code: Message, Routers, Tcp support. |
messageSrc | Contains the code to build specific message types. If new messages type are added this is the place. If it proves necessary to support an endless number of message types a better way of building new message types must be developed. Builds mpfMessageLib. |
registerSrc | Builds <top>/bin/<arch>/mpfRegister.o, which registers the mpf configuration commands. |
testSrc | Contains a test for watchDog |
ipSrc | Contains the code that interfaces to ipac (vxWorks only) |
epicsApp | Contains DevMpf.h and DevMpf.cpp. This is epics device support for mpf. |
testApp/Db and adl | Db contains the epics databases for the examples in iocepicsxxx. adl contains the example medm screen. |
testApp/src | Contains tcpTest, mpfTest, and DevMpfInt32Test. The Makfile is a good place to see how to build mpf applications. |
iocepicsclient iocepicsserver |
Epics (client) side and server side of example described previously. |
iocepicslocal | The example where client and server run on the same processor. |
iocmpfclient iocmpfserver |
Tests for remote MPF communication |
iocmpflocal | Test for local MPF communication |
ioctcp |
Test for TCP communication. |
MV167
MV162boot device : ei processor number : 0 host name : <host> file name : <full path>/vxWorks inet on ethernet (e) : <IPmv167>:<subnet mask> inet on back plane (b): <IPgateway> host inet (h) : <IPhost> gateway inet (g) : user (u) : <user> ftp password (pw) (blank = use rsh): <password> flags (f) : 0x0 target name (tn) : ioc167 startup script (s) : <full path name>/st.cmd other (o) :
boot device : sm=0x80000600 processor number : 1 host name : <host> file name : <full path>/vxWorks inet on ethernet (e) : inet on back plane (b): <IPmv162> host inet (h) : <IPhost> gateway inet (g) : <IPgateway> user (u) : <user> ftp password (pw) (blank = use rsh): <password> flags (f) : 0x0 target name (tn) : ioc162 startup script (s) : <full path name>/st.cmd
FreeList provides the same functionality as the freeList facility provided with epics base, in fact it is just freeList redone in C++.
DLList is a double linked list class that does not require nodes to be embedded in objects placed in a list. This is different than the ellList facility provided by epics base.
WatchDog is a class that provides functionality similar to the vxWorks wdLib. The major exception is that the user supplied callback is called by a WatchDog supplied task rather than being called at interrupt level.
DataFreeList is a class for data free lists. It provides free lists of sizes ranging from 16 bytes to 4096 bytes. It is used by the array message types.
Reboot can be used to turn off interrupts when a soft reboot is being performed.
These classes are not described in this document. If you want to use them in new code look at examples in existing code.
Copyright (c) 2002 University of Chicago. All rights reserved. MPF - Message Passing Facility MPF is distributed subject to the following license conditions: SOFTWARE LICENSE AGREEMENT Software: MPF 1. The "Software", below, refers to MPF (in either source code, or binary form and accompanying documentation). Each licensee is addressed as "you" or "Licensee." 2. The copyright holders shown above and their third-party licensors hereby grant Licensee a royalty-free nonexclusive license, subject to the limitations stated herein and U.S. Government license rights. 3. You may modify and make a copy or copies of the Software for use within your organization, if you meet the following conditions: a. Copies in source code must include the copyright notice and this Software License Agreement. b. Copies in binary form must include the copyright notice and this Software License Agreement in the documentation and/or other materials provided with the copy. 4. You may modify a copy or copies of the Software or any portion of it, thus forming a work based on the Software, and distribute copies of such work outside your organization, if you meet all of the following conditions: a. Copies in source code must include the copyright notice and this Software License Agreement; b. Copies in binary form must include the copyright notice and this Software License Agreement in the documentation and/or other materials provided with the copy; c. Modified copies and works based on the Software must carry prominent notices stating that you changed specified portions of the Software. 5. Portions of the Software resulted from work developed under a U.S. Government contract and are subject to the following license: the Government is granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable worldwide license in this computer software to reproduce, prepare derivative works, and perform publicly and display publicly. 6. WARRANTY DISCLAIMER. THE SOFTWARE IS SUPPLIED "AS IS" WITHOUT WARRANTY OF ANY KIND. THE COPYRIGHT HOLDERS, THEIR THIRD PARTY LICENSORS, THE UNITED STATES, THE UNITED STATES DEPARTMENT OF ENERGY, AND THEIR EMPLOYEES: (1) DISCLAIM ANY WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT, (2) DO NOT ASSUME ANY LEGAL LIABILITY OR RESPONSIBILITY FOR THE ACCURACY, COMPLETENESS, OR USEFULNESS OF THE SOFTWARE, (3) DO NOT REPRESENT THAT USE OF THE SOFTWARE WOULD NOT INFRINGE PRIVATELY OWNED RIGHTS, (4) DO NOT WARRANT THAT THE SOFTWARE WILL FUNCTION UNINTERRUPTED, THAT IT IS ERROR-FREE OR THAT ANY ERRORS WILL BE CORRECTED. 7. LIMITATION OF LIABILITY. IN NO EVENT WILL THE COPYRIGHT HOLDERS, THEIR THIRD PARTY LICENSORS, THE UNITED STATES, THE UNITED STATES DEPARTMENT OF ENERGY, OR THEIR EMPLOYEES: BE LIABLE FOR ANY INDIRECT, INCIDENTAL, CONSEQUENTIAL, SPECIAL OR PUNITIVE DAMAGES OF ANY KIND OR NATURE, INCLUDING BUT NOT LIMITED TO LOSS OF PROFITS OR LOSS OF DATA, FOR ANY REASON WHATSOEVER, WHETHER SUCH LIABILITY IS ASSERTED ON THE BASIS OF CONTRACT, TORT (INCLUDING NEGLIGENCE OR STRICT LIABILITY), OR OTHERWISE, EVEN IF ANY OF SAID PARTIES HAS BEEN WARNED OF THE POSSIBILITY OF SUCH LOSS OR DAMAGES.