Experimental Physics and
Industrial Control System

"D'Ewart, J. Mitch" <[email protected]> · Fri, 7 Mar 2014 09:57:18 -0800

The problem we are seeing

omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M

This seems to be a request and response not matching up.  In this case the response to getAxesArray() should be numerical, for example

"AM RP?"
responds with
"0,0,0,0,0,0,0"
but is getting
"MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN"

In this case sendOnly() is called from sendReceive().  sendOnly() flushes the response queue 

    /* see if junk at input port - should not be any data available */
    if ((epicsUInt16) pmotor->inGetIndex != (epicsUInt16) pmotor->inPutIndex)
    {
        // flush cards response Buffer
#ifdef DEBUG
        int deltaIndex = ((epicsUInt16)pmotor->inPutIndex) - ((epicsUInt16)pmotor->inGetIndex);
        Debug(32, "%s:%s:%s: flushing %d characters\n",
                driverName, functionName, portName, (((deltaIndex < 0) ? BUFFER_SIZE +
                        deltaIndex : deltaIndex)));
#endif
        putIndex = (epicsUInt16) pmotor->inPutIndex;
        pmotor->inGetIndex = putIndex;
        pmotor->status1_flag.Bits.text_response = 0;
        flag1.All = pmotor->status1_flag.All;
        pmotor->status1_flag.All = flag1.All;
        pmotor->msg_semaphore=0;

    }

 and then sends the new request.  sendReceive() then waits for a response flag to be set or timeout.  

    Debug(64, "omsMAXv::sendReceive: receiving\n");
    itera = 0;
    double time = 0.0;
    double timeout = epicsThreadSleepQuantum() + 0.001;
    while ((pmotor->status1_flag.Bits.text_response == 0) && (time < timeout)){
        Debug(32, "%s:%s:%s: Waiting for reponse, itera:%d\n",
                driverName, functionName, portName, itera);
        //  busy-waiting but not more than 2000 times
        if (itera > 2000){
            time += epicsThreadSleepQuantum();
            epicsThreadSleep(epicsThreadSleepQuantum());
        }
        itera++;
    }

It looks like the timeout may not be long enough so our messages are getting out of sync.  The two proposed solutions are to 1. increase the timeout  time and 2. flush multiple responses in sendOnly().

Mitch D'Ewart

________________________________________
From: [email protected] [[email protected]] On Behalf Of Mark Rivers [[email protected]]
Sent: Thursday, March 06, 2014 3:33 PM
To: Luchini, Kristi L.; [email protected]
Subject: RE: MAXv model-3 -  error reading axis position after 5 attempts

When sendOnly() is called does the MaxV actually send a response to that message?  If so then sendOnly should wait for it, even if it will discard the response.  If it does not then the next time a thread does a write/read it cannot reliably know if a response it got was from its query or a previous sendOnly response.

Mark

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Luchini, Kristi L.
Sent: Thursday, March 06, 2014 5:29 PM
To: [email protected]
Subject: MAXv model-3 - error reading axis position after 5 attempts

Hello,

Has anyone seen this error on the new model-3 version of the MAXv driver (omsAsyn) ?

   8544 omsBaseDriver:BasePoller:MAXv-01: error reading axis position after 5 attempts^M
   8545 omsBaseDriver:BasePoller:MAXv-01: error reading axis position after 5 attempts^M
   8546 omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M
   8547 2014/02/05 12:03:30.540 omsBaseDriver:readInt32:MAXv-01 wrong AUX state N^M
   8548 2014/02/05 12:03:30.670omsBaseDriver:getAxesArray:MAXv-01: array string conversion error, count: 0, axes: 8, input: >MNNN,PNNN,MNNN,PNLN,PNLN,PNLN,PNLN,PNLN<^M

We have seen this error in  a few motion control applications with MAXv controllers.  All of the IOCs were
 running motor-R6-8 asyn MAXv driver.  The problem seems to occur when we're using a number of axis, and in
some cases using virtual motors where multiple axis are moved at once.

1) MAXv  controller, with 2 steppers and 1 servo motors. This  MAXv has FW 1.45
2) MAXv controller , with 8 steppers. This MAXv has FW 1.44 (vitual motors moving 4-axis at once, and 2-axis at once using transform record)
3) MAXv controller, with 7 steppers.  This  MAXv has FW 1.31  (virtual motors, moves 4 axis at once, and 2-axis using transform record)

Mitch D'Ewart found that  "The omsMAXv::sendOnly function checks the input buffer and flushes one response if there is anything there but we seem to have come on a case where there are multiple responses backed up.  These responses are controlled by a message semaphore.  It seems that once sendOnly flushes the input buffer and clears the semaphore then the MAXv sends another backed up message response.  Finally the receive part of the calling function gets the wrong message.

There are two solutions that have been implemented recently at SLAC. Both will be booted on Monday 3/10/2014, in production.
So we don't know if the problem is fixed yet.
1) Ziga added eieio in omsMAXv.cpp by including basicIoOps.h from Till Strauman.

Example:
static void writeReg8(volatile epicsUInt8 *a8, epicsUInt8 value)
{
#ifdef HAS_IOOPS_H
    out_8(a8, value);
#else
    *a8 = value;
#endif
}

2) Mitch, changed the function sendOnly() to flush multiple responses.

Details:
Motor-R6-8
Asyn-R4-12
EPICS R3.14.12
RTEMS-4.9.4
MVME6100 CPU.
MAXV-8000

QUESTION:
How shold asyn support implementation  respond in this sort of case, where the response messages get out of sync with the queries?
Do you simply clear you the response buffer, buffer and start again, or is there another solution?  Has anyone else implemented a fix to this
Particular MAxv but or to another asyn support driver?

Regards,
  Kristi

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System