Experimental Physics and
Industrial Control System

"D. Peter Siddons" <[email protected]> · Thu, 30 Mar 2006 09:39:36 -0500

Ken,

Are you saying that it is a hardware problem with the Joerger device, or
a generic problem with the implementation of the scaler record?

Pete.

Kenneth Evans, Jr. wrote:

All,
     First I apologize for not giving this my full attention.  I have been
out of town, the Gateway is no longer my primary responsibility, and there
are only so many hours in the day.
     On looking at the original message I can guess that you are using a
Joerger scaler.  The Joerger scaler does not play well with channel access.
It does not work well with MEDM, either on Solaris or Linux.  I investigated
this some time ago, to the extent I was convinced it was not MEDM.  The beam
line people that were using it at that time did not have enough expertise to
determine the real cause of the problem.  The same problem has appeared at
various sites since.
     Puts to the Gateway come from the Gateway's clients (EDM in this case).
Straight puts go through, puts with callback are queued.  (I have not looked
at the code again, as Jeff stated.)  I would expect EDM is doing a put with
callback.  I tested yesterday and had no trouble putting enum values such as
SCAN rates through the Gateway.  Moreover, this is done here on a regular,
daily basis without problems, and has been tested extensively in the past.
     My recollection is that the workaround for the Joerger scaler is to
push the buttons alternately to get it to be done.  You can also have two
screens and use the buttons on both of them.  You'll have to experiment.
     Thus I think the problem is with the instrument rather than the
Gateway.  You will probably have problems without a Gateway.  Having said
this, it should not be giving Virtual circuit unresponsive.  VCR's have been
an ongoing problems with CA since they were introduced.  (On the positive
side, they solve some significant problems.)  I made a fix to some of the
VCR problems, which I tested.  Jeff fixed it in another way for 3.14.8,
which I have not tested, but I have no evidence here that Jeff's fix is not
working, and I would expect it to be.  Note that it is the client that
decides it has a VCR.  In many cases there was nothing wrong with the server
(Gateway in this case).
 -Ken
-----Original Message-----

From: Jeff Hill [mailto:[email protected]]
Sent: Wednesday, March 29, 2006 11:36 AM

To: 'Kenneth Evans, Jr.'; 'Dirk Zimoch'; 'Leicester, PJ (Pete)'

Cc: 'Tim Mooney'; 'Ralph Lange (BESSY)'; [email protected]

Subject: RE: gateway enum writes

Ok, so where are we. 
I think I understand that:
1) Strange behavior occurs only when the GW is attached to a PV that takes a
long time to complete a put callback.
2) Ken looked in the GW source code and verified that a put callback request
will postponed, when a put callback for that PV is already in progress,
until that put callback in progress completes.
Ken, perhaps you attempt to reproduce this issue against a record of this
type (with a very long put callback delay) at the APS?
Jeff
-----Original Message-----
From: Kenneth Evans, Jr. [mailto:[email protected]]
Sent: Wednesday, March 29, 2006 9:04 AM
To: 'Jeff Hill'; 'Dirk Zimoch'; 'Leicester, PJ (Pete)'
Cc: 'Tim Mooney'; [email protected]
Subject: RE: gateway enum writes
One could speculate that the original GW design might have intended to
perform all puts with completion notification in order to know exactly

when

to update the GW's cache.
One could further speculate that a solution, for the GW, might be to not
start a write for a channel if a write is already outstanding on that

same

channel. Instead the GW might save the value from the last write request

for

a busy channel and initiate another write request using this saved value

as

soon as the outstanding write request completes.

 That is what it does.
 -Ken
-----Original Message-----
From: Jeff Hill [mailto:[email protected]]
Sent: Wednesday, March 29, 2006 9:57 AM
To: 'Dirk Zimoch'; 'Leicester, PJ (Pete)'
Cc: 'Tim Mooney'; [email protected]; 'Ken Evans'
Subject: RE: gateway enum writes

Here my two centi-euro's worth.
if (ctx.getMsg()->m_cmmd == CA_PROTO_WRITE_NOTIFY) {
   docallback == GATE_DOCALLBACK;
} else {
   docallback == GATE_NOCALLBACK;
}

The context (in this case the variable is named ctx) is a class whose
definition is private to the portable server and subject to change in the
future. Its header file is typically not installed for public consumption.
We would not want, for example, to have the gateway code become too
dependent on the internal details of the server or its protocol. That
might
create maintenance headaches.
One could speculate that the original GW design might have intended to
perform all puts with completion notification in order to know exactly
when
to update the GW's cache.
Nevertheless, use of put callback just might cause behavior problems for
the
gateway if the destination record takes a long time to complete the write
request, and a 2nd write is started while the GW has a write to the same
record in progress. That would block the incoming protocol stream from the
GW to the IOC (for all channels on that IOC). The CA server is unable to
initiate another put notify request with this record until it completes
the
put notify request that is pending, and in that situation the current
server
design suspends processing of incoming requests until it can initiate the
put notify request (when the record completes processing).
One could further speculate that a solution, for the GW, might be to not
start a write for a channel if a write is already outstanding on that same
channel. Instead the GW might save the value from the last write request
for
a busy channel and initiate another write request using this saved value
as
soon as the outstanding write request completes.
Jeff
-----Original Message-----
From: Dirk Zimoch [mailto:[email protected]]
Sent: Wednesday, March 29, 2006 4:41 AM
To: Leicester, PJ (Pete)
Cc: Tim Mooney; [email protected]; Ken Evans
Subject: Re: gateway enum writes
This is from the gateway code (gateVc.cc):
caStatus gateVcData::write(const casCtx& ctx, const gdd& dd, gateChan
&/*chan*/)
{
	int docallback=GATE_DOCALLBACK;
[...]
	switch(at) {
[...]
	case gddAppType_ackt:
	case gddAppType_acks:
		docallback = GATE_NOCALLBACK;
		// Fall through
	default:
[...]
			caStatus stat = pv->put(&dd, docallback);
			if(stat != S_casApp_success) return stat;
			if(docallback) {
				// Start a pending write
#if DEBUG_GDD
				fflush(stderr);
				printf("pending_write\n");
				fflush(stdout);
#endif
				pending_write = new

gatePendingWrite(*this,ctx,dd);

				if(!pending_write) return S_casApp_noMemory;
				else return S_casApp_asyncCompletion;
			} else {
				return S_casApp_success;
			}
It seems that ALL puts except alarm handler ackt and acks are done with
callback. There is no direct flag to write() from the generic CA server
which indicates writes with or without callback, as far as I can see.
But the gateway should have look at the CA command in ctx, probably like
this:
if (ctx.getMsg()->m_cmmd == CA_PROTO_WRITE_NOTIFY) {
   docallback == GATE_DOCALLBACK;
} else {
   docallback == GATE_NOCALLBACK;
}

Dirk

Leicester, PJ (Pete) wrote:

I can confirm that the problem appears isolated to the scaler record

CNT

field and not enums in general.
It looks like Tim's ca_put_callback theory may be correct. Is there a
Gateway expert out there who can confirm this? How are writes done

from

the Gateway. Does it always use ca_put_callback regardless of how the
put reached the gateway?
Thanks for everyones responses on this.
Pete Leicester
Diamond
-----Original Message-----
From: Tim Mooney [mailto:[email protected]]
Sent: 28 March 2006 21:07
To: Leicester, PJ (Pete)
Cc: [email protected]
Subject: Re: gateway enum writes

Leicester, PJ (Pete) wrote:
I am getting some strange behaviour when writing enumerations through
the Gateway (version 2.0.0.0 on 3.14.8.2 and RedHat Enterprise 4).
The problem first showed itself in edm when pressing a button on a edm
screen to send the value resulted in the following error:
   CA.Client.Exception...............................................
       Warning: "Virtual circuit unresponsive"
       Context: "diamrs0005l.diamond.ac.uk:6064"
       Source File: ../tcpiiu.cpp line 896
       Current Time: Tue Mar 28 2006 16:19:01.334736000
   ..................................................................
I did some further tests using caput with similar results:
   [pjl45@pc0005 pjl45]$ caput  -w15 GDA:scaler2.CNT Count
   Old : GDA:scaler2.CNT                Done
   Read operation timed out: PV data was not read.
   New : GDA:scaler2.CNT
   CA.Client.Exception...............................................
       Warning: "Virtual circuit disconnect"
       Context: "op=0, channel=GDA:scaler2.CNT, type=DBR_TIME_STRING,
count=1, ctx="diamrs0005l.diamond.ac.uk:6064""
       Source File: ../getCopy.cpp line 82
       Current Time: Tue Mar 28 2006 17:04:24.488496000
.
Despite the above error message the write does actually reach the IOC.
However if I now try the change the value back as follows:
   [pjl45@pc0005 pjl45]$ caput  -w15 GDA:scaler2.CNT Done
   Old : GDA:scaler2.CNT                Count
   Read operation timed out: PV data was not read.
   New : GDA:scaler2.CNT
   CA.Client.Exception...............................................
       Warning: "Virtual circuit disconnect"
       Context: "op=0, channel=GDA:scaler2.CNT, type=DBR_TIME_STRING,
count=1, ctx="diamrs0005l.diamond.ac.uk:6064""
       Source File: ../getCopy.cpp line 82
       Current Time: Tue Mar 28 2006 17:22:36.167980000
   ..................................................................
I get an error again however this time the 'Done' value gets written
to
the IOC exactly ONE MINUTE after I entered the caput command. This is
long after the caput command has timed out so it appears the gateway

is
responsible for the delay?
Has anyone any idea what may be happening? Is there a 60 second
timeout
in CA or the gateway which may give a clue as to what I am seeing?
(For the record this test was done with a very lightly loaded test
gateway serving only 20 or so PV's. Also using caput -w70 results in

the
same timeouts)

Do you get this kind of result with *all* enum fields that you write

to

through the gateway, or only this one? The scaler record's CNT field

is

different from other enum's in that it starts an operation that may

take

a long time to complete.  I don't know what the gateway is using to do
put's, but if it should happen to be ca_put_callback(), the gateway

may

be waiting for a callback that will come only after the scaler has
finished counting.
--
Dr. Dirk Zimoch
Swiss Light Source
Paul Scherrer Institut
Computing and Controls
phone +41 56 310 5182
fax   +41 56 310 4413

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System