Experimental Physics and
Industrial Control System

From: "J. Lewis Muir" <[email protected]>
To: "EPICS Tech Talk" <[email protected]>
Sent: Wednesday, May 29, 2013 5:10:57 PM
Subject: autosave failure S_rpcLib_RPC_CANTSEND

I have a problem where, after a number of days of uptime, I start
getting the following errors on my IOC console:

=== console ===
save_restore:write_it: fprintf returned -1. [130528-120034]
../save_restore.c(1402): [0x2f0003]=write_it:S_rpcLib_RPC_CANTSEND
save_restore:write_it: fclose('auto_settings.sav') returned -1
../save_restore.c(1518): [0x2f0003]=write_it:S_rpcLib_RPC_CANTSEND
save_restore:write_it: Giving up on this attempt to write
'auto_settings.sav'. [130528-120035]
===============

Does anyone know what's wrong and how to fix this?  Or what would be a
good way to investigate and fix this?

The error looks like it's coming from the autosave module.  I'm guessing
maybe the RPC error suggests something wrong with communicating with the
NFS server where the auto_settings.sav file is being written, but I
don't know.  The NFS server log doesn't show any problem.  Once the
problem starts, it doesn't seem to go away until I reboot the IOC.

This problem has not always existed; it used to work fine.  I initially
thought it was a problem with autosave 5-0, since it seemed the problem
showed up for the first time after I had upgraded to 5-0.  But then I
rolled back to 4-8, and the problem is still appearing.  It's possible
the NFS server changed somehow, or maybe a network adapter, cable,
switch, etc. is failing, or none of these.

The IOC is running EPICS Base 3.14.12.2 and autosave 4-8.  The NFS
server is running RHEL 5.9.

I have another VME IOC running the same version of EPICS Base and
autosave and using the same NFS server, and it does not have this
problem.  This makes me lean in the direction of suspecting something is
wrong with the IOC or its network connectivity to the NFS server.

The VME IOC that does not have a problem has an auto_settings.sav file
that is 9120 lines long (244110 bytes), while the VME IOC that does have
the problem has an auto_settings.sav file that is 10448 lines long
(270096 bytes).  Maybe this is irrelevant.

More IOC environment details:
* VME processor module: Motorola MVME-5100 (vxWorks-ppc604_long)
  - CPU: MPC 750
  - Memory: 512 MB
* OS: VxWorks 5.5.2
* Kernel: WIND 2.6
* VxWorks image: mv5100-asd6 (from APS)
* Build toolchain: Tornado 2.2.2
* EPICS 3.14.12.2 (+ Known Problems patches)
* synApps 5.6 (includes autosave 4-8; + various module upgrades)

Thanks,

Lewis