EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Roper Quad-RO locks up with areaDetector/ccd apps
From: "Mark Rivers" <[email protected]>
To: "Ru Igarashi" <[email protected]>, <[email protected]>
Date: Sat, 8 Aug 2009 08:37:06 -0500
Ru,
 
> We essentially run a loop triggering the
> detector via the ccd/areaDetector's acquire PV, wait
> for a monitor from the acquire status, make sure the
> status is "Done", do a few other tasks, then trigger
> again.  There's at least 1.5 seconds between the end
> of the trigger "Done" change-of-state and the subsequent
> software trigger.
 
Just to be clear:  you should wait for the $(P)$(R)Acquire to go back to to 0 ("Done"), not the $(P)$(R)DetectorState_RBV PV.  But I think this is what you are doing.
 
I have not seen the problem you are seeing, but I will do a test next week to be sure, because I am not sure I have tested >1000 exposures in a loop.
 
The cameras I can test are the MicroMAX 5MHz, which uses the older Princeton Instruments PCI card, and the CoolSnap HQ2 which uses the Photometrics PCI card.  I don't have any Princeton Instruments Firewire cameras I can test.  
 
It could be a problem that is specific to the Firewire interface.  One thing to look at is the Windows Task Manager/Performance screen.  Look to see if the number of Handles or Commit Charge Total (virtual memory) are increasing monotonically as the application runs.  If so there is a resource leak.
 
Note that ccd and areaDetector use a completely different way of talking to the Microsoft COM interface which is used to control WinView.  ccd uses a small Visual Basic socket server that I wrote, and the COM calls are done from Visual Basic, which Princeton Instruments documents and "supports".  areaDetector calls COM directly from C++, which is not documented by Princeton, but works fine. 
 
What you need to do is figure out if this is a problem in the Princeton Instruments software or my ccd or areaDetector software.  Because ccd and areaDetector are so different I doubt if they both have the same bug, but it's always possible.  In order to see if the problem is in the Princeton software what you need to do is run a Visual Basic Script (.vbs file, not Visual Basic itself) that does the same loop that EPICS is doing and see if you can reproduce the problem.  That is what Princeton did to locate the resource leak in XP SP2 that was causing WinView to die after ~1600 images.  I am appending the VBS script that Princeton sent me that reproduced that problem.  It seems to me that this should work for you, just name this file test.vbs and double-click on it in Windows Explorer.  If you can get it to fail with this VBS script then you need to send it to Princeton, because they support VBS and will recognize it as a valid test.
 
Mark
 
test.vbs
**********
 
Option Explicit

Dim ExpVB: Set ExpVB = CreateObject("WinX32.ExpSetup")
Dim wins: set wins = CreateObject("WinX32.DocWindows")
dim win
dim doc
Dim I
dim numExps
dim myCheck
dim frame


wins.closeall

numExps = inputbox("enter number of experiments to run")
myCheck = isnumeric( numExps ) 
if myCheck then 
    ExpVB.SetParam 827, false

    For I = 1 to numExps
        set doc = ExpVB.Start2
        ExpVB.WaitForExperiment
        if i=1 then
             'create new display window
            dim docDisplay: set docDisplay = CreateObject("WinX32.DocFile")
            dim info: set info = CreateObject("WinX32.DocInfo")
            info.Name = "Display"
            info.dataType = doc.getParam( 9 )
            info.x = doc.getParam( 6 )
            info.y = doc.getParam( 7 )
            info.z = doc.getParam( 24 )
            info.bShowWindow = true
            info.bAppend = false
            info.filetype = 1
            docDisplay.OpenNew "", info
       end if
       doc.GetFrame 1, frame 
       doc.save
       doc.close
 
       docDisplay.PutFrame 1, frame
       docDisplay.Update
 
    Next

    ExpVB.SetParam 827, true

else
    msgbox "Please enter a valid number"
end 
 
 
________________________________

From: [email protected] on behalf of Ru Igarashi
Sent: Fri 8/7/2009 7:55 PM
To: [email protected]
Subject: Roper Quad-RO locks up with areaDetector/ccd apps



One of the beamlines at the Canadian Light Source (CLS)
uses a Princeton Instruments (Roper) Quad-RO CCD detector
(2084x2084). We have used the synApps 'ccd' application
in the past, and have recently attempted to use the
'areaDetector' app instead.  In both cases, however,
when we run multiple image sequential scans, the hardware
locks up.  We essentially run a loop triggering the
detector via the ccd/areaDetector's acquire PV, wait
for a monitor from the acquire status, make sure the
status is "Done", do a few other tasks, then trigger
again.  There's at least 1.5 seconds between the end
of the trigger "Done" change-of-state and the subsequent
software trigger. The sequence is typically set for
2500-4000 exposures.

With 'ccd' we could get about 2500-3000 exposures before
the hardware locked up.  At least we think it is the
hardware because WinView still operates (e.g. close image,
start acquire, empty image frame appears, but no image),
restarting WinView does not solve the problem, and only
a hardware power cycle (long wait) solves the problem.
With 'ccd', there was a proxy app that displayed comm
traffic, and at the onset of the problem, it was clear
that communications were out of sync (command acks too
early, missing data packets) and remained out of sync.

We were hoping a switch to 'areaDetector' would resolve
this issue but it seems to have actually made the situation
worse: we now only get at most a few hundred exposures
before locking up.  Adding more time between exposures
(in case there was some sort of interference with
communication before or after an exposure) did not help,
either.

It looks as though areaDetector holds the status flag
"busy" until after the data is written to file, and we
don't think we see an overlap with disk or network activity
(haven't thought through caching issues properly), so it
doesn't seem to be a problem with triggering during the
previous image's file I/O.

The CCD hardware is linked to a Windows XP box via firewire.
XP is Service Pack 3 (so it isn't the old SP 2 bug).
areaDetector is V1.4 (prebuilt), ccd was V1.6.  WinView
is V2.5.22.0.

Has anyone else experienced the same kind of problem
with their Roper CCD detector?  What can we do to either
further diagnose the problem or resolve it?  Perhaps
a Windows app that that can replace and perform the same
scan operations a few thousand times?

ru

--
Ru Igarashi                   Software & Instrumentation Specialist
e-mail: [email protected]  Canadian Light Source
ph: 306 657 3751              University of Saskatchewan
fax: 306 657 3535             Saskatoon SK S7N 0X4





Replies:
RE: Roper Quad-RO locks up with areaDetector/ccd apps Mark Rivers
References:
Roper Quad-RO locks up with areaDetector/ccd apps Ru Igarashi

Navigate by Date:
Prev: Roper Quad-RO locks up with areaDetector/ccd apps Ru Igarashi
Next: RE: Roper Quad-RO locks up with areaDetector/ccd apps Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Roper Quad-RO locks up with areaDetector/ccd apps Ru Igarashi
Next: RE: Roper Quad-RO locks up with areaDetector/ccd apps Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 31 Jan 2014 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·