Experimental Physics and
Industrial Control System

tieman <[email protected]> · Mon, 10 Aug 2009 08:34:40 -0500

Hi all,

I'm not sure if we're seeing the same problem or not, but it sounds

suspiciously familiar...

2IDD has a Quad-RO detector. They run theirs with an active firewire

extension cable which may/may not contribute to the problem. Recently

they started trying to run longer scans and have discovered that the

computer reboots after exactly 2496 frames. The reboot after 2496

frames has been very predictable. If we use WinX to test the detector,

it will run well past the 2496 frames. If we use my Image Server

application, the computer reboots. My software uses the PVCam libraries

directly--no WinView. We haven't tried WinView to test the detector.

Since we are moving away from the Image Server to areaDetector, the plan

was to try this camera with areaDetector during the upcomming shutdown

and to try and resolve the problem under areaDetector. We do have a

functional PVCam interface for areaDetector plus Mark's WinView COM

interface to work with.

Brian

Mark Rivers wrote:

Hi Ru,

I was just able to test areaDetector R1-5beta2 with a Princeton Instruments MicroMAX 5MHz, which uses the PI PCI card.

I used the EPICS "sscan" record to trigger the detector Acquire PV. I used a 0.1 second acquire time, and was collecting a 450x515 region of the detector. I was able to do a 2000 point scan twice with no problems, which thus saved 4000 images to disk. This was with NumExposures, NumImages, and NumAcquisitions all set to 1. There was 0 second detector settling time and positioner settling time in my sscan record, so it was immediately starting the next acquisition when the previous one completed.

So I think my conclusion is that there is not a fundamental problem with areaDetector or WinView. I suspect the problem is specific to your Firewire camera, unless you are doing something differently in your client.

Mark

________________________________

From: Mark Rivers
Sent: Sat 8/8/2009 8:37 AM
To: Ru Igarashi; [email protected]
Subject: RE: Roper Quad-RO locks up with areaDetector/ccd apps

Ru,

We essentially run a loop triggering the
detector via the ccd/areaDetector's acquire PV, wait
for a monitor from the acquire status, make sure the
status is "Done", do a few other tasks, then trigger
again.  There's at least 1.5 seconds between the end
of the trigger "Done" change-of-state and the subsequent
software trigger.

Just to be clear: you should wait for the $(P)$(R)Acquire to go back to to 0 ("Done"), not the $(P)$(R)DetectorState_RBV PV. But I think this is what you are doing.

I have not seen the problem you are seeing, but I will do a test next week to be sure, because I am not sure I have tested >1000 exposures in a loop.

The cameras I can test are the MicroMAX 5MHz, which uses the older Princeton Instruments PCI card, and the CoolSnap HQ2 which uses the Photometrics PCI card. I don't have any Princeton Instruments Firewire cameras I can test.

It could be a problem that is specific to the Firewire interface. One thing to look at is the Windows Task Manager/Performance screen. Look to see if the number of Handles or Commit Charge Total (virtual memory) are increasing monotonically as the application runs. If so there is a resource leak.

Note that ccd and areaDetector use a completely different way of talking to the Microsoft COM interface which is used to control WinView. ccd uses a small Visual Basic socket server that I wrote, and the COM calls are done from Visual Basic, which Princeton Instruments documents and "supports". areaDetector calls COM directly from C++, which is not documented by Princeton, but works fine.

What you need to do is figure out if this is a problem in the Princeton Instruments software or my ccd or areaDetector software. Because ccd and areaDetector are so different I doubt if they both have the same bug, but it's always possible. In order to see if the problem is in the Princeton software what you need to do is run a Visual Basic Script (.vbs file, not Visual Basic itself) that does the same loop that EPICS is doing and see if you can reproduce the problem. That is what Princeton did to locate the resource leak in XP SP2 that was causing WinView to die after ~1600 images. I am appending the VBS script that Princeton sent me that reproduced that problem. It seems to me that this should work for you, just name this file test.vbs and double-click on it in Windows Explorer. If you can get it to fail with this VBS script then you need to send it to Princeton, because they support VBS and will recognize it as a valid test.

Mark

test.vbs

**********

Option Explicit

Dim ExpVB: Set ExpVB = CreateObject("WinX32.ExpSetup")
Dim wins: set wins = CreateObject("WinX32.DocWindows")
dim win
dim doc
Dim I
dim numExps
dim myCheck
dim frame

wins.closeall

numExps = inputbox("enter number of experiments to run")

myCheck = isnumeric( numExps )

if myCheck then

 ExpVB.SetParam 827, false

    For I = 1 to numExps
        set doc = ExpVB.Start2
        ExpVB.WaitForExperiment
        if i=1 then
             'create new display window
            dim docDisplay: set docDisplay = CreateObject("WinX32.DocFile")
            dim info: set info = CreateObject("WinX32.DocInfo")
            info.Name = "Display"
            info.dataType = doc.getParam( 9 )
            info.x = doc.getParam( 6 )
            info.y = doc.getParam( 7 )
            info.z = doc.getParam( 24 )
            info.bShowWindow = true
            info.bAppend = false
            info.filetype = 1
            docDisplay.OpenNew "", info
       end if

 doc.GetFrame 1, frame

 doc.save

       doc.close

 docDisplay.PutFrame 1, frame

       docDisplay.Update

 Next

    ExpVB.SetParam 827, true

else
    msgbox "Please enter a valid number"

end

________________________________

From: [email protected] on behalf of Ru Igarashi
Sent: Fri 8/7/2009 7:55 PM
To: [email protected]
Subject: Roper Quad-RO locks up with areaDetector/ccd apps

One of the beamlines at the Canadian Light Source (CLS)
uses a Princeton Instruments (Roper) Quad-RO CCD detector
(2084x2084). We have used the synApps 'ccd' application
in the past, and have recently attempted to use the
'areaDetector' app instead.  In both cases, however,
when we run multiple image sequential scans, the hardware
locks up.  We essentially run a loop triggering the
detector via the ccd/areaDetector's acquire PV, wait
for a monitor from the acquire status, make sure the
status is "Done", do a few other tasks, then trigger
again.  There's at least 1.5 seconds between the end
of the trigger "Done" change-of-state and the subsequent
software trigger. The sequence is typically set for
2500-4000 exposures.

With 'ccd' we could get about 2500-3000 exposures before
the hardware locked up.  At least we think it is the
hardware because WinView still operates (e.g. close image,
start acquire, empty image frame appears, but no image),
restarting WinView does not solve the problem, and only
a hardware power cycle (long wait) solves the problem.
With 'ccd', there was a proxy app that displayed comm
traffic, and at the onset of the problem, it was clear
that communications were out of sync (command acks too
early, missing data packets) and remained out of sync.

We were hoping a switch to 'areaDetector' would resolve
this issue but it seems to have actually made the situation
worse: we now only get at most a few hundred exposures
before locking up.  Adding more time between exposures
(in case there was some sort of interference with
communication before or after an exposure) did not help,
either.

It looks as though areaDetector holds the status flag
"busy" until after the data is written to file, and we
don't think we see an overlap with disk or network activity
(haven't thought through caching issues properly), so it
doesn't seem to be a problem with triggering during the
previous image's file I/O.

The CCD hardware is linked to a Windows XP box via firewire.
XP is Service Pack 3 (so it isn't the old SP 2 bug).
areaDetector is V1.4 (prebuilt), ccd was V1.6.  WinView
is V2.5.22.0.

Has anyone else experienced the same kind of problem
with their Roper CCD detector?  What can we do to either
further diagnose the problem or resolve it?  Perhaps
a Windows app that that can replace and perform the same
scan operations a few thousand times?

ru

--
Ru Igarashi                   Software & Instrumentation Specialist
e-mail: [email protected]  Canadian Light Source
ph: 306 657 3751              University of Saskatchewan
fax: 306 657 3535             Saskatoon SK S7N 0X4

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System