Experimental Physics and
Industrial Control System

"Mark Rivers" <[email protected]> · Sun, 19 Sep 2010 19:39:43 -0500

Hi All,

I just did a test on my laptop (Windows XP, T9500 Dual Core, 2.6 GHz, 3.5 GB RAM).

I set area Detector R1-6 to use the simDetector, 2048x2048 4-bit images ( 4 MB/frame).

With no viewer active, and with the AcquireTime and AcquirePeriod both set to .001 simDetector is producing about 68 frames/sec, i.e. almost 400 MB/sec.  That uses about 70% of the CPU, where 100% means both CPUs are saturated.

I then started the ImageJ viewer.  simDetector dropped to producing about 35 frames/sec, because the CPU is now 100% busy, with simDetector using about 60% and javaw (ImageJ) using 40%.

ImageJ is displaying about 11 frames/sec.

But very significantly simDetector is only using 66 MB of virtual memory, and javaw is only using 60 MB of virtual memory.  javaw used 30MB of virtual memory before I started the areaDetector viewer display.

Thus, it is clear that it is possible to write a Java viewer that can display 4 MB frames at 11 frames/second without consuming a large amout of virtual memory, i.e. only an additional 30MB with the viewer running.

Mark

________________________________

From: Szalata, Zenon M. [mailto:[email protected]]
Sent: Sat 9/18/2010 1:50 PM
To: Malitsky, Nikolay D; Mark Rivers; Kasemir, Kay; Chen, Xihui
Cc: EPICS tech-talk
Subject: RE: o.c.swt.widgets: Improve intensity graph drawing performancebyusing pixel lookup table.

Hi All,
Just to clarify, the problems I encountered were not due to running out of CPU cycles.  It had to do with CSS/BOY entering a memory starvation state, from which it would only recover if the data flow was stopped for some length of time, like tens of seconds.  While in that state, the free memory would drop to below 50Mbytes (out of 500Mbytes of memory allocated to it) and a stream of error messages would be printed.  This would happen when image data of 4Mbytes was flowing at about 6 Hz.  I think, that it was triggered by a busting effect since I was running the camera in a "Free Running" mode and the rate as reported by areaDetector fluctuated between 3 and 6 Hz.  So, my issue is with the memory management either in java, eclipse, or CSS, I don't know which.

Thanks,
Zen

-----Original Message-----
From: Malitsky, Nikolay D [mailto:[email protected]]
Sent: Saturday, September 18, 2010 9:51 AM
To: Mark Rivers; Szalata, Zenon M.; Kasemir, Kay; Chen, Xihui
Cc: EPICS tech-talk; Malitsky, Nikolay D
Subject: RE: o.c.swt.widgets: Improve intensity graph drawing performancebyusing pixel lookup table.

I have also profiled several different versions of the plain Java-based approaches, such as BufferedImage, VolatileImage, etc.
According to the Netbeans profiler, drawing of  the 1024*768*3 frame takes 5ms and memory behavior is very similar to Kay's description.
A client-server demo was tested with simDetector from the areaDetector package. It run 15 Hz without problems.
These results are also consistent with the TINE/ACOP video system . 30 Hz (with the 1024*768*3 frame) however looks as a limit
for the present configuration. There are several ideas how to address it, but it should be driven by actual requirements.

Nikolay

-----Original Message-----
From: [email protected] on behalf of Mark Rivers
Sent: Sat 9/18/2010 10:39 AM
To: Szalata, Zenon M.; Kasemir, Kay; Chen, Xihui
Cc: EPICS tech-talk
Subject: RE: o.c.swt.widgets: Improve intensity graph drawing performancebyusing pixel lookup table.

I can some additional information that may be relevant.

I have written a plugin for the very popular ImageJ program that displays images in real-time from areaDetector cameras and detectors.

It also uses the native Java Channel Access implementation.  It can readily display 20 frames/second of 1360x1024 color video.  That is more than 4MB per frame.  It can probably run even faster on more powerful machines than I've been using.

It also preserves 8-bit images without expanding them to 16 bits, and correctly displays 8-bit images as unsigned.  I don't know how they do that, since Java does not natively support unsigned types, but they do, and their source code is readily available.

So it really looks like the problem is the BOY client, and not CAJ, or intrinsic Java problems.

Mark

________________________________

From: [email protected] on behalf of Szalata, Zenon M.
Sent: Fri 9/17/2010 11:57 PM
To: Kasemir, Kay; Chen, Xihui
Cc: EPICS tech-talk
Subject: RE: o.c.swt.widgets: Improve intensity graph drawing performance byusing pixel lookup table.

Hi Kay,
In spite of your elaborate argument, quite convincing at that, I am not convinced that there is no problem.  It may be that all is working as designed, but then it was designed poorly.  Whether it is java , eclipse, or CSS that is falling short, I can't tell.
I am not convinced that I should not expect to display a 4 MB image at 6 or 7 Hz rate on a machine with 3 GB of physical memory without generating errors.  This is actually a bit absurd.  Image size is not likely to get smaller.  It would be nice if a 4 MB image could stay 4 MB until it is painted.  On Windows XP, I can display these images at 30+ Hz using the software provided by the camera maker.
Anyway, presently the image rate is limited by the Ethernet adapter not being setup for jumbo frames.  These cameras will be used to monitor images along the beam line at 10 or 30 Hz when the hardware and software goes into production.  Presently I am testing these cameras in 8 bit Mono but they are capable of 16 bit color, so the volume of data will increase.  I understand that 8 bit or 16 bit is all the same for java.
I see that edm video widget outperforms CSS/BOY intensity graph and I hate to think that I should disqualify CSS/BOY due to poor performance.

I apologize for this harsh criticism but it irks me that a promising product like CSS/BOY may have to be put aside because it does not measure up to inferior products like EDM.

I am also disappointed that I have to wait a month or more for a simple fix to CSS/BOY, which is actually crucial for me to make progress, to trickle down to me.  In fact it has been more than a month since Xihui fixed a couple of bugs on my request, fixes which I still do not have.  I understand that there needs to be a CSS integrator here at SLAC.  Unfortunately, there seems to be little interest in CSS by the management personnel here...
I Would like if there was a way to get fixes to CSS/BOY delivered to customers outside of SNS in a day or two after the fix is made, so that we can test, help make it better, and enjoy using it.  Until such a mechanism is found using CSS/BOY is unproductive.  I know, I could get off my butt and start learning all there is to learn about eclipse and java and become my own integrator and stop complaining.  Unfortunately, I don't quite have time for this, so that is that.

Anyway, thanks for your help,
Zen

-----Original Message-----
From: Kasemir, Kay [mailto:[email protected]]
Sent: Thursday, September 16, 2010 9:25 AM
To: Szalata, Zenon M.; Chen, Xihui
Subject: Re: o.c.swt.widgets: Improve intensity graph drawing performance by using pixel lookup table.

Hi:

> I am using pure java.
That's good in the sense that this restricts the problem completely to the JVM, no worries about external code. All that's happening can be analyzed in the Java debugger or JProfiler.

Strictly speaking, I think there is no memory leak.
Meaning:
There is nothing that unnecessarily remembers each received value so that we eventually run out of memory.
Indeed, for update rates slow enough or images small enough everything is
fine:
> 1. Free memory bounces between 300M and 200M.  CSS/BOY is displaying
> image data which is 2073600 points.
> 2. Stopped the IOC.  Free memory goes up to about 400M and stays there...
So there's really no memory leaked.

But clearly memory is used while we receive values and update the image.
It's used temporarily, not leaked, all runs OK, but with a bigger image more memory is used to the point where you run out:
> changed the waveform size to 4177920 points ..
> Free memory drops to about 50 M and OutOfMemory errors are displayed.

Looking at the CPU view in JProfiler, the value chain has 3 key parts:

1) CAJ receives value from IOC.
We use the "DirectEventDispatcher", so the Channel Access client library is not queuing values internally except for maybe the low-level network code keeping received data in some buffers.
In the end, it calls into BOY code ending in
GUIRefreshThread.addIgnorableTask()

2) The GUIRefreshThread puts those tasks into the GUI 'main' loop.
This uses no CPU at all in the overall picture.

3) The 'main' loop redraws the image.

It could be that the GUIRefreshThread queue actually grows when the main loop is getting behind in redrawing the image.

GUIRefreshThread.addIgnorableTask()
is meant to ignore redraws that are already in the queue; don't redraw the same widget again if it's already scheduled for a redraw:

public synchronized void addIgnorableTask(final WidgetIgnorableUITask task) {
        if(tasksQueue.contains(task))
            tasksQueue.remove(task);
        tasksQueue.add(task);
}

In principle, that should work.
But it does mean that several copies of the data are kept in memory for a
while:

1) Redraw is scheduled to get from the old value V1 to a new value V2
2) Redraw is scheduled to get from that new value V3 to an
   even newer value V3.
addIgnorableTask() removes the 1st redraw for V1 -> V2, but until that's garbage collected we have 3 copies of the data in memory.

There might be a few more places where the array is copied.
One is in IntensityGraphEditPart where a change handler calls
 ((IntensityGraphFigure)figure).setDataArray(ValueUtil.getDoubleArray(value)
);

Again no memory is leaked, it's all cleaned up eventually, but when the image data gets too big, those temporaries must temporarily become too much.

---

At the upcoming EPICS meeting there's something about an "image server" and new network protocols on the agenda.
The EPICS practice of putting images into waveforms without any meta info regarding the image format is quite a hack anyway.
Maybe there'll be some news about better ways to handle images, for example including compression of the image data, and then we can optimize BOY to handle those.

Right now I think you're best off using smaller images, slower update rates or simply increasing your memory sizes (actual RAM and JVM maximum).

Thanks,
Kay

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System