Experimental Physics and
Industrial Control System

"Szalata, Zenon M." <[email protected]> · Fri, 17 Sep 2010 21:57:43 -0700

Hi Kay,
In spite of your elaborate argument, quite convincing at that, I am not convinced that there is no problem.  It may be that all is working as designed, but then it was designed poorly.  Whether it is java , eclipse, or CSS that is falling short, I can't tell.
I am not convinced that I should not expect to display a 4 MB image at 6 or 7 Hz rate on a machine with 3 GB of physical memory without generating errors.  This is actually a bit absurd.  Image size is not likely to get smaller.  It would be nice if a 4 MB image could stay 4 MB until it is painted.  On Windows XP, I can display these images at 30+ Hz using the software provided by the camera maker.
Anyway, presently the image rate is limited by the Ethernet adapter not being setup for jumbo frames.  These cameras will be used to monitor images along the beam line at 10 or 30 Hz when the hardware and software goes into production.  Presently I am testing these cameras in 8 bit Mono but they are capable of 16 bit color, so the volume of data will increase.  I understand that 8 bit or 16 bit is all the same for java.
I see that edm video widget outperforms CSS/BOY intensity graph and I hate to think that I should disqualify CSS/BOY due to poor performance.

I apologize for this harsh criticism but it irks me that a promising product like CSS/BOY may have to be put aside because it does not measure up to inferior products like EDM.

I am also disappointed that I have to wait a month or more for a simple fix to CSS/BOY, which is actually crucial for me to make progress, to trickle down to me.  In fact it has been more than a month since Xihui fixed a couple of bugs on my request, fixes which I still do not have.  I understand that there needs to be a CSS integrator here at SLAC.  Unfortunately, there seems to be little interest in CSS by the management personnel here...
I Would like if there was a way to get fixes to CSS/BOY delivered to customers outside of SNS in a day or two after the fix is made, so that we can test, help make it better, and enjoy using it.  Until such a mechanism is found using CSS/BOY is unproductive.  I know, I could get off my butt and start learning all there is to learn about eclipse and java and become my own integrator and stop complaining.  Unfortunately, I don't quite have time for this, so that is that.

Anyway, thanks for your help,
Zen

-----Original Message-----
From: Kasemir, Kay [mailto:[email protected]] 
Sent: Thursday, September 16, 2010 9:25 AM
To: Szalata, Zenon M.; Chen, Xihui
Subject: Re: o.c.swt.widgets: Improve intensity graph drawing performance by using pixel lookup table.

Hi:

> I am using pure java.
That's good in the sense that this restricts the problem completely to the JVM, no worries about external code. All that's happening can be analyzed in the Java debugger or JProfiler.

Strictly speaking, I think there is no memory leak.
Meaning:
There is nothing that unnecessarily remembers each received value so that we eventually run out of memory.
Indeed, for update rates slow enough or images small enough everything is
fine:
> 1. Free memory bounces between 300M and 200M.  CSS/BOY is displaying 
> image data which is 2073600 points.
> 2. Stopped the IOC.  Free memory goes up to about 400M and stays there...
So there's really no memory leaked.

But clearly memory is used while we receive values and update the image.
It's used temporarily, not leaked, all runs OK, but with a bigger image more memory is used to the point where you run out:
> changed the waveform size to 4177920 points ..
> Free memory drops to about 50 M and OutOfMemory errors are displayed.

Looking at the CPU view in JProfiler, the value chain has 3 key parts:

1) CAJ receives value from IOC.
We use the "DirectEventDispatcher", so the Channel Access client library is not queuing values internally except for maybe the low-level network code keeping received data in some buffers.
In the end, it calls into BOY code ending in
GUIRefreshThread.addIgnorableTask()

2) The GUIRefreshThread puts those tasks into the GUI 'main' loop.
This uses no CPU at all in the overall picture.

3) The 'main' loop redraws the image.

It could be that the GUIRefreshThread queue actually grows when the main loop is getting behind in redrawing the image.

GUIRefreshThread.addIgnorableTask()
is meant to ignore redraws that are already in the queue; don't redraw the same widget again if it's already scheduled for a redraw:

public synchronized void addIgnorableTask(final WidgetIgnorableUITask task) {
        if(tasksQueue.contains(task))
            tasksQueue.remove(task);
        tasksQueue.add(task);
}

In principle, that should work.
But it does mean that several copies of the data are kept in memory for a
while:

1) Redraw is scheduled to get from the old value V1 to a new value V2
2) Redraw is scheduled to get from that new value V3 to an
   even newer value V3.
addIgnorableTask() removes the 1st redraw for V1 -> V2, but until that's garbage collected we have 3 copies of the data in memory.

There might be a few more places where the array is copied.
One is in IntensityGraphEditPart where a change handler calls
 ((IntensityGraphFigure)figure).setDataArray(ValueUtil.getDoubleArray(value)
);

Again no memory is leaked, it's all cleaned up eventually, but when the image data gets too big, those temporaries must temporarily become too much.

---

At the upcoming EPICS meeting there's something about an "image server" and new network protocols on the agenda.
The EPICS practice of putting images into waveforms without any meta info regarding the image format is quite a hack anyway.
Maybe there'll be some news about better ways to handle images, for example including compression of the image data, and then we can optimize BOY to handle those.

Right now I think you're best off using smaller images, slower update rates or simply increasing your memory sizes (actual RAM and JVM maximum).

Thanks,
Kay

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System