[Xrays@aps.anl.gov] RESPONSE for 2006 XSD Scientific Software Workshop User Survey

[Andrew Allen]

June 21, 2006

Fundamental needs and opportunities in scientific software systems for  
x-ray data reduction, analysis, modeling and
simulation at the APS and in the X-ray community:

1. What are the limitations of current tools for
   x-ray data reduction, analysis, modeling, and simulation?

Many things are piecemeal and left to the user to sort out.
At some beam lines the basic reduction software is glitchy and labor  
intensive to use.
Their is no house style (hence standard) for x-ray data reduction.
For data analysis the situation is frequently worse. I am aware of very  
few data analysis packages, readily available to the user, that are  
home-grown at APS.
Frequently, the best course of action is to go to other external  
web-sites, frequently to ESRF - where there seems to be a large body of  
analysis software built up specifically for ESRF users.
Clearly ESRF is very different in this regard; so are the US neutron  
user facilities (at least some of them!). There are also some bright  
spots at APS, e.g., sector 33.

2. What additional tools are needed?

The cutting edge has to be with the high-Gb data-producing beam lines  
such as 34-ID, 2-BM, etc:
 From the many discs of data obtained over a few days, how does one  
extract the key results and salient conclusions within a few weeks?
This will require fast machines, large amounts of memory, and very  
sophisticated techniques to "mine" the data and pull together the  
relatively small proportion that satisfies the "necessary" and  
"sufficient" conditions for a successful result.
But NASA and NOAA have to do this all the time. The APS should consider  
setting up a partnership with these agencies (and others) in order to  
bring sufficient resources to bear on this increasingly daunting  
challenge. Having put into place some strategies and resources needed  
to meet the challenge, it will then be essential to provide the users  
with easy remote access to the data and to the processing system. [I  
think this item could form the basis of a budget proposal.]

  3. How can the existing tools be improved?

Major manpower effort is required to write reduction and analysis  
software tools, together with appropriate manuals, that will guide the  
user through to getting publishable results.
This means individual personnel taking responsibility for the packages  
produced, and interactively improving/advancing the software tools with  
an ongoing interaction with the user community.
Very little can be just written and left (orphaned!); there has to be  
an ongoing interaction in the various experiment techniques areas.  
While software specialists may be needed, it is essential that the beam  
line scientists adopt intellectual ownership of what is provided, but  
in such a way that it serves the user community, not just their own  
science.

4. What will most affect the scientific impact of your work?

Clearly, all of the above are relevant here, but I would add that some  
more thought could be given to how data are moved around from  
instrument to instrument and even between techniques.
As a crystallographic journal editor, I am becoming aware of an  
increasing number of software submissions that are more sophisticated  
in handling data from different places, in different forms, across  
platforms, etc. It might be worth the APS as a whole deciding which  
program languages will be supported, and then converting as much of the  
data reduction and analysis software as possible to run on many or all  
of the supported languages and platforms.

I realize that the above sets out some daunting tasks. However, to take  
some or all of these issues on in a serious coordinated way would put  
the APS at the center of x-ray science on the world stage to a greater  
degree than at present. Clearly, the ESRF has done this in Europe, very  
successfully. The fact that many APS users look to the ESRF for their  
data analysis tools says rather a lot! Obviously, the APS could not do  
the things I suggest above without a very active and ongoing  
collaboration with its users.

I hope something here is of use!

Andrew Allen

-------------------------------------------------
Dr. Andrew J. Allen, Physicist
U.S. Department of Commerce
National Institute of Standards and Technology
Building 223, Room B206
100 Bureau Drive, Stop 8520
Gaithersburg, MD 20899-8520
Phone: 301-975-5982
FAX: 301-975-5334
E-mail: andrew.allen at nist.gov
------------------------------------------------------------------------ 
---