3.1 SDDS Toolkit Programs by Category

3.1.1 Mathematical Operations Tools

• sddsbaseline — Remove baselines from column data. Example application: determining the noise level in a video signal and subtracting it from the signal.
• sddschanges — Analyzes changes in column data from page to page in a file, relative to a reference file or the first page. Example application: finding changes in a waveform that is acquired repeatedly, where successive waveforms are on successive pages.
• sddscliptails — Remove tails from column data, where a tail is dubious data on either side of a peak. Example application: removing halo or noise tails from video images of beam spots.
• sddsderiv — Does numerical differentiation of multiple data columns versus a single column, with optional error propogation.
• sddsinsideboundaries — Determines whether points in a two-dimensional space (x, y) are inside any of a series of closed boundaries (or contours).
• sddsinteg — Does numerical integration of multiple data columns versus a single column, with optional error propogation. Example application: finding the field integral an accelerator magnet from a longitudinal field scan.
• sddsinterp — Does interpolation of multiple data columns as a function of a single column. Example application: finding the required current to obtain a desired excitation in a magnet, or interpolating a curve at positions given in a second file.
• sddsnormalize — Normalizes data in multiple columns using various types of normalization factors, determined from the data.
• sddspeakfind — Finds values of columns at locations of peaks in a single column. Example application: finding the position and height of peaks in a power spectrum obtained from a FFT.
• sddsprocess — Probably the most-used toolkit program, excepting sddsplot. Allows creating new parameters and columns with user-specified equations; filtering and matching operations; printing, editing, scanning, and subprocess operations; statistical and waveform analysis of column data to produce new parameters; and much more.
• sddssmooth — Smooths columns of data using multipass nearest-neighbor averaging. Example application: reducing noise in a frequency spectrum prior to finding peaks.
• sddszerofind — Finds values of columns at locations of interpolated zeroes in a single column. Example application: finding zeros of a tabulated function that isn’t known analytically.

3.1.2 Statistics Tools

• sddscorrelate — Computes correlation coefficients and correlation significance between column data. Example application: finding correlations among time series data collected from process variables, and evaluating their signficance to find possible cause-and-effect relationships.
• sddsdistest — Performs statistical tests on data to determine whether the data is drawn from any of various distributions. Example application: determining if a component failure rate matches a Poisson distribution.
• sddsenvelope — Analyzes column data across pages to find minima, maxima, averages, standard-deviations, etc., on a row-by-row basis. Example application: finding the envelope and average of a set of waveforms.
• sddseventhist — Analyzes labeled events in a dataset to provide histograms of the occurences of each type of event. Can also histogram the overlap off all types of events with a single type of event. Example application: correlating the occurence times of alarm signals to determine which alarms usually occur together.
• sddshist — Does histograms of column data. Example application: finding the distribution of a readback that is sampled many times, or of particle coordinates from an accelerator tracking simulation.
• sddshist2d — Does two-dimensional histograms of column data. Example applications: finding the two-dimensional distribution of a pair of readbacks that are sampled many times, or of two particle coordinates (e.g., x and y position) from an accelerator tracking simulation.
• sddsmultihist — Does histograms of multiple columns of data. Example application: finding the distribution of a set of similar readbacks that are sampled many times.
• sddsoutlier — Eliminates statistical outliers from data. Example application: eliminating bad or nonrepresentative data points prior to searching for correlations with sddscorrelate, or computing statistics with sddsprocess.
• sddsprocess — Probably the most-used toolkit program, excepting sddsplot. Allows creating new parameters and columns with user-specified equations; filtering and matching operations; printing, editing, scanning, and subprocess operations; statistical and waveform analysis of column data to produce new parameters; and much more.
• sddsrowstats — Computes row-by-row statistics across multiple columns of data, creating new columns to contain the statistics. Example application: finding the mean value of a set of readout values from time-series data collection, where each readout is in a separate column.
• sddsrunstats — Computes running or blocked statistics of multiple columns. Example applications: smoothing noisy data; finding running averages and error bars for time-series data.
• sddsshiftcor — Computes correlation coefficients between column data as a function of shift position of a reference column. Example application: finding correlations among time series data collected from process variables, including the possibility of time-lags between the process variables due to physical or data collection effects.

3.1.3 Digital Signal Processing Tools

• sddsconvolve — Does FFT convolution, deconvolution, and correlation. Example application: computing the ideal impulse response of a system after you’ve measured the response to a pulse.
• sddsdigfilter — Performs time-domain digital filtering of column data. Example applications: low pass, high pass, band pass, or notch filtering of data to eliminate unwanted frequencies.
• sddsfdfilter — Performs frequency-domain filtering of column data. Example application: applying a filter that is specified as a table of attenuation and phase as a function of frequency.
• sddsfft — Does Fast Fourier Transforms of column data. Example application: finding signficant frequency components in time-varying data, or finding the integer tune of an accelerator from a difference orbit.
• sddsnaff — Does Numerical Analysis of Fundamental Frequencies, a more accurate method of determining principle frequencies in signals than the FFT.

3.1.4 Data Fitting Tools

• sddsexpfit — Does an exponential fit to column data. Example application: finding the exponential lifetime of a beam in a storage ring, or the half-life a radioactive sample.
• sddsgenericfit — Does generic fits to column data. Example application: fitting the sum of two gaussians.
• sddsgfit — Does gaussian fits to column data. Example application: finding the width of a resonance, or the rms size of a beam profile.
• sddsmpfit — Does polynomial fits to multiple columns data, including error analysis. Will do fits to specified orders, fits of specified symmetry, and adaptive fitting. Use sddspfit as a simpler and somewhat more capable alternative for fitting a single column of data. Use sdds2dpfit for two-dimensional fits.
• sddspfit — Does polynomial fits to column data, including error analysis. Will do fits to specified orders, fits of specified symmetry, and adaptive fitting.
• sdds2dpfit — Does two-dimensional polynomial fits to column data.

3.1.5 Data Manipulation Tools

• sddsbreak — Breaks data pages into new, separate pages based on changes in column data and other criterion. Example applications: reorganizing a file to have a limited number of rows in each page, or to have a new page started when a gap is seen in the data.
• sddscast — change the data type of the elements in an SDDS file.
• sddscollapse — Collapses a data set into a single data page by deleting the tabular data and turning the parameters into columns. Example application: abstraction of summary properties of data set following analysis with sddsprocess.
• sddscollect — Reorganizes tabular data from the input file to bring data from several groups of similarly named columns together into a single column per group. Example application: collecting several statistical analyses of many columns into a single column per analysis type.
• sddscombine — Combines any number of data sets into a single data set by adding data from each successive data set to a newly-created data set. Example application: bringing together comparable but distinct data for analysis with sddsprocess. Using sddsprocess, sddscombine, and sddscollapse in sequence repeatedly is a powerful way to analyze and collate large amounts of data.
• sddsconvert — Allows conversion of a data set between binary and ASCII, with optional deletion and renaming of columns, arrays, and parameters . Example application: conversion to binary of an ASCII data set created by a simple program, or by a text editor. N.B.: it is not recommended to use sddsconvert to convert a binary SDDS file to ASCII, then strip the header off and read the ASCII file. This completely bypasses the self-describing aspects of the SDDS file and is not robust. If the program that creates the SDDS file is changed so that the columns are created in a different order, the program that reads the ASCII file will produce unexpected results. Use sdds2plaindata, sddsprintout, or sdds2stream for conversion to non-self-describing files. In this way, you can assure the order of the data is fixed.
• sddsderef — Allows dereferencing (i.e., de-indexing) of array and column data. Example application: converting a column of integer indexes into a column of equivalent text messages, where the text messages are stored once each in an array in the input file (for space-savings).
• sddsdiff — Compares two SDDS files.
• sddsendian — Converts from little-endian to big-endian and vice-versa. Example application: converting binary data from the native format to a format used on another type of computer prior to transferring the data to the other computer.
• sddsexpand — Expands a data set into one page for each row, with column data promoted to parameter data. Essentially the inverse of sddscollapse.
• sddsregroup — Swaps the row indexing and page indexing of data in an SDDS file. That is, the ith row of the jth data page in the input file becomes the jth row of the ith data page of the output file. Example application: viewing the long-term evolution of a repeatedly-sampled waveform at each point in the waveform.
• sddstranspose — Transposes the tabular data in the input file, so that the output file contains one column for each row in the input. Example usage: tranpose an orbit response matrix as part of preparing to use it for feedback.
• sddsmakedataset — writes the input data into a file or pipe in SDDS format. It can be used to make add SDDS file consisting of a small amount of data from the script. It is more convenient than sdds save.
• sddsmatrixmult — Multiplies the tabular data in the two input files to produce a file containing a matrix of the product. Example usage: Multiply a vector of errors with a correction matrix to obtain a vector of corrections to apply in a step-by-step feedback system.
• sddsmatrixop — performs general matrix operations. The matrices and operations are specified on the command line and the operations will proceed in a rpn-like fashion.
• sddsselect — Copies rows from one file based on the presence or absence of matching data in another file. Example application: finding all of the rows from one file that do not appear in a second file.
• sddssort — Sorts the tabular data section of a data set by the values in named columns. Optionally eliminates duplicate rows.
• sddssortcolumn — rearrange the columns of an SDDS data.
• sddssplit — Places each page of a file in a separate, new file. Example application: getting selected pages of a file into separate, single-page files for use with a program that only recognizes the first page.
• sddsxref — Creates a new data set by adding selected rows from one data set to another data set. Example application: cross-referencing the turn-by-turn coordinates of particles in a tracking simulation with the initial coordinates using a particle ID number.

3.1.6 Graphics Tools

• sddscontour — Makes contour and color-map plots from an SDDS data set column, or from a rpn expression of the values in the columns of a data set. Supports FFT interpolation and filtering. Example application: displaying data from a two-dimensional magnetic field scan.
• sddsplot — A highly flexible, device-independent graphics program, equally capable of “quick-and-dirty” or publication quality graphics. Example application: making an X-windows movie of several columns of data that change from page to page in a file.

3.1.7 Image Processing Tools

• sddsimageconvert — Converts a single-column SDDS image file into a multi-column SDDS image file and vice versa.
• sddsimageprofiles — Extracts the profile from a multi-column SDDS image file.
• sddsspotanalysis — Used to locate and give details about spots in multi-column SDDS image files.

3.1.8 Miscellaneous Tools

• elegant2genesis — Processes particle output from the particle tracking code elegant[2] and makes a file suitable for use as the BEAMFILE with the FEL code GENESIS[8].
• sddssampledist — Draws samples from one or more probability distributions. Suitable for making input particle distributions for tracking codes, for example, using user-defined probability distributions.
• sddssequence — Creates an SDDS data set of arithmetic sequences. Example application: generating values for an independent variable, whose values can be used by sddsprocess to produce a mathematical function.
• sddscongen — Creates an SDDS data set by evaluating an rpn expression over a defined 2 dimensional grid. Example application: generating values of a function of two variables on a grid for plotting with sddscontour.
• sddstimeconvert — Converts time data between seconds-since-epoch and calendar breakdown formats. Example application: finding the year, month, and day corresponding to a system time value.

3.1.9 File Protocol Conversion Tools

• csv2sdds — Converts CSV (Comma-Separated-Values) data to SDDS.
• ?? — Converts Hewlett-Packard CITI files to SDDS.
• ?? — Converts Hewlett-Packard HP54542 scope internal format to SDDS.
• ?? — Converts Hewlett-Packard HP54542 scope text format to SDDS.
• hdf2sdds — Converts Hierarchical Data Format (HDF) to SDDS.
• ?? — Converts Spiricon Laser Beam Analyzer files to SDDS.
• plaindata2sdds — Converts plain data file with simple formatting to SDDS.
• sdds2math — Converts SDDS data to a format accepted by Mathematica.
• sdds2mpl — Extracts data columns or parameters from an SDDS data set and creates mpl data files[4].
• sdds2plaindata — Converts SDDS data to a plain data file with simple formatting. This is one of the recommended ways to convert SDDS data to plain ASCII or binary data for input to non-compliant programs. The advantage of using sdds2plaindata is that you can guarantee that the data is emitted in a fixed order.
• sdds2spreadsheet — Converts SDDS data to a format accepted by the Excel and Wingz spreadsheets. Obsolete. Use sddsprintout instead.
• ?? — Converts MAD/LEP TFS files to SDDS.

3.1.10 Text-based Data-review Tools

• sdds2stream — Takes column or parameter data from a list of SDDS data sets and delivers it to the standard output as a stream of values. Example application: getting data into a shell variable for use in a script. sdds2stream may be used to convert SDDS data to plain ASCII text.
• sddsprintout — Makes customized printouts from column, parameter, and array data in an SDDS data set. Also makes spreadsheet-compatible data and plain ASCII text. Example application: making a nicely-formatted printout of data that needs to be reviewed manually.
• sddsquery — Prints a summary of the SDDS header for a data set. Also prints bare lists of names of defined entities, suitable to use with shell scripts that need to detect the existence of entities in the data set.