API Reference

X-SOCS data processing is implemented in the xsocs.process package. It provides:

xsocs.process.merge: SPEC+EDF to HDF5

This module provides the API to convert SPEC+EDF to HDF5 files.

It provides a helper function merge_scan_data() to perform the merge.


merge_scan_data(output_dir, spec_fname, beam_energy=None, chan_per_deg=None, center_chan=None, scan_ids=None, img_dir=None, n_proc=None, version=1, nr_padding=None, nr_offset=None, compression='DEFAULT', overwrite=False, image_roi=None)

Creates a “master” HDF5 file and one HDF5 per scan. Those scan HDF5 files contain spec data (from spec_fname) as well as the associated image data. This file will either contain all valid scans or the one selected using the scan_ids parameter. A valid scan is a scan associated with an (existing) image file. Existing output files will be overwritten.

  • output_dir (str) – folder name into which output data (as well as temporary files) will be written.

  • spec_fname (str) – path to the spec file.

  • beam_energy (float) – beam energy in ….

  • chan_per_deg (array_like) – 2 elements array containing the number of channels per degree (v, h) (as defined by xrayutilitied, used when converting to reciprocal space coordinates).

  • center_chan (optional array_like) – 2 elements array containing the coordinates (v, h) of the direct beam position in the detector coordinates.

  • scan_ids (optional array of int) – array of scan numbers to add to the merged file. If None, all valid scans will be merged.

  • img_dir (str) – directory path. If provided the image files will be looked for into that folder instead of the one found in the scan headers.

  • n_proc (Union[int,None]) – Number of threads to use when merging files. If None, the number of processes used will be the default config value (usually the number of cores).

  • version (int) – version of the spec file. It is currently used to get the offset and padding to apply to the nextNr value found in the spec scan headers. This nextNr is then used to generate the image file name. Set it to 0 if you are merging data generated before April 2016 (TBC).

  • nr_padding (int) – zero padding to apply to the nextNr number found in the SPEC file.

  • nr_offset (int) – Offset to apply to the nextNr number found in the SPEC file.

  • compression (Union[str,int]) – The HDF5 compression to use.

  • overwrite (bool) – True to allow overwriting already existing output file

  • image_roi (Union[List[int],None]) – Detector image ROI (origin_row, origin_column, height, width) to save, or None (default) to save the whole image


a list of scan IDs that were merged

Return type


Sample Code

import time

from xsocs.process.merge import merge_scan_data

# output directory (some temporary files will also be written there)
output_dir = '/path/to/output/'

# path to the spec file
spec_f ='/path/to/spec/scan.spec'

# path to the image files, if stored in a different place than the path written
# in the spec file (else, set to None)
img_base = '/path/to/img/dir/'

# the beam energy (note that this value can be changed later when calling the
# img_2_qpeak function)
# Set to None to read it from scan headers
beam_energy = 8000.

# merge_scan_data will take all scans from the spec file that have a matching
# image file. If you only want a subset of those scans you can provide a list
# of scan numbers (i.e : the #S xxx lines in the scan headers)
# for example, if we only want scans 48.1, 54.1 and 68.1 we would write :
# scan_ids = ['48.1', '54.1', '68.1']
scan_ids = None

# this (temporary?) keyword is used to tell the function about the format
# of the spec file. So far there is only two supported values :
# 0 : files created before March 2016 (To Be Confirmed)
# 1 : files creqted after March 2016 (To Be Confirmed)
# At the moment it changed the way the nextNr value in the scan header is
# used to generate the image file name
# (in version 0 we actualy have to take nextNr-1 and pad it to get a 4 digit
# number, while in version 1 we pad the value to to 5 digits)
version = 1

# you can also set the offset and padding with the nr_offset and nr_padding
# keywords to the merge_scan_data function, or ignore them if you want
#  to use the default values specified by "version".
# nr_offset = -1
# nr_padding = 4

# channels (pix.) per degree (used by xrayutilities when converting to
# qspace coordinates)
# (not that this value can also be changed later when calling the
# img_2_qpeak function)
chan_per_deg = [318., 318.]

# direct beam position in the detector coordinates
center_chan = [140, 322]

# Detector image ROI to save as (row, column, height, width), example:
# image_roi = [100, 100, 200, 200]
# Set to None to save the whole image:
image_roi = None

# checks if some of the output files already exist
# set it to True if you dont care about overwriting files
overwrite = False

t_merge = time.time()


t_merge = time.time() - t_merge
print('Total time spent : {0}'.format(t_merge))

xsocs.process.qspace: QSpace conversion

This modules provides the API to convert image stacks to QSpace histogram

It provides a helper function kmap_2_qspace() to run the conversion


kmap_2_qspace(xsocsH5_f, output_f, qspace_dims, medfilt_dims=None, maxipix_correction=False, normalizer=None, roi=None, mask=None, n_proc=None, overwrite=False)
  • xsocsH5_f (str) – path to the HDF5 file containing the scan counters and images

  • output_f (optional str) – name of the file that will contain the conversion results.

  • qspace_dims (array_like) – qspace dimensions along the qx, qy and qz axis.

  • medfilt_dims (array_like) – 2-tuple indicating size of median filter to be applied to each detector image.

  • maxipix_correction (bool) – if True, interpolate the maxipix4 gaps

  • normalizer (str) – spec file column containing the intensity of the incident beam for normalization

  • output_f – Name of the output file the results will written to. This file will be created in output_dir. If not set, the file will be named ‘qspace.h5’. This file will be overwritten if it already exists.

  • roi (optional array_like (x_min, x_max, y_min, y_max)) – rectangular region which will be converted to qspace. This must be a four elements array containing x_min, x_max, y_min, y_max.

  • mask (optional 2D numpy.ndarray) – mask array indicating bad pixels of the detector. A non-zero value means that the pixel is masked.

  • n_proc (Union[int,None]) – number of process to use. If None, the number of processes used will be the default config value (usually the number of cores).

  • overwrite (bool) – if set to False, an exception will be raise if the output file already exists.

Sample Code

from xsocs.process.qspace import kmap_2_qspace, QSpaceConverter

# ========================
# Files
# ========================
# output HDF5 file name, default is qspace.h5 if not set
output_f = 'my_qspace.h5'

# path to the hdf5 "master" file (see id01_spec function)
xsocs_h5 = '/path/to/data.h5'

# Conversion parameters

# number of "bins" for the qspace cubes
qspace_dims = (28, 154, 60)

# size of median filter window to apply to each frame
medfilt_dims = None # e.g. (3,3)

# set disp_times to True if you want to output some info in stdout
QSpaceConverter.disp_times = True

# positions (on the sample) to convert to qspace
# if pos_indices will be ignored if rect_roi is provided
# rect_roi = [x_min, x_max, y_min, y_max] (sample positions)
roi = None

# whether to redistribute the intensity to fill the mpx4 gaps
maxipix_correction = False

# data column containing the monitor readings of primary beam intensity
monitor = "cnt1"

# Mask:
# boolean array of equal shape to detector frames to filter out bad pixels
# A non-zero value means that the pixel is masked
mask = None # numpy.ndarray

# set to true if you want to overwrite the output file
# otherwise an exception will be raised if the file already exists
overwrite = False

# number of processes to use
# If None, will use the number of availble core (see multiprocessing.cpu_count)
n_proc = None


xsocs.process.fit: Fit/COM

This module provides Gaussian fit/Center-of-Mass QSpace reduction processing

It provides the PeakFitter class to run QSpace histogram projection on the axes and either a gaussian fit or Center-of-mass with optional background subtraction.


class PeakFitter(qspace_f, fit_type=0, indices=None, n_proc=None, roi_indices=None, background='None')

Class performing fit/com processing

  • qspace_f (str) – path to the HDF5 file containing the qspace cubes

  • fit_type (FitTypes) –

  • indices (Union[numpy.ndarray,None]) – indices of the cubes (in the input HDF5 dataset) for which the dim0/dim1/dim2 peaks coordinates will be computed. E.g : if the array [1, 2, 3] is provided, only those cubes will be fitted.

  • n_proc (Union[int,None]) – Number of process to use. If None, the config value is used.

  • roi_indices (Union[List[List[int]],None]) – QSpace ROI to process

  • background (BackgroundTypes) – The background subtraction to perform

class FitTypes

Kind of fit available

class BackgroundTypes

Enum of background subtraction types:

  • NONE: No background

  • CONSTANT: Remove constant (= min of the data) background

  • SNIP: Statistics-sensitive Non-linear Iterative Peak-clipping algorithm

Sample Code

# coding: utf-8
This scripts illustrates the API of the gaussian/center-of-mass processing

import sys
import numpy
from xsocs.process.fit import PeakFitter, FitTypes, BackgroundTypes

# path to the hdf5 file written by the kmap_2_qspace function
qspace_f = '/path/to/qspace.h5'

# Name of the text file where to store the resuls
result_file = 'results.txt'

# List of QSpace indices to process
# This selects sample positions for which QSpace are processed.
# If None, all QSpace are processed
indices = None

# Number of processors to use
# If None, all available cores are used.
n_proc = None

# Set-up the processing for a gaussian fit without background subtraction
fitter = PeakFitter(qspace_f,
                    roi_indices=None,  # QSpace ROI

# Run the processing and get the results
results = fitter.peak_fit()

# Check for errors
if fitter.status != fitter.DONE:
    print("Fit process failed")

# Prepare columns header and data
# Start with position (x, y) on the sample
headers = ['sample_x', 'sample_y']
values = [results.sample_x, results.sample_y]

# Add one column for each parameter of the fit/COM result
# For each axis of the QSpace
for dimension, axis_name in enumerate(results.qspace_dimension_names):
    # For each fitted/computed parameter of the result
    for parameter_name in results.available_result_names:
        # Add its name to the header and its data to the values
        headers.append(axis_name + '_' + parameter_name)
        values.append(results.get_results(dimension, parameter_name))

# Transpose values from (parameters x points) to (points x parameters)
values = numpy.transpose(numpy.array(values))

# Write results to text file
with open(result_file, 'w') as res_f:
    res_f.write('\t'.join(headers) + '\n')
    for row_values in values:
        res_f.write('\t'.join(str(v) for v in row_values) + '\n')