Savu - notes on basic use

Note: The following tutorial for I13 users is out-of-date.

Tomography Reconstruction : Reconstruction from image data in the HDF format: Savu - notes on standard use


General advice

To make Savu available in a Linux session, execute:

Linux command
module load savu

To make Savu Configurator available in a Linux session, execute:

Linux command
module load savu
savu_config

Any savu_mpi jobs are run on the new compute cluster, named hamilton (after Margaret Hamilton), and can be monitored by executing:

Linux command
module load hamilton
qstat

(note lower-case hamilton).

Suggested processes to include in a process list for tomography reconstruction

A typical Savu process list in DLS may contain all or most of the following processes in its linear chain of operations (ordered in a similar way to that indicated below):


Process positionProcess nameProcess categoryBrief description of processComment(s)Common alternative or more suitable process(es)*
1NxtomoLoaderloaderTo specify the location of raw dataset(s) to be used as input.
  1. Enables one to specify the location of dark- and flat-field datasets for use by any subsequent process that requires this information (e. g. DarkFlatFieldCorrection or Dezinger).
  2. Capable of handling the case of dark- and flat-field images being supplied in individual NeXus datasets.
  3. Expects its image-data input to be integer-valued.
  1. ImageLoader
  2. Hdf5TemplateLoader
  3. SavuNexusLoader
2DezingerfilterTo remove zingers from raw data.
  1. Current implementation of Dezinger expects its image-data input to be integer-valued.
  2. Outputs data in the floating-point format.
  1. DezingerSinogram
  2. DezingerSimple
3DarkFlatFieldCorrectioncorrectorTo apply the standard dark-and-flat-field normalisation to sample projections.
  1. Applies: (projection - dark) / (flat - dark), where dark and flat stand for the averaged dark- and flat-field images, respectively.
  2. Outputs data in the floating-point format.
  3. Needs to be explicitly included in every process list that seeks this correction (specifying the location of dark- and flat-field data in a loader, e. g. NxtomoLoader, is insufficient in this respect).

4DistortionCorrectioncorrectorTo correct data for lens distortion.
  1. Requires information about the centre and coefficients of distortion, which need to be measured beforehand.
  2. Expects its image-data input to be in the floating-point format (generated, for example, by Dezinger or DarkFlatFieldCorrection).
  3. As this correction involves some intensity interpolation, it can result in some smoothing effects which can improve many undesirable artefacts, including ring artefacts.

5CcpiRingArtefactFilterfilterTo suppress ring artefacts.CcpiRingArtefactFilter is positioned before PaganinFilter as it tends to be efficient at suppressing ring artefacts found in typical datasets, but appears to be somewhat less successful on diffused images (which can occasionally be produced by PaganinFilter).
  1. RemoveAllRings
  2. RingRemovalWaveletfft
6VoCenteringfilterTo find an optimal value of CoR automatically.It may be prudent to test the outcome of VoCentering on a small, representative subset of data (use the preview parameter) before embarking on the reconstruction of the entire dataset, using the automatically-determined value of CoR.VoCenteringIterative
7PaganinFilterfilterTo retrieve the phase-contrast information.
  1. If the output images from PaganinFilter are reasonably free from the halo and shade-off artefacts, then it may be preferable to apply VoCentering immediately after this filter (rather than before it).
  2. Since PaganinFilter is essentially a special low-pass filter, its application results in some smoothing effects which can improve many undesirable artefacts, including ring artefacts. In view of this, it may be preferable to apply this filter before applying any dedicated ring-suppression process(es).
FresnelFilter
8RavenFilterfilterTo suppress ring artefacts.RavenFilter may complementarily be used after PaganinFilter as it can lead to an additional reduction of ring artefacts in somewhat diffused images that can occasionally be produced by PaganinFilter.RingRemovalWaveletfft
9AstraReconGpureconstructorTo reconstruct normalised and conditioned data.
  1. AstraReconCpu
  2. TomopyRecon
  3. TomobarRecon
10Hdf5saversaverTo save reconstruction(s) to output file(s).By default, Hdf5saver is the final process in every process list (hence there is normally no need to add it explicitly).TiffSaver

* Note that, when a process is replaced in a given process list by an alternative process, the latter may need to be included at a different position to that of the original process. For example, DezingerSinogram needs be included after DarkFlatFieldCorrection while Dezinger needs to be included before it.


Examples

Example 1: Simple process list including VoCentering

Example 1: Process-list description (click to expand)...
Process positionProcess nameProcess parametersDesired outcomeShape of input image datasetShape of output image datasetComment(s)
1NxtomoLoader
  1. with non-default setting for the preview parameter;
  2. all other parameters at default settings
To load a particular 5-slice subset of the entire source dataset (found at the default location in the input NeXus scan file), representing 5 consecutive sinograms corresponding to tomography slices with indices from 1093 to 1097 (see 1.preview).(Z, img_L, img_W)(Z, 5, img_W)For example, (img_L, img_W) = (2160, 2560) for full-size images recorded by PCO Edge camera, hence the slice located at height 1093 lies approximately in the middle. These images contain 16-bit unsigned-integer data.
2Dezingerat default settingsTo remove zingers from the 5-sinogram subset previously loaded in Process 1.(Z, 5, img_W)(Z, 5, img_W)
3DarkFlatFieldCorrectionat default settingsTo apply the standard dark-and-flat-field normalisation to each of the 5 sinograms loaded in Process 1 (using the dark- and flat-field data found at the default locations in the input NeXus scan file).(Z, 5, img_W)(Z, 5, img_W)
4CcpiRingArtefactFilter

at default settings

To suppress ring artefacts in each of the 5 sinograms loaded by Process 1.(Z, 5, img_W)(Z, 5, img_W)
5VoCentering

at default settings

  1. To find an optimal value of CoR for each of the first 3 sinograms from the 5-sinogram subset loaded in Process 1;
  2. To select a common optimal value of CoR from the above 3 (potentially different) values of CoR, with the view of applying it to reconstruct all, or any subset, of the 5-sinogram subset loaded in Process 1.
(Z, 5, img_W)n/aNote that the preview parameter of this VoCentering process selects the first 3 array slices (indexed 0, 1 and 2) in the 2-nd dimension (axis=1), which correspond to the middle 3 array slices (in the same dimension) with indices 1093, 1094, 1095 in the entire source dataset.
6AstraReconGpuat default settingsTo reconstruct the 5 sinograms loaded in Process 1 with the common optimal value of CoR determined by Process 5 (part 2).(Z, 5, img_W)(img_W, 5, img_W)Note that this AstraReconGpu will automatically receive the value of CoR determined by the VoCentering in Process 5, so it is harmless to leave the centre_of_rotation parameter of this AstraReconGpu at its default value of 0.0.
7Hdf5saverat default settingsTo save the 5 reconstructed tomography slices as a single 3d dataset in the output HDF5 file.(img_W, 5, img_W)(img_W, 5, img_W)This process is implicit.

Example 1: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : [:, 1093:1098, :]
    2)         image_key_path : entry1/tomo_entry/instrument/detector/image_key
    3)                   name : tomo
    4)               3d_to_4d : False
    5)                   flat : [None, None, 1]
    6)              data_path : entry1/tomo_entry/data/data
    7)                   dark : [None, None, 1]
    8)                 angles : None
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) Dezinger
    1)            in_datasets : []
    2)           out_datasets : []
    3)                   mode : 0
    4)            kernel_size : 5
    5)             outlier_mu : 10.0
-------------------------------------------------------------------------------------
 3) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------
 4) CcpiRingArtefactFilter
    1)             num_series : 1
    2)                param_r : 0.005
    3)            in_datasets : []To load a particular 5-slice subset of the entire source dataset (found at the default location in the input NeXus scan file), representing 5 consecutive sinograms corresponding to tomography slices with indices from 1093 to 1097 (see 1.preview).
    4)           out_datasets : []
    5)                param_n : 0
-------------------------------------------------------------------------------------
 5) VoCentering
    1)                preview : [:, 0:3, :]
    2)            start_pixel : None
    3)            search_area : [-50, 50]
    4)            in_datasets : []
    5)          search_radius : 6
    6)                  ratio : 0.5
    7)           out_datasets : ['cor_raw', 'cor_fit']
    8)   datasets_to_populate : []
    9)               row_drop : 20
   10)                   step : 0.5
-------------------------------------------------------------------------------------
 6) AstraReconGpu
    1)               init_vol : None
    2)                preview : []
    3)                    log : True
    4)              algorithm : FBP_CUDA
    5)           n_iterations : 1
    6)               res_norm : False
    7)     centre_of_rotation : 0.0
    8)             FBP_filter : ram-lak
    9)            in_datasets : []
   10)                  ratio : 0.95
   11)           out_datasets : []
   12)             centre_pad : False
   13)              outer_pad : False
   14)             force_zero : [None, None]
-------------------------------------------------------------------------------------


Where to look for the optimal value of CoR determined by VoCentering?

  • To find out which pixel coordinate was actually selected by VoCentering for setting the centre_of_rotation parameter of a subsequent reconstruction process, inspect a dataset located at /entry/final_result_tomo/meta_data/centre_of_rotation/centre_of_rotation in the primary output file, named <scan-number>_processed.nxs. Alternatively, see a dataset located at /<position-in-process-list>-VoCentering-cor_broadcast/data in the intermediate output file named cor_broadcast_p<position-in-process-list>_vo_centering.h5.
  • Incidentally, a dataset located at /entry/final_result_tomo/meta_data/cor_preview/cor_preview in the primary output file contains all (not-necessarily identical) values of CoR determined by VoCentering individually for each sinogram specified by the preview parameter of this process. Alternatively, a dataset located at /<position-in-process-list>-VoCentering-cor_preview/data in the intermediate output file cor_preview_p<position-in-process-list>_vo_centering.h5 contains the same information.



Example 2: More advanced process list including VoCentering

Example 2: Process-list description (click to expand)...


Process positionProcess nameProcess parametersDesired outcomeShape of input image datasetShape of output image datasetComment(s)
1NxtomoLoader
  1. with non-default setting for the preview parameter;
  2. all other parameters at default settings

To load a particular 5-slice subset of the entire source dataset (found at the default location in the input NeXus scan file), representing 5 consecutive sinograms corresponding to tomography slices with indices from 1093 to 1097 (see 1.preview).

(Z, img_L, img_W)(Z, 5, img_W)For example, (img_L, img_W) = (2160, 2560) for full-size images recorded by PCO Edge camera, hence the slice located at height 1093 lies approximately in the middle. These images contain 16-bit unsigned-integer data.
2Dezingerat default settingsTo remove zingers from the 5-sinogram subset previously loaded in Process 1.(Z, 5, img_W)(Z, 5, img_W)
3DarkFlatFieldCorrectionat default settingsTo apply the standard dark-and-flat-field normalisation to each of the 5 sinograms loaded in Process 1 (using the dark- and flat-field data found at the default locations in the input NeXus scan file).(Z, 5, img_W)(Z, 5, img_W)
4CcpiRingArtefactFilter
  1. with parameter tuning being applied on the param_r parameter;
  2. all other parameters at default settings
To suppress ring artefacts in each of the 5 sinograms loaded in Process 1, using the following 7 tuning values for param_r: 0.5, 0.05, 0.005 (default), 0.0005, 0.0001, 0.01, and 0.001.from the 5-sinogram subset loaded in Process 1;(Z, 5, img_W)(Z, 5, img_W, 7)Note that the parameter tuning performed on the param_r of this CcpiRingArtefactFilter adds an extra (4-th) dimension to the original image dataset.
5VoCentering
  1. with non-default setting for the preview parameter;
  2. all other parameters at default settings
  1. To find an optimal value of CoR for each of the first 3 sinograms (of the 5-sinogram subset loaded in Process 1) that were subsequently processed by Process 4 using the 2nd (index 1) value of param_r, i. e. param_r = 0.05 (see 5.preview);
  2. To select a common optimal value of CoR from the above 3 (potentially different) values of CoR, with the view of applying it to reconstruct all, or any subset, of the 35 (=5*7) parameter–tuned sinograms that are currently available for reconstruction in the pipeline.
(Z, 5, img_W, 1)n/aNote that the preview parameter of this VoCentering process selects the first 3 array slices (indexed 0, 1 and 2) in the 2-nd dimension (axis=1), which correspond to the middle 3 array slices (in the same dimension) with indices 1093, 1094, 1095 in the entire source dataset.
6AstraReconGpuat default settingsTo reconstruct all the 35 (=5*7) parameter–tuned sinograms with the common optimal value of CoR determined by Process 5 (part 2).(Z, 5, img_W, 7)(img_W, 5, img_W, 7)Note that this AstraReconGpu will automatically receive the value of CoR determined by the VoCentering in Process 5, so it is harmless to leave the centre_of_rotation parameter of this AstraReconGpu at its default value of 0.0.
7Hdf5saverat default settingsTo save all the 35 (=5*7) reconstructed tomography slices as a single 4d dataset in the output HDF5 file.(img_W, 5, img_W, 7)(img_W, 5, img_W, 7)This process is implicit.

Example 2: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : [:, 1093:1098, :]
    2)         image_key_path : entry1/tomo_entry/instrument/detector/image_key
    3)                   name : tomo
    4)               3d_to_4d : False
    5)                   flat : [None, None, 1]
    6)              data_path : entry1/tomo_entry/data/data
    7)                   dark : [None, None, 1]
    8)                 angles : None
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) Dezinger
    1)            in_datasets : []
    2)           out_datasets : []
    3)                   mode : 0
    4)            kernel_size : 5
    5)             outlier_mu : 10.0
-------------------------------------------------------------------------------------
 3) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------
 4) CcpiRingArtefactFilter
    1)             num_series : 1
    2)                param_r : 0.5;0.05;0.005;0.0005;0.0001;0.01;0.001
    3)            in_datasets : []
    4)           out_datasets : []
    5)                param_n : 0
-------------------------------------------------------------------------------------
 5) VoCentering
    1)                preview : [:, 0:3, :, 1]
    2)            start_pixel : None
    3)            search_area : [-50, 50]
    4)            in_datasets : []
    5)          search_radius : 6
    6)                  ratio : 0.5
    7)           out_datasets : ['cor_raw', 'cor_fit']
    8)   datasets_to_populate : []
    9)               row_drop : 20
   10)                   step : 0.5
-------------------------------------------------------------------------------------
 6) AstraReconGpu
    1)               init_vol : None
    2)                preview : []
    3)                    log : True
    4)              algorithm : FBP_CUDA
    5)           n_iterations : 1
    6)               res_norm : False
    7)     centre_of_rotation : 0
    8)             FBP_filter : ram-lak
    9)            in_datasets : []
   10)                  ratio : 0.95
   11)           out_datasets : []
   12)             centre_pad : False
   13)              outer_pad : False
   14)             force_zero : [None, None]
-------------------------------------------------------------------------------------



Example 3: Simple process list for generating sinograms


The process list below provides an example of how to generate 3 sinograms corresponding to 3 middle tomographic cross-sections:

Example 3: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : [:, mid-1:mid+1, :]
    2)         image_key_path : entry1/tomo_entry/instrument/detector/image_key
    3)                   name : sino
    4)               3d_to_4d : False
    5)                   flat : [None, None, 1]
    6)              data_path : entry1/tomo_entry/data/data
    7)                   dark : [None, None, 1]
    8)                 angles : None
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------
 3) TiffSaver
    1)            in_datasets : []
    2)                 prefix : None
    3)                pattern : SINOGRAM
-------------------------------------------------------------------------------------



Example 4a: Simple process list for generating dark-and-flat-field-corrected radiography images


Note that in the case of radiography the information about image keys or sampling angles is usually not required (and therefore typically absent), and dark- and flat-field images are usually provided in two separate scan or data files, respectively. This scenario is reflected in the example below in which the image_key_path and angles parameters are set to None, and the flat- and dark-field datasets are provided in two separate data files, miro_projections_84122.hdf and miro_projections_84123.hdf, respectively.

Example 4a: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : []
    2)         image_key_path : None
    3)                   name : radiography
    4)               3d_to_4d : False
    5)                   flat : [/dls/i12/data/2019/ee20903-1/rawdata/miro_projections_84122.hdf, entry/data/data, 1]
    6)              data_path : entry/data/data
    7)                   dark : [/dls/i12/data/2019/ee20903-1/rawdata/miro_projections_84123.hdf, entry/data/data, 1]
    8)                 angles : None
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------



Example 4b: Simple process list for generating dark-and-flat-field-corrected tomography raw images


The process list below provides an example of how to select 3 raw images from the middle of a dataset to generate the correspondong 3 dark-and-flat-field-corrected raw images:

Example 4b: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : [mid:mid+3, :, :]
    2)         image_key_path : entry1/tomo_entry/instrument/detector/image_key
    3)                   name : tomo
    4)               3d_to_4d : False
    5)                   flat : [None, None, 1]
    6)              data_path : entry1/tomo_entry/data/data
    7)                   dark : [None, None, 1]
    8)                 angles : None
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------



Example 5: Process list for overriding internal flat- and dark-field images with external ones


Similarly to the process list in Example 4a, the flat- and dark-field datasets in this example are provided in two separate scan files, 122639.nxs and 122640.nxs, respectively. Note that the angles parameter is now set to np.linspace(0.0,180.0,1801), which is appropriate for reconstructing a tomography dataset containing exactly 1801 sample projections (taken over the range of 180 deg) and any number of internal flat- and dark-field images. Savu identifies, and then ignores, these internal flat- and dark-field images using the information provided by the image_key_path parameter, which therefore needs to be specified. This type of process list is useful if the internal flat- or dark-field images are in some way defective.

Example 5: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : []
    2)         image_key_path : entry1/tomo_entry/instrument/detector/image_key
    3)                   name : tomo
    4)               3d_to_4d : False
    5)                   flat : [/dls/i13/data/2019/mg23980-1/raw/122639.nxs, /entry1/pco1_hw_hdf_nochunking/data, 2]
    6)              data_path : entry1/tomo_entry/data/data
    7)                   dark : [/dls/i13/data/2019/mg23980-1/raw/122640.nxs, /entry1/pco1_hw_hdf_nochunking/data, 2]
    8)                 angles : np.linspace(0.0,180.0,1801)
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------
 3) DistortionCorrection
    1)        center_from_top : 1342.27794492
    2)      polynomial_coeffs : [0.9998625936864155, 5.356359274168777e-07, 2.074129717987741e-09, -2.7810309488306555e-13, 5.571390974300683e-17]
    3)       center_from_left : 1072.19611109
    4)              file_path : None
    5)            in_datasets : []
    6)             crop_edges : 20
    7)           out_datasets : []
-------------------------------------------------------------------------------------
 4) RemoveAllRings
    1)            in_datasets : []
    2)           out_datasets : []
    3)                la_size : 71
    4)                    snr : 3.0
    5)                sm_size : 31
-------------------------------------------------------------------------------------
 5) AstraReconGpu
    1)               init_vol : None
    2)                preview : []
    3)                    log : True
    4)              algorithm : FBP_CUDA
    5)           n_iterations : 1
    6)               res_norm : False
    7)     centre_of_rotation : 1262
    8)             FBP_filter : ram-lak
    9)            in_datasets : []
   10)                  ratio : 0.95
   11)           out_datasets : []
   12)             centre_pad : True
   13)              outer_pad : True
   14)               log_func : np.nan_to_num(-np.log(sino))
   15)             force_zero : [None, None]
   16)              vol_shape : fixed
-------------------------------------------------------------------------------------
 6) TiffSaver
    1)            in_datasets : []
    2)                 prefix : None
    3)                pattern : VOLUME_XZ
-------------------------------------------------------------------------------------

Example 6: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) ImageLoader
    1)                preview : []
    2)            data_prefix : None
    3)            flat_prefix : None
    4)            dark_prefix : None
    5)                 angles : None
    6)              frame_dim : 0
    7)           dataset_name : grating
-------------------------------------------------------------------------------------
 2) DistortionCorrection
    1)        center_from_top : 1501.65022718
    2)      polynomial_coeffs : [1.0046313220318024, -1.4178700554417107e-05, 2.1224019269640804e-08, -8.779934725830327e-12, 1.4068158454435102e-15]
    3)       center_from_left : 1739.36341759
    4)              file_path : None
    5)            in_datasets : []
    6)             crop_edges : 45
    7)           out_datasets : []
-------------------------------------------------------------------------------------
 3) TiffSaver
    1)            in_datasets : []
    2)                 prefix : None
    3)                pattern : PROJECTION
-------------------------------------------------------------------------------------



Example 7: Process list for reconstructing a series of identical tomography scans (stored sequentially in a single 3d dataset)


An example process list for reconstructing middle 500 slices (c.f. mid-250:mid+250 in [:, mid-250:mid+250, :, 1] of 1.preview) of the first two scans (c.f. 1 in [:, mid-250:mid+250, :, 1] in 1.preview) in a series of N ≥ 2 tomography scans, each containing 901 images (c.f. 901 in 1.3d_to_4d), with VoCentering being used on every 25-th slice (c.f. 0:end:25 in [:, 0:end:25, :, 0] in 3.preview) of the first scan only (c.f. the rightmost 0 in [:, 0:end:25, :, 0] in 3.preview):

Example 7: Process list in Savu Configurator

>>> disp - a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : [:, mid-250:mid+250, :, 1]
    2)         image_key_path : entry1/tomo_entry/instrument/detector/image_key
    3)                   name : tomo
    4)               3d_to_4d : 901
    5)                   flat : [/dls/b16/data/2020/mm24130-1/tmp/SOTomo/342919.nxs, entry1/pco1_sw_hdf/data, 1]
    6)              data_path : entry1/tomo_entry/data/data
    7)                   dark : [/dls/b16/data/2020/mm24130-1/tmp/SOTomo/342918.nxs, entry1/pco1_sw_hdf/data, 1]
    8)                 angles : None
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------
 3) VoCentering
    1)                preview : [:, 0:end:25, :, 0]
    2)            start_pixel : None
    3)            search_area : [-500, 500]
    4)                  ratio : 0.5
    5)            in_datasets : []
    6)               row_drop : 20
    7)       broadcast_method : median
    8)           out_datasets : [cor_raw, cor_fit]
    9)   datasets_to_populate : []
   10)          search_radius : 6
   11)                   step : 0.5
   12)         average_radius : 0
-------------------------------------------------------------------------------------
 4) RemoveAllRings
    1)            in_datasets : []
    2)           out_datasets : []
    3)                la_size : 71
    4)                    snr : 3.0
    5)                sm_size : 31
-------------------------------------------------------------------------------------
 5) AstraReconGpu
    1)               init_vol : None
    2)                preview : []
    3)                    log : True
    4)              algorithm : FBP_CUDA
    5)           n_iterations : 1
    6)               res_norm : False
    7)     centre_of_rotation : 1280
    8)             FBP_filter : ram-lak
    9)            in_datasets : []
   10)                  ratio : 0.95
   11)           out_datasets : []
   12)             centre_pad : False
   13)              outer_pad : False
   14)               log_func : np.nan_to_num(-np.log(sino))
   15)             force_zero : [None, None]
   16)              vol_shape : fixed
-------------------------------------------------------------------------------------
 6) TiffSaver
    1)            in_datasets : []
    2)                 prefix : None
    3)                pattern : VOLUME_XZ
-------------------------------------------------------------------------------------



Example 8: Simple process list for reconstructing a flat-and-dark-field-corrected dataset that was previously generated in Savu


The process list below provides an example of how to load a flat-and-dark-field-corrected dataset from a Savu Nexus file (the latter having been previously generated using a process list similar to that described in Example 4b) to reconstruct the 100 middle slices:

Example 6: Process list in Savu Configurator

>>> disp -a
-------------------------------------------------------------------------------------
 1) SavuNexusLoader
    1)                preview : {data: [:, mid-50:mid+50, :]}
    2)               datasets : [entry/final_result_tomo]
    3)                  names : [data]
-------------------------------------------------------------------------------------
 2) RemoveAllRings
    1)            in_datasets : []
    2)           out_datasets : []
    3)                la_size : 71
    4)                    snr : 3.0
    5)                sm_size : 31
-------------------------------------------------------------------------------------
 3) AstraReconGpu
    1)               init_vol : None
    2)                preview : []
    3)                    log : True
    4)              algorithm : FBP_CUDA
    5)           n_iterations : 1
    6)               res_norm : False
    7)     centre_of_rotation : 1114
    8)             FBP_filter : ram-lak
    9)            in_datasets : []
   10)                  ratio : 0.95
   11)           out_datasets : []
   12)             centre_pad : True
   13)              outer_pad : True
   14)               log_func : np.nan_to_num(-np.log(sino))
   15)             force_zero : [None, None]
   16)              vol_shape : fixed
-------------------------------------------------------------------------------------
 4) TiffSaver
    1)            in_datasets : []
    2)                 prefix : None
    3)                pattern : VOLUME_XZ
-------------------------------------------------------------------------------------



Example 9: Simple process list for reconstructing an off-centre scan taken over the range of 360 deg


The process list below provides an example of how to employ Convert360180Sinogram to process an off-centre (double-FOV) scan taken over the range of 360 deg. In this example, an optimisation of sinogram stitching is performed using a single (middle) slice (41 slices are loaded by NxtomoLoader, of which 40 slices are cropped out by DistortionCorrection):

Example 6: Process list in Savu Configurator

>>> disp -a

-------------------------------------------------------------------------------------
 1) NxtomoLoader
    1)                preview : [:, mid-20:mid+21:1, :]
    2)         image_key_path : entry1/tomo_entry/instrument/detector/image_key
    3)                   name : tomo
    4)               3d_to_4d : False
    5)                   flat : [None, None, 1]
    6)              data_path : entry1/tomo_entry/data/data
    7)                   dark : [None, None, 1]
    8)                 angles : None
    9)           ignore_flats : None
-------------------------------------------------------------------------------------
 2) DarkFlatFieldCorrection
    1)            in_datasets : []
    2)            upper_bound : None
    3)           out_datasets : []
    4)            lower_bound : None
    5)                pattern : PROJECTION
    6)        warn_proportion : 0.05
-------------------------------------------------------------------------------------
 3) DistortionCorrection
    1)        center_from_top : 1252.39204012
    2)      polynomial_coeffs : [0.9994616257658503, 1.969316697623304e-06, -8.765055925711061e-11, 2.101099317181636e-12, -5.6298542416383075e-16]
    3)       center_from_left : 1200.19325348
    4)              file_path : None
    5)            in_datasets : []
    6)             crop_edges : 20
    7)           out_datasets : []
-------------------------------------------------------------------------------------
 4) Convert360180Sinogram
    1)            in_datasets : []
    2)           out_datasets : [in_datasets[0], cor]
    3)                 center : 2500:2520:1;
-------------------------------------------------------------------------------------
 5) AstraReconGpu
    1)               init_vol : None
    2)                preview : []
    3)                    log : True
    4)              algorithm : FBP_CUDA
    5)           n_iterations : 1
    6)               res_norm : False
    7)     centre_of_rotation : cor
    8)             FBP_filter : ram-lak
    9)            in_datasets : [tomo]
   10)                  ratio : 0.95
   11)           out_datasets : [tomo]
   12)             centre_pad : True
   13)              outer_pad : True
   14)               log_func : np.nan_to_num(-np.log(sino))
   15)             force_zero : [None, None]
   16)              vol_shape : fixed
-------------------------------------------------------------------------------------




Appendices


Appendix A: Interoperability of the parameter-tuning and previewing operations

Since each instance of parameter tuning in Savu adds an extra dimension to the input dataset, it is often desirable to use the previewing (subset-selection) mechanism to reduce the workload of any subsequent resource-demanding processes, e. g. VoCentering or AstraReconGpu. This can be accomplished by setting the preview parameter of VoCentering or, respectively, that of AstraReconGpu to a desirable rank-3 slice of the incoming higher-order dataset, generated by one or more parameter-tuning operations invoked by the preceding processes (see, e.g., Example 2).


Appendix B: Brief reference guide to specifying values for the parameter-tuning operations

Note that not every parameter in Savu is tunable. For any tunable parameter, one can input its tuning values using the methods described below:

TaskSyntaxComment(s)Example(s)
To input an ordered list of n individual values.val_1; val_2;... val_n
  1. Input values need to be semi-colon separated.
  2. Note the absence of the trailing semi-colon.
  3. Input values can be non-numeric.

mod 6.7 1000.8;1010.8;1020.8;1030.8;1040.8

mod 6.centre_of_rotation 1000.8;1010.8;1020.8;1030.8;1040.8

mod 6.4 FBP_CUDA;CGLS_CUDA

mod 6.algorithm FBP_CUDA;CGLS_CUDA

To input an arithmetic sequence of all numbers starting from val_from and not exceeding val_to, calculated using the common difference of cmn_diff.

val_from:val_to:cmn_diff;

  1. The three defining numeric arguments need to be colon separated.
  2. Note the presence of the trailing semi-colon.
  3. The val_to limit will not be included in the output values unless it is one of the elements in the specified arithmetic sequence.
  4. It is sometimes useful to expand this Savu expression in Unix shell using the sequence-of-numbers (seq) command (note a different syntax), followed by the number-lines (nl) command, e.g.

    Linux command: seq <FIRST> <INCREMENT> <LAST>
    ~>seq 1000.8 10.0 1040.8 | nl
         1    1000.8
         2    1010.8
         3    1020.8
         4    1030.8
         5    1040.8
    ~>seq 1000.8 10.0 1040.8 | nl -v 0
         0    1000.8
         1    1010.8
         2    1020.8
         3    1030.8
         4    1040.8
    ~>

    (note that the indexing scheme of ImageJ is one-based, whereas that of DAWN, hdfview and TiffSaver's file output is zero-based). Alternatively, one can use:

    n = 1, 2, 3,...
    1-based seq(n) = FIRST_VAL + (n-1) * INCREMENT

    or

    n = 0, 1, 2,...
    0-based seq(n) = FIRST_VAL + n * INCREMENT

mod 6.7 1000.8:1040.8:10.0;

mod 6.centre_of_rotation 1000.8:1040.8:10.0;

mod 6.centre_of_rotation 1020.8-2*10.0:1020.8+2*10.0:10.0;



Appendix C: Batch processing

Batch processing in Savu can be done with the help of the savu-batch command, whose arguments are ordered in a logically identical way to that employed by the savu_mpi command, the only difference being the first argument, which is used to supply a path to a user-prepared batch file:

savu_mpi                      <SCAN_DATA_FILE> <SAVU_PROCESS_LIST_FILE> <OUT_DIR>
savu-batch <BATCH_PARAM_FILE> <SCAN_DATA_DIR>  <SAVU_PROCESS_LIST_DIR>  <OUT_DIR> 


To make the savu-batch command available in Linux terminal, execute 'module load python/ana' and 'module load tomography':

Linux Command: Savu-batch

~>module load python/ana
~>module load tomography
~>savu-batch
Usage: savu-batch <BATCH_PARAM_FILE> <SCAN_DATA_DIR> <SAVU_PROCESS_LIST_DIR> <OUT_DIR> [-d <INTERMED_DIR>] [-e EMAIL_ADDRESS_1[;EMAIL_ADDRESS_2;... EMAIL_ADDRESS_N]] [-t <TIME_OUT_MINUTES>] [-v]
Run a batch of reconstructions using Savu.
     -h                  display this help and exit.
     -d INTERMED_DIR     save all intermediate files in INTERMED_DIR directory.
     -e EMAILS           send e-mail notification(s) to (semicolon-separated) address(es) in EMAILS.
     -t TIMEOUT_MINUTES  monitor each cluster job no longer than TIMEOUT_MINUTES.
     -v                  be verbose: send e-mail notifications on completion of each cluster job (in addition to sending a summary notification on completion of the entire batch).
~>


The first argument of savu-batch, <BATCH_PARAM_FILE>, is a path to comma-separated values (CSV) file, containing a table of desired batch parameters, e.g.

batch.csv

#<scan number>,<proces list>[,[CoR(s)],[output dir]]                                            # commented-out lines are ignored (and so are any comments on the RHS of valid lines)
102923,  4x/spl_f102921_d102922.nxs, 1197.75, 4x/%s
102974,  4x/spl_f102921_d102922.nxs, 1255.50, 4x/%s                                                 # same process-list file as in line 2, used with different CoR to reconstruct a different scan
102923,  4x/spl_f102921_d102922_pci.nxs, 1197.75, 4x/%s/pci                                 # same scan as in line 2, reconstructed with phase-contrast retrieval
102990,  4x/spl_f102984_d102985.nxs, 1254.6:1255.9:0.1;, preview/4x/%s              # CoR tuning in dedicated sub-directory
102986,  4x/spl_f102988_d102989.nxs, 1256.50, 4x/%s
103038, 10x/spl_f103036_d103037.nxs,, 10x/%s                                                                # CoR intentionally NOT provided (CoR found in <SAVU_PROCESS_LIST_DIR>/10x/spl_f103036_d103037.nxs is used as is)
103050, 10x/spl_f103048_d103049.nxs, 919.50, 10x/%s
103050, 10x/spl_f103048_d103049_lensdistortcrop40.nxs, 919.50-40, 10x/%s    # Savu-interpretable mathematical expression for adjusting CoR by 40 pixels due to the 40-pixel cropping in DistortionCorrection
104022, spl_f103048_d103049.nxs, 1110.25, tests                                                             # scan-numbered output sub-directory intentionally NOT provided (output goes to <OUT_DIR>/tests)
103052, spl_f103048_d103049.nxs, 910.50,                                                                    # output sub-directory intentionally NOT provided (output goes to <OUT_DIR>)


Notes:

  1. savu-batch reconstructs scans using the savu_mpi command (on the cluster).
  2. savu-batch reconstructs scans sequentially one-by-one, monitoring and waiting (for the duration of <TIMEOUT_MINUTES>) for each reconstruction job to complete (on the cluster) before launching the next one.
  3. savu-batch reconstructs scans in the same order as that specified in <BATCH_PARAM_FILE>.
  4. savu-batch makes its own, internal copies of all input process-list files, specified in <BATCH_PARAM_FILE>, at the time of executing the savu-batch command, and then, if required, each of these copies is appropriately edited just before the savu_mpi command is executed on each of them.
  5. The values of CoR specified in <BATCH_PARAM_FILE> are used as they are to set the centre_of_rotation parameter of any (active) reconstructor process found in the corresponding Savu process-list file (hence the user is responsible for making any adjustments needed to account for any additional cropping requested by, for example, the crop_edges parameter of DistortionCorrection, etc.).
  6. If no value of CoR is specified for some item in <BATCH_PARAM_FILE>, then the value of the centre_of_rotation parameter found (at the time of launching savu-batch) in the corresponding Savu process-list file is used.
  7. savu-batch is able to handle process lists that don't include any processes requiring CoR (in which case an empty space, or any numerical value, can be provided as nothing depends on it).


Example:

Linux command: savu-batch

>>> savu-batch /dls/i13/data/2018/mt12345-6/processing/process_lists/batch/batch.csv /dls/i13/data/2018/mt12345-6/raw /dls/i13/data/2018/mt12345-6/processing/process_lists /dls/i13/data/2018/mt12345-6/processing/reconstruction -d /dls/i13/data/2018/mt12345-6/tmp -e &quot;whoever@wherever&quot;


In the above savu-batch example, the arguments are as follows:

ArgumentArgument ValueComment(s)
<BATCH_PARAM_FILE>
/dls/i13/data/2018/mt12345-6/processing/process_lists/batch/batch.csv

<SCAN_DATA_DIR>
/dls/i13/data/2018/mt12345-6/raw
Contains all Nexus scan files specified in <BATCH_PARAM_FILE>, e.g. /dls/i13/data/2018/mt12345-6/raw/102923.nxs, etc.
<SAVU_PROCESS_LIST_DIR>
/dls/i13/data/2018/mt12345-6/processing/process_lists
  1. Contains all Savu process-list files specified in <BATCH_PARAM_FILE>, e.g. /dls/i13/data/2018/mt12345-6/processing/process_lists/4x/spl_f102921_d102922.nxs, etc.
  2. If <BATCH_PARAM_FILE> contains an absolute path to some Savu process-list file (as opposed to a path specified relative to <SAVU_PROCESS_LIST_DIR>), then this absolute path is always respected (i.e. <SAVU_PROCESS_LIST_DIR> is overridden).

<OUT_DIR>
/dls/i13/data/2018/mt12345-6/processing/reconstruction
  1. Any intermediate output sub-directories are automatically created if they don't already exist on the file system (provided the user has sufficient file-access permisions).
  2. For example, reconstruction of scan 102923, specified in line 2 of <BATCH_PARAM_FILE>, will be saved by Savu in /dls/i13/data/2018/mt12345-6/processing/reconstruction/4x/102923/<YYYYMMDDhhmmss>_102923/, i.e. the 4x/%s output directory, specified in the same line of <BATCH_PARAM_FILE>, is expanded to 4x/102923/, whereas the last (innermost) sub-directory, <YYYYMMDDhhmmss>_102923/, is automatically generated by Savu itself.
  3. If <BATCH_PARAM_FILE> contains an absolute path to some output directory (as opposed to a path specified relative to <OUT_DIR>), then this absolute path is always respected (i.e. <OUT_DIR> is overridden).
<INTERMED_DIR>
/dls/i13/data/2018/mt12345-6/tmp


Additional notes:

  1. When preparing batch files, please use a Linux text editor as files created under Windows are not compatible.
  2. If you intend to run multiple batches, please wait for each individual batch to finish before submitting another one.
  3. When supplying an e-mail address, please bear in kind that some mail boxes may automatically block machine-generated e-mail notifications.