Reconstruction from image data in the HDF format: the tomo-centre and tomo-recon commands


Introduction

The functionality of the Tomo-Recon GUI (in Dawn) relies on a number of scripts which can also be executed directly from the Linux command line. This may be necessary if one aims at reconstructing a non-standard tomography dataset or if one would like to improve the quality of reconstructed slices. For example, it can be useful to execute these scripts from the Linux command line in the following situations:

  • flat- or dark-field images are stored in separate Nexus datasets or files;
  • the information about tomography angles, normally stored in the Nexus scan file, is for some reason missing or inaccurate;
  • to optimise ring-artefact suppression (use your own copy of the default settings file, /dls_sw/apps/tomopy/tomopy/src/settings.xml, for this task - see Appendix A);
  • to use some more advanced tomography reconstruction options, specified via settings.xml, e.g. to reduce the bit-depth of output images (use your own copy of the default settings file, /dls_sw/apps/tomopy/tomopy/src/settings.xml, for this purpose - see Appendix B);
  • to run a batch reconstruction of similar scans (see Appendix C).

The scripts form part of a dedicated module, called tomography. As with any other software-environment module in DLS, please execute:


Linux command
module load tomography

to make all these scripts available in your current Linux session; the scripts include the following items:

Script NameScript TypeScript DescriptionScript Usage
Comment(s)
tomo-centreBashTo find an optimal value of the centre of rotation (CoR).
tomo-centre [options] <nexus_file> <output_directory>

Given a slice number and an initial trial value of CoR, this scripts reconstructs this slice with a series of different values of CoR which are smaller or larger than the input value of CoR. These trial reconstructions facilitate the task of finding an optimal value of CoR by visual inspection. 

The functionality of this script is similar to that of qcentrexml.py for image data in the TIFF format.

tomo-reconBashTo perform tomography reconstruction of data.
tomo-recon [options] <nexus_file> <output_directory>

The functionality of this script is similar to that of recon_arrayxml.py for image data in the TIFF format.

tomo-fixBashTo detect and, if necessary, generate any missing reconstructed images (this is required, for example, if a cluster node is down or in similar circumstances).
tomo-fix [options] <nexus_file> <directory_to_check>
This script is automatically executed whenever tomo-recon is run.


Please note that absolute paths need to be supplied for the <nexus_file>, <output_directory>, and <directory_to_check> arguments. Please also note that the input <nexus_file> must contain a well-formed tomo_entry, described in more detail on Image data in the HDF format.

Important note

If you copy-and-paste any commands or code from this page, please make sure that all your input arguments are correct before executing anything. In particular, please make sure that the relevant input arguments match the size of your images and that all your input paths to files or directories point to some existing and accessible objects on the file system.

Tip

The easiest way to select (for copying) a long command displayed in a code-block is to double-click anywhere on this command's text.

Log files

Recent log files can be found in a relevant sub-directory of the /dls/tmp/tomopy directory:

Linux command
~>cd /dls/tmp/tomopy/
~>ll
total 27560
drwxrwxrwx. 2 ssg37927 ssg37927  3313664 Jul  7 16:05 dt64
...
drwxrwxrwx. 2 vxu94780 vxu94780  5787648 Jul  7 17:44 tomo-centre
drwxrwxrwx. 2 vxu94780 vxu94780   118784 Jul  4 21:13 tomo-compress
drwxrwxrwx. 2 fsy24678 fsy24678 18866176 Jul  7 13:08 tomo-recon
~>


tomo-centre


The tomo-centre script expects a certain number of arguments and provides some additional options to choose from. When executed, it first creates an appropriate Linux environment and then invokes a Python script called selection_recon.py. All the input arguments and options, supplied for running tomo-centre, are passed in to selection_recon.py. As usual, a more detailed description of all those arguments and options can be viewed by executing:

Linux command
tomo-centre --help

or

Linux command
tomo-centre -h


At the time of writing this section (18 Sep 2017), either of the above two commands outputs the following description:

tomo-centre -h

         Welcome to the DLS compute cluster

         For MPI jobs, please use 'module load openmpi'.

         If using a different OpenMPI installation,
         or manually specifying path to OpenMPI, option
         '-mca orte_forward_job_control'
         must be added to mpirun to ensure cluster functionality.

         Please report any issues to linux.manager@diamond.ac.uk

Loading 64-bit Oracle instantclient, version 11.2
Loading 64-bit python, version 2.7.2
Loading 64-bit numpy, version 1.6.1
Usage: selection_recon.py [options] data output_directory

Options:
  -h, --help            show this help message and exit
  -m MACHINES, --machines=MACHINES
                        Number of machines to deploy to
  -s SLICE, --slice=SLICE
                        Slice selected for processing
  -t TEMPLATE, --template=TEMPLATE
                        Template XML file
  -w WSAMP, --width_sample=WSAMP
                        Set the subsampling of the sinograms width
  -l LSAMP, --length_sample=LSAMP
                        Set the subsampling of the sinograms length
  -c CSTART, --cstart=CSTART
                        Starting value for the centre of rotation
  --ctot=CTOT           Total number of different values for the centre of
                        rotation
  --cstep=CSTEP         The step between two consecutive values for the centre
                        of rotation
  -r RUN_SLICES, --run_slices_loc=RUN_SLICES
                        set the run_slices.sh location
  -n, --new_cluster     use the new cluster
  --dark_file=DARK_FILE
                        Path to the file containing dark images
  --dark_path=DARK_PATH
                        path in the dark file to the data
  --flat_file=FLAT_FILE
                        Path to the file containing flat images
  --flat_path=FLAT_PATH
                        path in the dark file to the data
  --recon_range=RECON_RANGE
                        range for the reconstruction to be done over
  --d0f1                If option included (True), use 0's for dark- and 1's
                        for flat-field images
  --scan_id             If option included (True), incorporate the ID of the
                        input Nexus scan file into output filenames



Option SwitchOption's Default ValueComment(s)
-m
2

-sNone (Python keyword)Slice number must be less than the vertical size (in pixels) of the images. 
-t
/dls_sw/apps/tomopy/tomopy/src/settings.xml
1. Absolute path needs to be supplied (if different from default).
2. Equivalent to --template.
-w
1

-l
1

-c
2004

-r
/dls_sw/apps/tomopy/tomopy/bin/run_slices.sh
Path to a tomography reconstruction script to be used (advanced users only).
-n
False

--dark_file
n/a
1. Path to the Nexus file containing dark-field images.
2. Do not specify this path if dark-field images are in the same Nexus file as projections.
--dark_path
/entry1/pco1_hw_hdf/data
1. To be supplied only if --dark_file option is used (see --dark_file).
2. Nexus path to dark-field data residing in the file specified by --dark_file, eg /entry1/instrument/pco1_hw_hdf_nochunking/data. If in doubt, use Dawn or hdfview to verify it.
3. Data must contain only dark-field images (if image key is present, it is ignored).
4. These external dark-field images override any internal dark-field images that may be stored alongside projections in <nexus_file>.
--flat_file
n/a
1. Path to the Nexus file containing flat-field images.
2. Do not specify this path if flat-field images are in the same Nexus file as projections.
--flat_path
/entry1/pco1_hw_hdf/data
1. To be supplied only if --flat_file option is used (see --flat_file).
2. Nexus path to flat-field data residing in the file specified by --flat_file, eg /entry1/instrument/pco1_hw_hdf_nochunking/data. If in doubt, use Dawn or hdfview to verify it.
3. Data must contain only flat-field images (if image key is present, it is ignored).
4. These external flat-field images override any internal flat-field images that may be stored alongside projections in <nexus_file>.
--recon_range
-1
If default is used, angular range is automatically determined from rotation-angle data stored in the input Nexus file, ie <nexus_file>. 
This option is particularly useful for limited-angle reconstruction.
--d0f1 
False (implicit)
If this option is included on the command line, then a synthetic dark-field image consisting of all 0's and another synthetic flat-field image of all 1's are used during 
reconstruction. This work-around enables one to reconstruct datasets that have been dark-and-flat-field-corrected beforehand.
--scan_id
False (implicit)
If this option is included on the command line, then the scan ID is automatically included in the output filenames, eg for Nexus scan file 91809.nxs, a typical output filename 
would be recon_91809_127850_02032.tif (as opposed to recon_127850_02032.tif).


BASIC EXAMPLE


In most cases, basic use of tomo-centre is adequate.

Linux command
tomo-centre -s 1000 -c 1269.5 --ctot 10 --cstep 0.1 -n /dls/i13/data/2014/mt9377-2/raw/29384.nxs /dls/i13/data/2014/mt9377-2/processing/reconstruction/29384/


In the above basic example, the command structure is as follows:

Usage
tomo-centre [options] <nexus_file> <output_directory>
Command Argument(s)Example Command Argument(s)Comment(s)
[options]
-s 1000 
-c 1269.5 --ctot 10 --cstep 0.1
-n
SLICE (used by the s-option) should not exceed the (pixel) height of raw images.
<nexus_file>
/dls/i13/data/2014/mt9377-2/raw/29384.nxs 

<output_directory>
/dls/i13/data/2014/mt9377-2/processing/reconstruction/29384/ 
If this output directory doesn't already exist, it will automatically be created.


MOST EXTENSIVE EXAMPLE


In more complicated cases, tomo-centre needs to be executed with  all or an appropriate selection of additional input arguments.

Linux command
tomo-centre -s 1000 -c 1269.5 --ctot 10 --cstep 0.1 -n --dark_file=/dls/i13/data/2014/mt9377-2/raw/29375.nxs --dark_path=/entry1/pco1_sw/data --flat_file=/dls/i13/data/2014/mt9377-2/raw/29383.nxs --flat_path=/entry1/pco1_sw/data /dls/i13/data/2014/mt9377-2/raw/29384.nxs /dls/i13/data/2014/mt9377-2/processing/reconstruction/29384/ --recon_range=180.0 --template=/dls/i13/data/2014/mt9377-2/processing/drain1/settingsHDF.xml


In the above most extensive example, the command structure is as follows:

Usage
tomo-centre [options] <nexus_file> <output_directory>
Command Argument(s)Example Command Argument(s)Comment(s)
[options]
-s 1000 
-c 1269.5 --ctot 10 --cstep 0.1
-n
--dark_file=/dls/i13/data/2014/mt9377-2/raw/29375.nxs --dark_path=/entry1/pco1_sw/data
--flat_file=/dls/i13/data/2014/mt9377-2/raw/29383.nxs --flat_path=/entry1/pco1_sw/data
--recon_range=180.0
--template=/dls/i13/data/2014/mt9377-2/processing/drain1/settingsHDF.xml
SLICE (used by the s-option) should not exceed the (pixel) height of raw images.
<nexus_file>
/dls/i13/data/2014/mt9377-2/raw/29384.nxs 

<output_directory>
/dls/i13/data/2014/mt9377-2/processing/reconstruction/29384/ 
If this output directory doesn't already exist, it will automatically be created.



tomo-recon


The tomo-recon script expects a certain number of arguments and provides some additional options to choose from. When executed, it first creates an appropriate Linux environment and then invokes a Python script called full_recon.py. All the input arguments and options, supplied for running tomo-recon, are passed in to full_recon.py. As usual, a more detailed description of all those arguments and options can be viewed by executing:

Linux command
tomo-recon --help

or

Linux command
tomo-recon -h


At the time of writing this section (18 Sep 2017), either of the above two commands outputs the following description:

tomo-recon -h

     Welcome to the DLS compute cluster

     For MPI jobs, please use 'module load openmpi'.

     If using a different OpenMPI installation,
     or manually specifying path to OpenMPI, option
     '-mca orte_forward_job_control'
     must be added to mpirun to ensure cluster functionality.

     Please report any issues to linux.manager@diamond.ac.uk

Loading 64-bit Oracle instantclient, version 11.2
Loading 64-bit python, version 2.7.2
Loading 64-bit numpy, version 1.6.1
Starting Full Recon
Usage: full_recon.py [options] data output_directory

Options:
  -h, --help            show this help message and exit
  -m MACHINES, --machines=MACHINES
                        Number of machines to deploy to
  -b SLICE_BEGIN, --slice_begin=SLICE_BEGIN
                        Start Slice number
  -e SLICE_END, --slice_end=SLICE_END
                        End Slice Number
  -t TEMPLATE, --template=TEMPLATE
                        Template XML file
  -w WSAMP, --width_sample=WSAMP
                        Set the subsampling of the sinograms width
  -l LSAMP, --length_sample=LSAMP
                        Set the subsampling of the sinograms length
  -c CENTRE, --centre=CENTRE
                        Set the centre of rotation
  -r RUN_SLICES, --run_slices_loc=RUN_SLICES
                        set the run_slices.sh location
  -n, --new_cluster     use the new cluster
  -p, --preview         Run a preview reconstruction
  -a, --angles          Use angular information to reconstruct, do not use
                        with a ROI
  -o, --old_cluster     Use the old cluster
  --dark_file=DARK_FILE
                        Path to the file containing dark images
  --dark_path=DARK_PATH
                        path in the dark file to the data
  --flat_file=FLAT_FILE
                        Path to the file containing flat images
  --flat_path=FLAT_PATH
                        path in the dark file to the data
  --recon_range=RECON_RANGE
                        range for the reconstruction to be done over
  --d0f1                If option included (True), use 0&#39;s for dark- and 1&#39;s
                        for flat-field images
  --scan_id             If option included (True), incorporate the ID of the
                        input Nexus scan file into output filenames



Option SwitchOption's Default ValueComment(s)
-m
2

-b
0

-e
128

-t
/dls_sw/apps/tomopy/tomopy/src/settings.xml
1. Absolute path needs to be supplied (if different from default). 
2. Equivalent to --template.
-w
1

-l
1

-r
/dls_sw/apps/tomopy/tomopy/bin/run_slices.sh
Path to a tomography reconstruction script to be used (advanced users).
-n
False

-p
False

-a
False

-o
False

--dark_file
None (Python keyword)
1. Path to the Nexus file containing dark-field images.
2. Do not specify this path if dark-field images are in the same Nexus file as projections.
--dark_path
/entry1/pco1_hw_hdf/data
1. To be supplied only if --dark_file option is used (see --dark_file). 
2. Nexus path to dark-field data residing in the file specified by --dark_file, eg /entry1/instrument/pco1_hw_hdf_nochunking/data. If in doubt, use Dawn or hdfview to verify it.
3. Data must contain only dark-field images (if image key is present, it is ignored).
4. These external dark-field images override any internal dark-field images that may be stored alongside projections in <nexus_file>.
--flat_file
None (Python keyword)
1. Path to the Nexus file containing flat-field images.
2. Do not specify this path if flat-field images are in the same Nexus file as projections.
--flat_path
/entry1/pco1_hw_hdf/data
1. To be supplied only if --flat_file option is used (see --flat_file).
2. Nexus path to flat-field data residing in the file specified by --flat_file, eg /entry1/instrument/pco1_hw_hdf_nochunking/data. If in doubt, use Dawn or hdfview to verify it.
3. Data must contain only flat-field images (if image key is present, it is ignored).
4. These external flat-field images override any internal flat-field images that may be stored alongside projections in <nexus_file>.
--recon_range
-1
If default is used, angular range is automatically determined from rotation-angle data stored in the input Nexus file, ie <nexus_file>. 
This option is particularly useful for limited-angle reconstruction. 
--d0f1
False (implicit)
If this option is included on the command line, then a synthetic dark-field image consisting of all 0's and another synthetic flat-field image of all 1's are used during 
reconstruction. This work-around enables one to reconstruct datasets that have been dark-and-flat-field-corrected beforehand.
--scan_id
False (implicit)
If this option is included on the command line, then the scan ID is automatically included in the output filenames, eg for Nexus scan file 91809.nxs, a typical output filename 
would be recon_91809_127850_02032.tif (as opposed to recon_127850_02032.tif).


BASIC EXAMPLE


In most cases, basic use of tomo-recon is adequate.

Linux command
tomo-recon -m 20 -b 0 -e 2159 -c 1271.1 /dls/i13/data/2014/mt9377-2/raw/30119.nxs /dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/


In the above basic example, the command structure is as follows:

Usage
tomo-recon [options] <nexus_file> <output_directory>
Command Argument(s)Example Command Argument(s)Comment(s)
[options]
-m 20 
-b 0 -e 2159
-c 1271.1

SLICE_END (used by the e-option) should not exceed the (pixel) height of raw images.

<nexus_file>
/dls/i13/data/2014/mt9377-2/raw/30119.nxs 

<output_directory>
/dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/
This output directory must exist before executing tomo-recon.


MOST EXTENSIVE EXAMPLE


In more complicated cases,  tomo-recon needs to be executed with  all or an appropriate selection of additional input arguments.

Linux command
tomo-recon -m 20 -b 0 -e 2159 -c 1271.1 --dark_file=/dls/i13/data/2014/mt9377-2/raw/29375.nxs --dark_path=/entry1/pco1_sw/data --flat_file=/dls/i13/data/2014/mt9377-2/raw/30535.nxs --flat_path=/entry1/pco1_sw/data --recon_range=180.0 --template=/dls/i13/data/2014/mt9377-2/processing/drain1/settingsHDF.xml /dls/i13/data/2014/mt9377-2/raw/30119.nxs /dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/


In the above most extensive example, the command structure is as follows:

Usage
tomo-recon [options] <nexus_file> <output_directory>
Command Argument(s)Example Command Argument(s)Comment(s)
[options]
-m 20 
-b 0 -e 2159
-c 1271.1
--dark_file=/dls/i13/data/2014/mt9377-2/raw/29375.nxs --dark_path=/entry1/pco1_sw/data
--flat_file=/dls/i13/data/2014/mt9377-2/raw/30535.nxs --flat_path=/entry1/pco1_sw/data
--recon_range=180.0
--template=/dls/i13/data/2014/mt9377-2/processing/drain1/settingsHDF.xml
SLICE_END (used by the e-option) should not exceed the (pixel) height of raw images.
<nexus_file>
/dls/i13/data/2014/mt9377-2/raw/30119.nxs 

<output_directory>
/dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/
This output directory must exist before executing tomo-recon.



tomo-fix


The tomo-fix script expects a certain number of arguments and provides some additional options to choose from. When executed, it first creates an appropriate Linux environment and then invokes a Python script called tomo-fix.py. All the input arguments and options, supplied for running tomo-fix, are passed in to tomo-fix.py. As mentioned at the top of this page, this script is automatically executed whenever tomo-recon is run. As usual, a more detailed description of all those arguments and options can be viewed by executing:

Linux command
tomo-fix --help

or

Linux command
tomo-fix -h


At the time of writing this section (18 Sep 2017), either of the above two commands outputs the following description:

tomo-fix -h

Loading 64-bit Oracle instantclient, version 11.2
Loading 64-bit python, version 2.7.2
Loading 64-bit numpy, version 1.6.1
Usage: tomo-fix.py [options] nexus_file directory_to_check

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -m MACHINES, --machines=MACHINES
                        Number of machines to deploy to
  -b SLICE_BEGIN, --slice_begin=SLICE_BEGIN
                        Start Slice number
  -e SLICE_END, --slice_end=SLICE_END
                        End Slice Number
  -t TEMPLATE, --template=TEMPLATE
                        Template XML file
  -w WSAMP, --width_sample=WSAMP
                        Set the subsampling of the sinograms width
  -l LSAMP, --length_sample=LSAMP
                        Set the subsampling of the sinograms length
  -c CENTRE, --centre=CENTRE
                        Set the centre of rotation
  -r RUN_SLICES, --run_slices_loc=RUN_SLICES
                        set the run_slices.sh location
  -n, --new_cluster     use the new cluster
  -p, --preview         Run a preview reconstruction
  -a, --angles          Use angular information to reconstruct, do not use
                        with a ROI
  -o, --old_cluster     Use the old cluster
  --dark_file=DARK_FILE
                        Path to the file containing dark images
  --dark_path=DARK_PATH
                        path in the dark file to the data
  --flat_file=FLAT_FILE
                        Path to the file containing flat images
  --flat_path=FLAT_PATH
                        path in the dark file to the data
  --recon_range=RECON_RANGE
                        range for the reconstruction to be done over
  --d0f1                If option included (True), use 0's for dark- and 1's
                        for flat-field images
  --scan_id             If option included (True), incorporate the ID of the
                        input Nexus scan file into output filenames
~>



Option SwitchOption's Default ValueComment(s)
-m
2

-b
0

-e
128

-t
/dls_sw/apps/tomopy/tomopy/src/settings.xml
1. Absolute path needs to be supplied (if different from default).
2. Equivalent to --template.
-w
1

-l
1

-r
/dls_sw/apps/tomopy/tomopy/bin/run_slices.sh
Path to a tomography reconstruction script to be used (advanced users).
-n
False

-p
False

-a
False

-o
False

--dark_file
n/a
1. Path to the Nexus file containing dark-field images.
2. Do not specify this path if dark-field images are in the same Nexus file as projections.
--dark_path
n/a
1.To be supplied only if --dark_file option is used (see --dark_file).
2. Nexus path to dark-field data residing in the file specified by --dark_file, eg /entry1/instrument/pco1_hw_hdf_nochunking/data. If in doubt, use Dawn or hdfview to verify it.
2. Data must contain only dark-field images (if image key is present, it is ignored).
3. These external dark-field images override any internal dark-field images that may be stored alongside projections in <nexus_file>.
--flat_file
n/a
1. Path to the Nexus file containing flat-field images.
2. Do not specify this path if flat-field images are in the same Nexus file as projections.
--flat_path
n/a
1. To be supplied only if --flat_file option is used (see --flat_file).
2. Nexus path to flat-field data residing in the file specified by --flat_file, eg /entry1/instrument/pco1_hw_hdf_nochunking/data. If in doubt, use Dawn or hdfview to verify it.
3. Data must contain only flat-field images (if image key is present, it is ignored).
4. These external flat-field images override any internal flat-field images that may be stored alongside projections in <nexus_file>.
--d0f1
False (implicit)
If this option is included on the command line, then a synthetic dark-field image consisting of all 0's and another synthetic flat-field image of all 1's are used during 
reconstruction. This work-around enables one to reconstruct datasets that have been dark-and-flat-field-corrected beforehand.
--scan_id
False (implicit)
If this option is included on the command line, then the scan ID is automatically included in the output filenames, eg for Nexus scan file 91809.nxs, a typical output filename 
would be recon_91809_127850_02032.tif (as opposed to recon_127850_02032.tif).


BASIC EXAMPLE


In most cases, basic use of tomo-fix is adequate.

Linux command
tomo-fix -m 20 -b 0 -e 2159 -c 1271.1 /dls/i13/data/2014/mt9377-2/raw/30119.nxs /dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/


In the above basic example, the command structure is as follows:

Usage
tomo-fix [options] <nexus_file> <directory_to_check>
Command Argument(s)Example Command Argument(s)Comment(s)
[options]
-m 20 
-b 0 -e 2159
-c 1271.1
SLICE_END (used by the e-option) should not exceed the (pixel) height of raw images.
<nexus_file>
/dls/i13/data/2014/mt9377-2/raw/30119.nxs 

<directory_to_check>
/dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/
This output directory must exist before executing tomo-recon.

MOST EXTENSIVE EXAMPLE


In more complicated cases, tomo-fix needs to be executed with  all or an appropriate selection of additional input arguments.

Linux command
tomo-fix -m 20 -b 0 -e 2159 -c 1271.1 --dark_file=/dls/i13/data/2014/mt9377-2/raw/29375.nxs --dark_path=/entry1/pco1_sw/data --flat_file=/dls/i13/data/2014/mt9377-2/raw/30535.nxs --flat_path=/entry1/pco1_sw/data --recon_range=180.0 --template=/dls/i13/data/2014/mt9377-2/processing/drain1/settingsHDF.xml /dls/i13/data/2014/mt9377-2/raw/30119.nxs /dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/


In the above most extensive example, the command structure is as follows:

Usage
tomo-fix [options] <nexus_file> <directory_to_check>
Command Argument(s)Example Command Argument(s)Comment(s)
[options]
-m 20 
-b 0 -e 2159
-c 1271.1
--dark_file=/dls/i13/data/2014/mt9377-2/raw/29375.nxs --dark_path=/entry1/pco1_sw/data
--flat_file=/dls/i13/data/2014/mt9377-2/raw/30535.nxs --flat_path=/entry1/pco1_sw/data
--recon_range=180.0
--template=/dls/i13/data/2014/mt9377-2/processing/drain1/settingsHDF.xml

SLICE_END (used by the e-option) should not exceed the (pixel) height of raw images.

<nexus_file>
/dls/i13/data/2014/mt9377-2/raw/30119.nxs 

<directory_to_check>
/dls/i13/data/2014/mt9377-2/processing/reconstruction/30119/
This output directory must exist before executing tomo-recon.



Appendix A: Ring-artefact suppression

Ring-artefact suppression can be performed as part of the reconstruction process with the help of an appropriately configured settings.xml. Note that the default /dls_sw/apps/tomopy/tomopy/src/settings.xml has ring-artefact suppression enabled with the following parameters: 

/dls_sw/apps/tomopy/tomopy/src/settings.xml (part)
...
<RingArtefacts>
   <Type info="No, Column, AML">AML</Type>
   <ParameterN>0.000</ParameterN>
   <ParameterR>0.0050</ParameterR>
   <NumSeries info="1 - default, if greater, then nonlinear">1.0</NumSeries>
</RingArtefacts>
...

To optimise ring-artefact suppression, you need to first modify the ParameterR or NumSeries parameters in your own copy of the default /dls_sw/apps/tomopy/tomopy/src/settings.xml and then explicitly specify a path to this modified copy to run tomo-recon (use the t-option) in order to reconstruct a few representative slices for subsequent visual inspection (this workflow is similar to that employed for finding an optimal value of CoR). ParameterR controls the strength of ring-artefact suppression, and it is best to change it in order-of-magnitude steps, e. g. 0.5, 0.05, 0.005 (default), 0.0005. Note that the smaller ParameterR, the more aggressive ring-artefact suppression. NumSeries is used for controlling the high-aspect-ratio compensation (for plate-like samples). 

If desired, ring-suppression can be completely switched off by changing Type (from the default value of AML) to No:

...
<RingArtefacts>
   <Type info="No, Column, AML">No</Type>
   <ParameterN>0.000</ParameterN>
   <ParameterR>0.0050</ParameterR>
   <NumSeries info="1 - default, if greater, then nonlinear">1.0</NumSeries>
</RingArtefacts>
...


Appendix B: Optional bit-depth reduction of output TIFFs

Bit-depth reduction can be performed as part of the reconstruction process with the help of an appropriately configured settings.xml:

/dls_sw/apps/tomopy/tomopy/src/settings.xml (part)
...
<OutputData>
  ...
  <BitsType info="Input, Given">Given</BitsType>
  <Bits>32</Bits>
  <Restrictions info="Yes, No">No</Restrictions>
  <ValueMin>0.5</ValueMin>
  <ValueMax>1.1</ValueMax>
  ...
</OutputData>
...

The above snippet of XML shows the default (32-bit) settings (in the default /dls_sw/apps/tomopy/tomopy/src/settings.xml file). Before modifying these settings (in your own copy of the default /dls_sw/apps/tomopy/tomopy/src/settings.xml), it is crucial to find some sensible replacement values for the above ValueMin and ValueMax. Since these values are image-dependent, it is best to select them with the help of a histogram (e.g. in ImageJ) of your typical, 32-bit reconstructed slice.  

Alternatively, bit-depth reduction can be done after the reconstruction process, as described in

Extraction of TIFF images from image data in the HDF format (with optional bit-depth reduction) and related matters

Appendix C: Batch reconstruction

It is sometimes desirable to reconstruct a batch of similar tomography scans. The Bash script presented below provides a simple example of how to do it for a sequence of consecutive scans sharing the same centre of rotation. More precisely, this script reconstructs a batch of 4 consecutive scans, numbered from 10000 to 10003, using the same centre of rotation, 1234.5, and the same flat- and dark-field datasets, stored in 2 separate Nexus files (one for flats and the other for darks). The values of 0 and 2159 for the b and, respectively, the e option have been chosen to match the full size of images produced by PCO Edge.

Please note that, to prevent the DLS compute cluster from getting overloaded, the script reconstructs scans one after another (sequentially) as opposed to reconstructing all scans in parallel at the same time.   


Batch Example 1
#! /bin/bash

echo "Running batch reconstruction..."

### USER INPUT: START
# What is your visit's ID and year?
visitID="xx12345-6"
year=2016

# What is the COMMON centre of rotation for all the scans?
CENTRE=1234.5

# What is the first scan number?
START=10000

# What is the last scan number?
END=10003

# Which directory contains the scans (with sample projections)?
RAWDIR="/dls/i13/data/${year}/${visitID}/raw"

# Which directory contains the flat scan?
FLATDIR="/dls/i13/data/${year}/${visitID}/raw"

# What is the flat-scan number?
FLAT=11111

# Which directory contains the dark scan?
DARKDIR="/dls/i13/data/${year}/${visitID}/raw"

# What is the dark-scan number?
DARK=66666

# Which directory is to be used for saving reconstructed images in scan-numbered sub-directories?
PRODIR="/dls/i13/data/${year}/${visitID}/processing/reconstruction"
### USER INPUT: END

module add tomography

echo "BATCH START = ${START}"
echo "BATCH END = ${END}"

for (( i=$START; i<=$END; i++ ))

do
    cd $PRODIR
    echo "in directory: "`pwd`
    echo "making reconstruction directory for scan: $i"
    [ -d $i ] || mkdir $i
    echo "reconstructing scan: $i"
    echo "tomo-recon command is:"
    # the next two lines print out the command before it is executed (in the 3rd line), which is useful for testing, etc
    echo \
    tomo-recon -m 20 -b 0 -e 2159 -c $CENTRE --dark_file=$DARKDIR/$DARK.nxs --dark_path=/entry1/pco1_hw_hdf_nochunking/data --flat_file=$FLATDIR/$FLAT.nxs --flat_path=/entry1/pco1_hw_hdf_nochunking/data $RAWDIR/$i.nxs $PRODIR/$i/ --recon_range=180.0 --template=/dls/i13/data/2014/visitID/processing/settingsHDF.xml
    tomo-recon -m 20 -b 0 -e 2159 -c $CENTRE --dark_file=$DARKDIR/$DARK.nxs --dark_path=/entry1/pco1_hw_hdf_nochunking/data --flat_file=$FLATDIR/$FLAT.nxs --flat_path=/entry1/pco1_hw_hdf_nochunking/data $RAWDIR/$i.nxs $PRODIR/$i/ --recon_range=180.0 --template=/dls/i13/data/2014/visitID/processing/settingsHDF.xml

    echo "finished reconstructing scan: $i"
done

echo "Finished batch reconstruction."


If your batch contains scans which are not consecutively numbered or if they do not share the common centre of rotation, then the script presented below can be used to reconstruct such a batch. More precisely, this script reconstructs a batch of 2 non-consecutive scans, numbered 10000 and 10003, using 2 different centres of rotation (1234.5 and 5432.1, respectively), and the same flat- and dark-field datasets, stored in 2 separate Nexus files (one for flats and the other for darks):


Batch Example 2
#! /bin/bash

echo "Running batch reconstruction..."

# Declare associative array
declare -A arr

### USER INPUT: START
# What is your visit's ID and year?
visitID="xx12345-6"
year=2016

# What are your scan numbers and the corresponding centres of rotations?
# TEMPLATE: arr[scan_number]=centre_of_rotation
arr[10000]=1234.5
arr[10003]=5432.1

# Which directory contains the scans (with sample projections)?
RAWDIR="/dls/i13/data/${year}/${visitID}/raw"

# Which directory contains the flat scan?
FLATDIR="/dls/i13/data/${year}/${visitID}/raw"

# What is the flat-scan number?
FLAT=11111

# Which directory contains the dark scan?
DARKDIR="/dls/i13/data/${year}/${visitID}/raw"

# What is the dark-scan number?
DARK=66666

# Which directory is to be used for saving reconstructed images in scan-numbered sub-directories?
PRODIR="/dls/i13/data/${year}/${visitID}/processing/reconstruction"
### USER INPUT: END

module add tomography
echo "BATCH:"
item=1
for i in ${!arr[@]]}
do
   echo ${item}. scan=${i} CoR=${arr[${i}]}
   item=$((item+1))
done

for i in ${!arr[@]]}

do
    cd $PRODIR
    echo "in directory: "`pwd`
    echo "making reconstruction directory for scan: $i"
    [ -d $i ] || mkdir $i
    echo "reconstructing scan: $i"
    echo "tomo-recon command is:"
    # the next two lines print out the command before it is executed (in the 3rd line), which is useful for testing, etc
    echo \
    tomo-recon -m 20 -b 0 -e 2159 -c ${arr[${i}]} --dark_file=$DARKDIR/$DARK.nxs --dark_path=/entry1/pco1_hw_hdf_nochunking/data --flat_file=$FLATDIR/$FLAT.nxs --flat_path=/entry1/pco1_hw_hdf_nochunking/data $RAWDIR/$i.nxs $PRODIR/$i/ --recon_range=180.0 --template=/dls/i13/data/2014/visitID/processing/settingsHDF.xml
    tomo-recon -m 20 -b 0 -e 2159 -c ${arr[${i}]} --dark_file=$DARKDIR/$DARK.nxs --dark_path=/entry1/pco1_hw_hdf_nochunking/data --flat_file=$FLATDIR/$FLAT.nxs --flat_path=/entry1/pco1_hw_hdf_nochunking/data $RAWDIR/$i.nxs $PRODIR/$i/ --recon_range=180.0 --template=/dls/i13/data/2014/visitID/processing/settingsHDF.xml

    echo "finished reconstructing scan: $i"
done

# Clear the array
unset arr

echo "Finished batch reconstruction."


To use either of these two example scripts for batch processing, please follow these steps:

  • copy-and-paste the above code into your own .sh file (e.g. mybatch.sh);
  • modify the script in this file to suit your particular needs, making sure that all input arguments match the size of your images and that all input paths to files or directories point to some existing and accessible objects (the script automatically creates the output directory, if it does not exist);  
  • save the file;
  • make the file executable (e.g. chmod u+x mybatch.sh);
  • execute the script (e.g. ./mybatch.sh)

It is always a good practice to first test your new script without running any jobs on the compute cluster. This can be accomplished by commenting out (c.f. the leading #) the last occurrence of the two identical lines beginning with 'tomo-recon', i.e.        

Batch testing
echo \
tomo-recon -m 20 -b 0 -e 2159 -c $CENTRE --dark_file=$DARKDIR/$DARK.nxs --dark_path=/entry1/pco1_hw_hdf_nochunking/data --flat_file=$FLATDIR/$FLAT.nxs --flat_path=/entry1/pco1_hw_hdf_nochunking/data $RAWDIR/$i.nxs $PRODIR/$i/ --recon_range=180.0 --template=/dls/i13/data/2014/visitID/processing/settingsHDF.xml
#tomo-recon -m 20 -b 0 -e 2159 -c $CENTRE --dark_file=$DARKDIR/$DARK.nxs --dark_path=/entry1/pco1_hw_hdf_nochunking/data --flat_file=$FLATDIR/$FLAT.nxs --flat_path=/entry1/pco1_hw_hdf_nochunking/data $RAWDIR/$i.nxs $PRODIR/$i/ --recon_range=180.0 --template=/dls/i13/data/2014/visitID/processing/settingsHDF.xml


Please note that the first occurrence of this line simply prints out the intended tomo-recon command with its expanded arguments (which is useful for spotting any typos, etc), whereas the second occurrence would normally execute this command for real, sending a number of jobs to run on the compute cluster. This dry run should help one verify that all input arguments are correct and that all input paths to files or directories point to some existing and accessible objects on the file system (the script automatically creates the output directory, if the latter does not already exist). If no problems are detected with the script, then the commented-out line needs, of course, to be commented back in before the script can be executed for real.  


Appendix D: Troubleshooting

If the reconstruction scripts do not produce any output files, experienced users can find the logs at the following location:

log file directories
/dls/tmp/tomopy/tomo-centre/
/dls/tmp/tomopy/tomo-recon/

For finding the latest log files, sort the contents and open the logfile for example with gedit:

[user@machine tomo-centre]$ ls -lthr
-rw-r--r--. 1 fedid123 fedid123    0 Feb 13 15:43 run_slices.sh.po22535829
-rw-r--r--. 1 fedid123 fedid123    0 Feb 13 15:43 run_slices.sh.pe22535829
-rw-r--r--. 1 fedid123 fedid123  599 Feb 13 15:43 run_slices.sh.e22535827
-rw-r--r--. 1 fedid123 fedid123  599 Feb 13 15:43 run_slices.sh.e22535828
-rw-r--r--. 1 fedid123 fedid123  599 Feb 13 15:43 run_slices.sh.e22535829
-rw-r--r--. 1 fedid123 fedid123  599 Feb 13 15:43 run_slices.sh.e22535826
-rw-r--r--. 1 fedid123 fedid123 4.1K Feb 13 15:43 run_slices.sh.o22535829
-rw-r--r--. 1 fedid123 fedid123 4.1K Feb 13 15:43 run_slices.sh.o22535828
-rw-r--r--. 1 fedid123 fedid123 4.1K Feb 13 15:43 run_slices.sh.o22535826
-rw-r--r--. 1 fedid123 fedid123 4.1K Feb 13 15:43 run_slices.sh.o22535827
[user@machine tomo-centre] gedit  run_slices.sh.o22535827