Savu - notes on basic use¶
Note: The following tutorial for I13 users is out-of-date.
- General advice
- Suggested processes to include in a process list for tomography reconstruction
- Examples
- Example 1: Simple process list including VoCentering
- Example 2: More advanced process list including VoCentering
- Example 3: Simple process list for generating sinograms
- Example 4a: Simple process list for generating dark-and-flat-field-corrected radiography images
- Example 4b: Simple process list for generating dark-and-flat-field-corrected tomography raw images
- Example 5: Process list for overriding internal flat- and dark-field images with external ones
- Example 6: Simple process list for applying lens-distortion correction to grating-interferometry TIFF images
- Example 7: Process list for reconstructing a series of identical tomography scans (stored sequentially in a single 3d dataset)
- Example 8: Simple process list for reconstructing a flat-and-dark-field-corrected dataset that was previously generated in Savu
- Example 9: Simple process list for reconstructing an off-centre scan taken over the range of 360 deg
- Appendices
General advice |
---|
To make Savu available in a Linux session, execute:
module load savu
To make Savu Configurator available in a Linux session, execute:
module load savu savu_config
Any savu_mpi jobs are run on the new compute cluster, named hamilton (after Margaret Hamilton), and can be monitored by executing:
module load hamilton qstat
(note lower-case hamilton).
Suggested processes to include in a process list for tomography reconstruction |
---|
A typical Savu process list in DLS may contain all or most of the following processes in its linear chain of operations (ordered in a similar way to that indicated below):
Process position | Process name | Process category | Brief description of process | Comment(s) | Common alternative or more suitable process(es)* |
---|---|---|---|---|---|
1 | NxtomoLoader | loader | To specify the location of raw dataset(s) to be used as input. |
|
|
2 | Dezinger | filter | To remove zingers from raw data. |
|
|
3 | DarkFlatFieldCorrection | corrector | To apply the standard dark-and-flat-field normalisation to sample projections. |
| |
4 | DistortionCorrection | corrector | To correct data for lens distortion. |
| |
5 | CcpiRingArtefactFilter | filter | To suppress ring artefacts. | CcpiRingArtefactFilter is positioned before PaganinFilter as it tends to be efficient at suppressing ring artefacts found in typical datasets, but appears to be somewhat less successful on diffused images (which can occasionally be produced by PaganinFilter). |
|
6 | VoCentering | filter | To find an optimal value of CoR automatically. | It may be prudent to test the outcome of VoCentering on a small, representative subset of data (use the preview parameter) before embarking on the reconstruction of the entire dataset, using the automatically-determined value of CoR. | VoCenteringIterative |
7 | PaganinFilter | filter | To retrieve the phase-contrast information. |
| FresnelFilter |
8 | RavenFilter | filter | To suppress ring artefacts. | RavenFilter may complementarily be used after PaganinFilter as it can lead to an additional reduction of ring artefacts in somewhat diffused images that can occasionally be produced by PaganinFilter. | RingRemovalWaveletfft |
9 | AstraReconGpu | reconstructor | To reconstruct normalised and conditioned data. | ||
10 | Hdf5saver | saver | To save reconstruction(s) to output file(s). | By default, Hdf5saver is the final process in every process list (hence there is normally no need to add it explicitly). | TiffSaver |
* Note that, when a process is replaced in a given process list by an alternative process, the latter may need to be included at a different position to that of the original process. For example, DezingerSinogram needs be included after DarkFlatFieldCorrection while Dezinger needs to be included before it.
Examples |
---|
Appendices |
---|
Appendix A: Interoperability of the parameter-tuning and previewing operations |
---|
Since each instance of parameter tuning in Savu adds an extra dimension to the input dataset, it is often desirable to use the previewing (subset-selection) mechanism to reduce the workload of any subsequent resource-demanding processes, e. g. VoCentering or AstraReconGpu. This can be accomplished by setting the preview parameter of VoCentering or, respectively, that of AstraReconGpu to a desirable rank-3 slice of the incoming higher-order dataset, generated by one or more parameter-tuning operations invoked by the preceding processes (see, e.g., Example 2).
Appendix B: Brief reference guide to specifying values for the parameter-tuning operations |
---|
Note that not every parameter in Savu is tunable. For any tunable parameter, one can input its tuning values using the methods described below:
Task | Syntax | Comment(s) | Example(s) |
---|---|---|---|
To input an ordered list of n individual values. | val_1; val_2;... val_n |
| mod 6.7 1000.8;1010.8;1020.8;1030.8;1040.8 mod 6.centre_of_rotation 1000.8;1010.8;1020.8;1030.8;1040.8 mod 6.4 FBP_CUDA;CGLS_CUDA mod 6.algorithm FBP_CUDA;CGLS_CUDA |
To input an arithmetic sequence of all numbers starting from val_from and not exceeding val_to, calculated using the common difference of cmn_diff. | val_from:val_to:cmn_diff; |
| mod 6.7 1000.8:1040.8:10.0; mod 6.centre_of_rotation 1000.8:1040.8:10.0; mod 6.centre_of_rotation 1020.8-2*10.0:1020.8+2*10.0:10.0; |
Appendix C: Batch processing |
---|
Batch processing in Savu can be done with the help of the savu-batch command, whose arguments are ordered in a logically identical way to that employed by the savu_mpi command, the only difference being the first argument, which is used to supply a path to a user-prepared batch file:
savu_mpi <SCAN_DATA_FILE> <SAVU_PROCESS_LIST_FILE> <OUT_DIR> savu-batch <BATCH_PARAM_FILE> <SCAN_DATA_DIR> <SAVU_PROCESS_LIST_DIR> <OUT_DIR>
To make the savu-batch command available in Linux terminal, execute 'module load python/ana' and 'module load tomography':
Linux Command: Savu-batch
~>module load python/ana
~>module load tomography
~>savu-batch
Usage: savu-batch <BATCH_PARAM_FILE> <SCAN_DATA_DIR> <SAVU_PROCESS_LIST_DIR> <OUT_DIR> [-d <INTERMED_DIR>] [-e EMAIL_ADDRESS_1[;EMAIL_ADDRESS_2;... EMAIL_ADDRESS_N]] [-t <TIME_OUT_MINUTES>] [-v]
Run a batch of reconstructions using Savu.
-h display this help and exit.
-d INTERMED_DIR save all intermediate files in INTERMED_DIR directory.
-e EMAILS send e-mail notification(s) to (semicolon-separated) address(es) in EMAILS.
-t TIMEOUT_MINUTES monitor each cluster job no longer than TIMEOUT_MINUTES.
-v be verbose: send e-mail notifications on completion of each cluster job (in addition to sending a summary notification on completion of the entire batch).
~>
The first argument of savu-batch, <BATCH_PARAM_FILE>, is a path to comma-separated values (CSV) file, containing a table of desired batch parameters, e.g.
batch.csv
#<scan number>,<proces list>[,[CoR(s)],[output dir]] # commented-out lines are ignored (and so are any comments on the RHS of valid lines)
102923, 4x/spl_f102921_d102922.nxs, 1197.75, 4x/%s
102974, 4x/spl_f102921_d102922.nxs, 1255.50, 4x/%s # same process-list file as in line 2, used with different CoR to reconstruct a different scan
102923, 4x/spl_f102921_d102922_pci.nxs, 1197.75, 4x/%s/pci # same scan as in line 2, reconstructed with phase-contrast retrieval
102990, 4x/spl_f102984_d102985.nxs, 1254.6:1255.9:0.1;, preview/4x/%s # CoR tuning in dedicated sub-directory
102986, 4x/spl_f102988_d102989.nxs, 1256.50, 4x/%s
103038, 10x/spl_f103036_d103037.nxs,, 10x/%s # CoR intentionally NOT provided (CoR found in <SAVU_PROCESS_LIST_DIR>/10x/spl_f103036_d103037.nxs is used as is)
103050, 10x/spl_f103048_d103049.nxs, 919.50, 10x/%s
103050, 10x/spl_f103048_d103049_lensdistortcrop40.nxs, 919.50-40, 10x/%s # Savu-interpretable mathematical expression for adjusting CoR by 40 pixels due to the 40-pixel cropping in DistortionCorrection
104022, spl_f103048_d103049.nxs, 1110.25, tests # scan-numbered output sub-directory intentionally NOT provided (output goes to <OUT_DIR>/tests)
103052, spl_f103048_d103049.nxs, 910.50, # output sub-directory intentionally NOT provided (output goes to <OUT_DIR>)
Notes:
- savu-batch reconstructs scans using the savu_mpi command (on the cluster).
- savu-batch reconstructs scans sequentially one-by-one, monitoring and waiting (for the duration of <TIMEOUT_MINUTES>) for each reconstruction job to complete (on the cluster) before launching the next one.
- savu-batch reconstructs scans in the same order as that specified in <BATCH_PARAM_FILE>.
- savu-batch makes its own, internal copies of all input process-list files, specified in <BATCH_PARAM_FILE>, at the time of executing the savu-batch command, and then, if required, each of these copies is appropriately edited just before the savu_mpi command is executed on each of them.
- The values of CoR specified in <BATCH_PARAM_FILE> are used as they are to set the centre_of_rotation parameter of any (active) reconstructor process found in the corresponding Savu process-list file (hence the user is responsible for making any adjustments needed to account for any additional cropping requested by, for example, the crop_edges parameter of DistortionCorrection, etc.).
- If no value of CoR is specified for some item in <BATCH_PARAM_FILE>, then the value of the centre_of_rotation parameter found (at the time of launching savu-batch) in the corresponding Savu process-list file is used.
- savu-batch is able to handle process lists that don't include any processes requiring CoR (in which case an empty space, or any numerical value, can be provided as nothing depends on it).
Example:
Linux command: savu-batch
>>> savu-batch /dls/i13/data/2018/mt12345-6/processing/process_lists/batch/batch.csv /dls/i13/data/2018/mt12345-6/raw /dls/i13/data/2018/mt12345-6/processing/process_lists /dls/i13/data/2018/mt12345-6/processing/reconstruction -d /dls/i13/data/2018/mt12345-6/tmp -e "whoever@wherever"
In the above savu-batch example, the arguments are as follows:
Argument | Argument Value | Comment(s) |
---|---|---|
<BATCH_PARAM_FILE> | /dls/i13/data/2018/mt12345-6/processing/process_lists/batch/batch.csv | |
<SCAN_DATA_DIR> | /dls/i13/data/2018/mt12345-6/raw | Contains all Nexus scan files specified in <BATCH_PARAM_FILE>, e.g. /dls/i13/data/2018/mt12345-6/raw/102923.nxs, etc. |
<SAVU_PROCESS_LIST_DIR> | /dls/i13/data/2018/mt12345-6/processing/process_lists |
|
<OUT_DIR> | /dls/i13/data/2018/mt12345-6/processing/reconstruction |
|
<INTERMED_DIR> | /dls/i13/data/2018/mt12345-6/tmp |
Additional notes:
- When preparing batch files, please use a Linux text editor as files created under Windows are not compatible.
- If you intend to run multiple batches, please wait for each individual batch to finish before submitting another one.
- When supplying an e-mail address, please bear in kind that some mail boxes may automatically block machine-generated e-mail notifications.