How to develop a Savu plugin¶
The architecture of the savu package is that new plugins can be easily developed without having to take the framework into consideration. This is mostly true for simple cases, however there are some things which cannot be done with such simple methodologies and the developer may need to take more into consideration.
Every plugin in Savu needs to be a subclass of the Plugin class, however there are several convenience subclasses which already exist for doing standard processes such as filtering, reconstruction and augmentation/passthrough
Although there are many plugins defined in the core savu code, plugins can be written anywhere and included easily as shown below without having to be submitted to the core code.
Required Files¶
To create a plugin for Savu you will need to create two modules:
A plugin module, named plugin_name.py containing a class PluginName
This file contains your plugin definitions and statements.
A plugin tools module named plugin_name_tools.py, containing a class PluginNameTools
This file contains the parameter details and citations in a yaml format. You can assign each parameter a data type, a description, a visibility level and a default value.
Note
A module is a file containing python definitions and statements.
Note
PluginName should be replaced by the name of your plugin without any spaces. The words should be capitalised. plugin_name should be replaced by the name of your plugin in lowercase, seperated by underscores.
Examples are:
A plugin module astra_recon_cpu.py containing a class AstraReconCpu
A plugin tools module astra_recon_cpu_tools.py containing a class AstraReconCpuTools
A plugin module remove_all_rings.py containing a class RemoveAllRings
A plugin tools module remove_all_rings_tools.py containing a class RemoveAllRingsTools
The plugin file and the plugin tools file should both be stored at the same directory inside ../Savu/savu/plugins/.
1. Introduction to creating a Plugin¶
A docstring is a piece of text contained by three quotation marks “”” <docstring_text> “””. In the beginning docstring, write your plugin name and a short sentence to describe what it can do. This will later be shown to the user.
Docstring template:
"""
.. module:: <plugin_name>
:platform: Unix
:synopsis: <plugin_description>
.. moduleauthor:: <author_name>
"""
Docstring example:
"""
.. module:: no_process
:platform: Unix
:synopsis: Plugin to test loading and saving without processing
.. moduleauthor:: Nicola Wadeson <scientificsoftware@diamond.ac.uk>
"""
Next, import any classes which you may need. Most plugins would require the Plugin class and the function decorator register_plugin.
Import example:
from savu.plugins.plugin import Plugin
from savu.plugins.utils import register_plugin
from savu.plugins.driver.cpu_plugin import CpuPlugin
Initialise the class template:
@register_plugin
class PluginName(ParentPlugin):
def __init__(self):
super(PluginName, self).__init__("PluginName")
Below is the template plugin class.
# Copyright 2014 Diamond Light Source Ltd.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
.. module:: plugin_name_example
:platform: Unix
:synopsis: (Change this) A template to create a simple plugin that takes
one dataset as input and returns a similar dataset as output.
.. moduleauthor:: (Change this) Developer Name <email@address.ac.uk>
"""
from savu.plugins.utils import register_plugin
from savu.plugins.plugin import Plugin
# Import any additional libraries or base plugins here.
# This decorator is required for the configurator to recognise the plugin
@register_plugin
class PluginNameExample(Plugin):
# Each class must inherit from the Plugin class and a driver
def __init__(self):
super(PluginNameExample, self).__init__("PluginNameExample")
def pre_process(self):
""" This method is called immediately after base_pre_process(). """
pass
def process_frames(self, data):
"""
This method is called after the plugin has been created by the
pipeline framework and forms the main processing step
:param data: A list of numpy arrays for each input dataset.
:type data: list(np.array)
"""
pass
def post_process(self):
"""
This method is called after the process function in the pipeline
framework as a post-processing step. All processes will have finished
performing the main processing at this stage.
:param exp: An experiment object, holding input and output datasets
:type exp: experiment class instance
"""
pass
def setup(self):
"""
Initial setup of all datasets required as input and output to the
plugin. This method is called before the process method in the plugin
chain.
"""
in_dataset, out_dataset = self.get_datasets()
You can download it here
.
An extended version is available here
.
All template downloads are available here: Plugin templates.
Below is an example of the entire NoProcess plugin class.
Plugin Class example:
"""
.. module:: no_process
:platform: Unix
:synopsis: Plugin to test loading and saving without processing
.. moduleauthor:: Nicola Wadeson <scientificsoftware@diamond.ac.uk>
"""
from savu.plugins.plugin import Plugin
from savu.plugins.utils import register_plugin
from savu.plugins.driver.cpu_plugin import CpuPlugin
@register_plugin
class NoProcess(Plugin, CpuPlugin):
def __init__(self):
super(NoProcess, self).__init__("NoProcess")
def process_frames(self, data):
return data[0]
def setup(self):
"""
Initial setup of all datasets required as input and output to the
plugin. This method is called before the process method in the plugin
chain.
"""
in_dataset, out_dataset = self.get_datasets()
out_dataset[0].create_dataset(in_dataset[0])
in_pData, out_pData = self.get_plugin_datasets()
if self.parameters['pattern']:
pattern = self.parameters['pattern']
else:
pattern = in_dataset[0].get_data_patterns().keys()[0]
in_pData[0].plugin_data_setup(pattern, self.get_max_frames())
out_pData[0].plugin_data_setup(pattern, self.get_max_frames())
def get_max_frames(self):
return 'multiple'
def nInput_datasets(self):
return 1
def nOutput_datasets(self):
return 1
2. How to create the tools class and documentation¶
This tools class holds the parameter details in a yaml format. There is advice on this format here. To begin, import the PluginTools class.
from savu.plugins.plugin_tools import PluginTools
Beneath the class definition, write a docstring with a sentence to describe in further detail what your plugin does.
class NoProcessTools(PluginTools):
"""The base class from which all plugins should inherit.
"""
Inside the method define_parameters you should write a docstring which will contain the parameter details. The text should be in a yaml format.
An example of a plugin tools class.
from savu.plugins.plugin_tools import PluginTools
class NoProcessTools(PluginTools):
"""The base class from which all plugins should inherit.
"""
def define_parameters(self):
"""
pattern:
visibility: basic
dtype: [None, str]
description: Explicitly state the slicing pattern.
default: None
other:
visibility: advanced
dtype: int
description: Temporary parameter for testing.
default: 10
yaml_file:
visibility: advanced
dtype: yamlfilepath
description: Yaml file path.
default: savu/plugins/loaders/full_field_loaders/nxtomo_loader.yaml
"""
Yaml Text¶
pattern:
visibility: advanced
dtype: [None,str]
description: Explicitly state the slicing pattern.
default: None
other:
visibility: advanced
dtype: int
description: Temporary parameter for testing.
default: 10
yaml_file:
visibility: advanced
dtype: yamlfilepath
description: Yaml file path.
default: savu/plugins/loaders/full_field_loaders/nxtomo_loader.yaml
You should list the names of parameters required. After each name you need a colon. Then you include an indent and put four pieces of information: visibility, dtype, description, default. The indentation level should be consistent.
The parameter information included should be:
visibility - There are five visibility levels. The level helps the user to identify which parameters must be changed on every savu run and which need to be changed less frequently.
dtype - The data type of the parameter value
description - A description of the parameter
default - A default value
Visibility¶
You should choose one of five options.
basic - A basic parameter will need to be adjusted with each use of the plugin and will be on display to all users as default.
intermediate - An intermediate parameter can be used to tailor the plugin result more carefully.
advanced - Advanced parameters should only need to be changed occasionally by developers.
datasets - This is used for the in_datasets and out_datasets parameter only.
hidden - Hidden parameters are not editable from the user interface. This may be useful during plugin development.
Dtype¶
Choose the required data type. This is used to check if the parameter input is valid.
Basic types¶
The basic data types are included in the table below:
Data Type |
Description |
---|---|
int |
An integer |
str |
A string |
float |
A float |
bool |
A boolean |
filepath |
A file path |
h5path |
An hdf5 path |
yamlfilepath |
A yaml file path |
dir |
A directory |
nptype |
A numpy type |
preview |
Preview slice list |
list |
A list |
dict |
A dictionary |
List combinations¶
You can create a custom list data type containing the basic types by using the following syntax.
list[]
Inside the encasing brackets, you should state the required type.
By including one required type inside the brackets (eg. list[int]) the list will be expected to contain any number of values of that type. By including two or more items within the brackets, you can specify the number of entries inside the list (eg. list[int,int,int]) and the list will be expected to contain that number of entries.
List syntax |
Description |
---|---|
list |
A list of any length containing a combination of basic types |
list[btype] |
A list of any length containing only the btype provided |
list[btype, btype, …, btype] |
A list of btype of fixed length = N = number of btype |
list[] |
An empty list |
Where btype stands for a type from the basic type table. The btype can be the same type, or of different types.
For example:
Data type example |
Description |
---|---|
list[int] |
A list of integers of any length |
list[string, string] |
A list containing two strings |
list[list[string, float], int] |
A list containing one list of one string and one float and then one separate integer value |
list[filepath, h5path, int] |
A list containing a filepath, a hdf5path and an integer value |
list[int, int, int] |
A list containing three integer values |
Lists containing multiple data types¶
List syntax |
Description |
---|---|
list |
A list of any length containing a combination of basic types |
list[ [btype(1), btype(2)] ] |
A list of any length containing btype(1) or btype(2) entries |
Where btype stands for a type from the basic type table.
Note
None may be used as an additional data type in this case.
For example:
Data type example |
Description |
---|---|
list[[str, int]] |
A list of any length, containing strings or integers |
list[[[int, str, float], None]] |
A list that either contains a trio of [int, str, float] values or the keyword None |
Dictionaries¶
You can create a custom dictionary data type containing the data type options by using the following syntax.
dict{}
Inside the enclosing brackets, you should state the required type. Only a subset of basic types are available as dictionary ‘key’ types.
Dictionary syntax |
Description |
---|---|
dict |
A dictionary of any length containing a combination of basic types |
dict{btype(1): btype(2)} |
A dictionary of keys of btype(1) with values of btype(2) |
dict{} |
An empty dictionary |
Where btype stands for a type from the basic type table. The btype can be the same, or of different types.
Note
None may be used as an additional data type in this case.
For example:
Data type example |
Description |
---|---|
dict{int: float} |
A dictionary of integer keys and float values |
Options list¶
If more than one data type is allowed, then include these in a list format. Each data type should be separated by a comma.
Note
None may be used as an additional data type in this case.
Examples:
Data type example |
Description |
---|---|
[int, float] |
integers or floats are valid |
[list[], list[int]] |
An empty list or a list containing integers |
[None, list[str]] |
None or a list containing strings |
[list[float], float] |
A list containing floats or a float value |
[int, None] |
An integer or None |
[list[string], list[]] |
A list of strings of any length or an empty list |
In the table below are some more specific data type examples:
Data Type |
Description |
---|---|
list[int] |
A list of integers |
list[str] |
A list of strings |
list[float] |
A list of numbers |
list[int, int] |
A list containing two integer values |
list[filepath, h5path, int] |
A sequence of items. The first item in the list should be a file path, the next should be an interior file path, the last item should be an integer. [<file path>, <hdf5 interior path>, <integer>] |
list[h5path, int] |
A sequence of items. The first item in the list should be an interior file path, the last item should be an integer. [<hdf5 interior path> , <integer>] |
dict{int: float} |
A dictionary holding ineteger keys and float values. {int:float} |
Description¶
Any string of text
Default¶
The default value of the parameter. For example, False, 0, 0.01. This default value must adhere to the correct data type, as specified by dtype.
iterations:
default: 0.01
This can be filled in with further details if this parameter is reliant on another. For example, depending on the current value for the regularisation method, the default value for ‘iterations’ will change. When the method selected is ROF_TV, then the iterations value should be 1000.
iterations:
default:
regularisation_method:
ROF_TV: 1000
FGP_TV: 500
PD_TV: 100
SB_TV: 100
LLT_ROF: 1000
NDF: 1000
DIFF4th: 1000
Optional fields:¶
Additional fields whaich may be included:
dependency
options
and additional fields which should be included within the description key are:
summary
verbose
range
options
Dependency¶
If the parameter is only applicable to certain other parameter choices you can make it only appear when a certain value is chosen.
For example, the NDF penalty should only be displayed when the regularisation method is NDF.
NDF_penalty:
dependency:
regularisation_method: NDF
Options¶
If you have a closed list of options for your parameter, you should enter them here. The case will matter.
regularisation_method:
options: [ROF_TV, FGP_TV, PD_TV, SB_TV, LLT_ROF, NDF, Diff4th]
When you add options, you can include more fields within the description. The indentation level should be consistent.
NDF_penalty:
description:
summary: Penalty dtype
verbose: Nonlinear/Linear Diffusion model (NDF) specific penalty
type.
options:
Huber: Huber
Perona: Perona-Malik model
Tukey: Tukey
Description¶
If you are giving a more detailed description, then you can extend the description key. Summary, verbose, range, and options descriptions can be included here.
Summary¶
The summary holds a short description of your parameter.
data_Huber_thresh:
description:
summary: Threshold parameter for Huber data fidelity.
Verbose¶
Verbose is for a more in depth explanation of your parameter.
data_Huber_thresh:
description:
summary: Threshold parameter for Huber data fidelity.
verbose: Parameter which controls the level of suppression
of outliers in the data
Range¶
If you have a value which must be within a certain range, describe this range here.
Range is a descriptor key at the moment, and it does not perform a validation check.
iteration:
description:
range: Between 10 and 100
Citation Text¶
The citations are included inside the method define_citations. You should write a docstring which will contain the citation details. The text should be in a yaml format.
from savu.plugins.plugin_tools import PluginTools
class NoProcessTools(PluginTools):
"""The base class from which all plugins should inherit.
"""
def define_parameters(self):
"""
"""
def citation(self):
"""
Short description
bibtex:
Bibtex block of text.
endnote:
Endnote block of text.
doi: doi link
"""
def citation2(self):
"""
Short description
bibtex:
Bibtex block of text.
endnote:
Endnote block of text.
doi: doi link
"""
Below is a longer example of the yaml text.
The CCPi-Regularisation toolkit provides a set of
variational regularisers (denoisers) which can be embedded in
a plug-and-play fashion into proximal splitting methods for
image reconstruction. CCPi-RGL comes with algorithms that can
satisfy various prior expectations of the reconstructed object,
for example being piecewise-constant or piecewise-smooth nature.
bibtex:
@article{kazantsev2019ccpi,
title={Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms},
author={Kazantsev, Daniil and Pasca, Edoardo and Turner, Martin J and Withers, Philip J},
journal={SoftwareX},
volume={9},
pages={317--323},
year={2019},
publisher={Elsevier}
}
endnote:
%0 Journal Article
%T Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms
%A Kazantsev, Daniil
%A Pasca, Edoardo
%A Turner, Martin J
%A Withers, Philip J
%J SoftwareX
%V 9
%P 317-323
%@ 2352-7110
%D 2019
%I Elsevier
short_name_article: ccpi regularisation toolkit for CT
doi: "10.1016/j.softx.2019.04.003"
Description¶
This is a string describing the citation.
description: A string to describe the citation
Bibtex¶
The bibtex text.
bibtex:
@article{kazantsev2019ccpi,
title={Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms},
author={Kazantsev, Daniil and Pasca, Edoardo and Turner, Martin J and Withers, Philip J},
journal={SoftwareX},
volume={9},
pages={317--323},
year={2019},
publisher={Elsevier}
}
Endnote¶
The endnote text.
endnote:
@article{kazantsev2019ccpi,
title={Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms},
author={Kazantsev, Daniil and Pasca, Edoardo and Turner, Martin J and Withers, Philip J},
journal={SoftwareX},
volume={9},
pages={317--323},
year={2019},
publisher={Elsevier}
}
Digital Object Identifier¶
The DOI identifies a journal or article. It should remain fixed over the lifetime of the journal. It is a character string divided into two parts, a prefix and a suffix, separated by a slash.
doi: "10.1016/0167-2789(92)90242-F"
Dependency¶
If the citation is only applicable to certain other parameter choices you can make it only appear when a certain value is chosen.
For example, the ROF_TV citation should only be displayed when the method is ROF_TV.
dependency:
method: ROF_TV
Document your plugin in restructured text¶
If you are creating your plugin with the ‘savu_plugin_generator’ command, then the restructured text file will be created automatically for you and the link to this file will be printed to the terminal window.
You will need to open the linked file and write down instructions about how to use your plugin. The language this file should be written in is reStructured text. This is described in more detail here.
An example would be:
.. ::process_list:: test_data/process_lists/vo_centering_process.nxs
The Plugin Module
###################
A text description including different steps to follow when using the plugin.
* Item 1
* Item 2
* Item 3
This is an example image:
.. figure:: plugins/corrections/tomo_68067_00000.png
:figwidth: 50 %
:align: center
:figclass: align-center
Equations can be included as:
.. math::
\frac{1}{2}\pi
A section heading
===================
Section content.
Example piece of code
=======================
>>> add plugin_module
>>> mod 1.1 'New value'
>>> mod 1.5 4.5
>>> mod 1.8 True
Process List
=============
If you would like to load and use a process list within your example
code, you can specify the file path at the start of this rst file.
The included process list file will be checked for every sphinx documentation
build when the directive '.. ::process_list:: <file_path>' is included at
the top of your rst file.
For example '.. ::process_list:: test_data/process_lists/vo_centering_process.nxs'
Plugin Documentation¶
Below is a list of the current plugins grouped by type. You may also use the search bar on the left to find a specific one.
3. How to create a test¶
Testing something else in line with text example:
In order to submit a new plugin to Savu on Github, you MUST provide a test for your new plugin. To create a test follow the steps below:
Choose a test template
Choose a test dataset
Amend the parameters r1,…,r8 in the file.
Save the file.
Add the file to your local repository.
Test templates¶
If your plugin is not dependent on any others (i.e. it can be run on its own on raw or corrected data), then
download the sample test WITHOUT a process list
. This will test
the plugin with default parameters.
If your plugin is dependent on other plugins, you will need to create a process list create a process list link
and download the sample test WITH a process list
.
Test data¶
List of test data available. What to do if you require different test data. You can submit a new test dataset to Savu, with the requirement that it is less than 10MB in size.
Amending the parameters¶
- See the real test modules:
median_filter_test.py
tests the median_filter_plugin.py plugin WITH NO PROCESS LIST.median_filter_test.py
tests the median_filter_plugin.py plugin WITH NO PROCESS LIST.
Save the file as “your_module_name.py”
Warning
Ensure the test file name has the same name as the module name (r1)
Note
Have a look at the real test
for the median_filter_plugin.py module.