How to develop a Savu plugin

The architecture of the savu package is that new plugins can be easily developed without having to take the framework into consideration. This is mostly true for simple cases, however there are some things which cannot be done with such simple methodologies and the developer may need to take more into consideration.

Every plugin in Savu needs to be a subclass of the Plugin class, however there are several convenience subclasses which already exist for doing standard processes such as filtering, reconstruction and augmentation/passthrough

Although there are many plugins defined in the core savu code, plugins can be written anywhere and included easily as shown below without having to be submitted to the core code.

Required Files

To create a plugin for Savu you will need to create two modules:

  1. A plugin module, named plugin_name.py containing a class PluginName

    This file contains your plugin definitions and statements.

  2. A plugin tools module named plugin_name_tools.py, containing a class PluginNameTools

    This file contains the parameter details and citations in a yaml format. You can assign each parameter a data type, a description, a visibility level and a default value.

Note

A module is a file containing python definitions and statements.

Note

PluginName should be replaced by the name of your plugin without any spaces. The words should be capitalised. plugin_name should be replaced by the name of your plugin in lowercase, seperated by underscores.

Examples are:

    1. A plugin module astra_recon_cpu.py containing a class AstraReconCpu

    2. A plugin tools module astra_recon_cpu_tools.py containing a class AstraReconCpuTools

    1. A plugin module remove_all_rings.py containing a class RemoveAllRings

    2. A plugin tools module remove_all_rings_tools.py containing a class RemoveAllRingsTools

The plugin file and the plugin tools file should both be stored at the same directory inside ../Savu/savu/plugins/.

1. Introduction to creating a Plugin

A docstring is a piece of text contained by three quotation marks “”” <docstring_text> “””. In the beginning docstring, write your plugin name and a short sentence to describe what it can do. This will later be shown to the user.

Docstring template:

"""
 .. module:: <plugin_name>
    :platform: Unix
    :synopsis: <plugin_description>
 .. moduleauthor:: <author_name>

 """

Docstring example:

"""
 .. module:: no_process
    :platform: Unix
    :synopsis: Plugin to test loading and saving without processing
 .. moduleauthor:: Nicola Wadeson <scientificsoftware@diamond.ac.uk>

 """

Next, import any classes which you may need. Most plugins would require the Plugin class and the function decorator register_plugin.

Import example:

from savu.plugins.plugin import Plugin
from savu.plugins.utils import register_plugin
from savu.plugins.driver.cpu_plugin import CpuPlugin

Initialise the class template:

@register_plugin
class PluginName(ParentPlugin):
    def __init__(self):
        super(PluginName, self).__init__("PluginName")

Below is the template plugin class.

# Copyright 2014 Diamond Light Source Ltd.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
.. module:: plugin_name_example
   :platform: Unix
   :synopsis: (Change this) A template to create a simple plugin that takes 
    one dataset as input and returns a similar dataset as output.

.. moduleauthor:: (Change this) Developer Name <email@address.ac.uk>
"""
from savu.plugins.utils import register_plugin
from savu.plugins.plugin import Plugin
# Import any additional libraries or base plugins here.

# This decorator is required for the configurator to recognise the plugin
@register_plugin
class PluginNameExample(Plugin):
# Each class must inherit from the Plugin class and a driver

    def __init__(self):
        super(PluginNameExample, self).__init__("PluginNameExample")

    def pre_process(self):
        """ This method is called immediately after base_pre_process(). """
        pass  

    def process_frames(self, data):
        """
        This method is called after the plugin has been created by the
        pipeline framework and forms the main processing step

        :param data: A list of numpy arrays for each input dataset.
        :type data: list(np.array)
        """
        pass

    def post_process(self):
        """
        This method is called after the process function in the pipeline
        framework as a post-processing step. All processes will have finished
        performing the main processing at this stage.

        :param exp: An experiment object, holding input and output datasets
        :type exp: experiment class instance
        """
        pass

    def setup(self):
        """
        Initial setup of all datasets required as input and output to the
        plugin.  This method is called before the process method in the plugin
        chain.
        """
        in_dataset, out_dataset = self.get_datasets()
  

You can download it here. An extended version is available here.

All template downloads are available here: Plugin templates.

Below is an example of the entire NoProcess plugin class.

Plugin Class example:

"""
    .. module:: no_process
       :platform: Unix
       :synopsis: Plugin to test loading and saving without processing
    .. moduleauthor:: Nicola Wadeson <scientificsoftware@diamond.ac.uk>

"""
from savu.plugins.plugin import Plugin
from savu.plugins.utils import register_plugin
from savu.plugins.driver.cpu_plugin import CpuPlugin

@register_plugin
class NoProcess(Plugin, CpuPlugin):
    def __init__(self):
        super(NoProcess, self).__init__("NoProcess")

    def process_frames(self, data):
        return data[0]

    def setup(self):
        """
        Initial setup of all datasets required as input and output to the
        plugin.  This method is called before the process method in the plugin
        chain.
        """
        in_dataset, out_dataset = self.get_datasets()
        out_dataset[0].create_dataset(in_dataset[0])
        in_pData, out_pData = self.get_plugin_datasets()

        if self.parameters['pattern']:
            pattern = self.parameters['pattern']
        else:
            pattern = in_dataset[0].get_data_patterns().keys()[0]

        in_pData[0].plugin_data_setup(pattern, self.get_max_frames())
        out_pData[0].plugin_data_setup(pattern, self.get_max_frames())

    def get_max_frames(self):
        return 'multiple'

    def nInput_datasets(self):
        return 1

    def nOutput_datasets(self):
        return 1

2. How to create the tools class and documentation

This tools class holds the parameter details in a yaml format. There is advice on this format here. To begin, import the PluginTools class.

from savu.plugins.plugin_tools import PluginTools

Beneath the class definition, write a docstring with a sentence to describe in further detail what your plugin does.

class NoProcessTools(PluginTools):
    """The base class from which all plugins should inherit.
    """

Inside the method define_parameters you should write a docstring which will contain the parameter details. The text should be in a yaml format.

An example of a plugin tools class.

from savu.plugins.plugin_tools import PluginTools

class NoProcessTools(PluginTools):
    """The base class from which all plugins should inherit.
    """
    def define_parameters(self):
        """
        pattern:
            visibility: basic
            dtype: [None, str]
            description: Explicitly state the slicing pattern.
            default: None
        other:
            visibility: advanced
            dtype: int
            description: Temporary parameter for testing.
            default: 10
        yaml_file:
            visibility: advanced
            dtype: yamlfilepath
            description: Yaml file path.
            default: savu/plugins/loaders/full_field_loaders/nxtomo_loader.yaml

        """

Yaml Text

pattern:
    visibility: advanced
    dtype: [None,str]
    description: Explicitly state the slicing pattern.
    default: None
other:
    visibility: advanced
    dtype: int
    description: Temporary parameter for testing.
    default: 10
yaml_file:
    visibility: advanced
    dtype: yamlfilepath
    description: Yaml file path.
    default: savu/plugins/loaders/full_field_loaders/nxtomo_loader.yaml

You should list the names of parameters required. After each name you need a colon. Then you include an indent and put four pieces of information: visibility, dtype, description, default. The indentation level should be consistent.

The parameter information included should be:

  • visibility - There are five visibility levels. The level helps the user to identify which parameters must be changed on every savu run and which need to be changed less frequently.

  • dtype - The data type of the parameter value

  • description - A description of the parameter

  • default - A default value

Visibility

You should choose one of five options.

  • basic - A basic parameter will need to be adjusted with each use of the plugin and will be on display to all users as default.

  • intermediate - An intermediate parameter can be used to tailor the plugin result more carefully.

  • advanced - Advanced parameters should only need to be changed occasionally by developers.

  • datasets - This is used for the in_datasets and out_datasets parameter only.

  • hidden - Hidden parameters are not editable from the user interface. This may be useful during plugin development.

Dtype

Choose the required data type. This is used to check if the parameter input is valid.

Basic types

The basic data types are included in the table below:

Data Type

Description

int

An integer

str

A string

float

A float

bool

A boolean

filepath

A file path

h5path

An hdf5 path

yamlfilepath

A yaml file path

dir

A directory

nptype

A numpy type

preview

Preview slice list

list

A list

dict

A dictionary

List combinations

You can create a custom list data type containing the basic types by using the following syntax.

  • list[]

Inside the encasing brackets, you should state the required type.

By including one required type inside the brackets (eg. list[int]) the list will be expected to contain any number of values of that type. By including two or more items within the brackets, you can specify the number of entries inside the list (eg. list[int,int,int]) and the list will be expected to contain that number of entries.

List syntax

Description

list

A list of any length containing a combination of basic types

list[btype]

A list of any length containing only the btype provided

list[btype, btype, …, btype]

A list of btype of fixed length = N = number of btype

list[]

An empty list

Where btype stands for a type from the basic type table. The btype can be the same type, or of different types.

For example:

Data type example

Description

list[int]

A list of integers of any length

list[string, string]

A list containing two strings

list[list[string, float], int]

A list containing one list of one string and one float and then one separate integer value

list[filepath, h5path, int]

A list containing a filepath, a hdf5path and an integer value

list[int, int, int]

A list containing three integer values

Lists containing multiple data types

List syntax

Description

list

A list of any length containing a combination of basic types

list[ [btype(1), btype(2)] ]

A list of any length containing btype(1) or btype(2) entries

Where btype stands for a type from the basic type table.

Note

None may be used as an additional data type in this case.

For example:

Data type example

Description

list[[str, int]]

A list of any length, containing strings or integers

list[[[int, str, float], None]]

A list that either contains a trio of [int, str, float] values or the keyword None

Dictionaries

You can create a custom dictionary data type containing the data type options by using the following syntax.

  • dict{}

Inside the enclosing brackets, you should state the required type. Only a subset of basic types are available as dictionary ‘key’ types.

Dictionary syntax

Description

dict

A dictionary of any length containing a combination of basic types

dict{btype(1): btype(2)}

A dictionary of keys of btype(1) with values of btype(2)

dict{}

An empty dictionary

Where btype stands for a type from the basic type table. The btype can be the same, or of different types.

Note

None may be used as an additional data type in this case.

For example:

Data type example

Description

dict{int: float}

A dictionary of integer keys and float values

Options list

If more than one data type is allowed, then include these in a list format. Each data type should be separated by a comma.

Note

None may be used as an additional data type in this case.

Examples:

Data type example

Description

[int, float]

integers or floats are valid

[list[], list[int]]

An empty list or a list containing integers

[None, list[str]]

None or a list containing strings

[list[float], float]

A list containing floats or a float value

[int, None]

An integer or None

[list[string], list[]]

A list of strings of any length or an empty list

In the table below are some more specific data type examples:

Data Type

Description

list[int]

A list of integers

list[str]

A list of strings

list[float]

A list of numbers

list[int, int]

A list containing two integer values

list[filepath, h5path, int]

A sequence of items. The first item in the list should be a file path, the next should be an interior file path, the last item should be an integer. [<file path>, <hdf5 interior path>, <integer>]

list[h5path, int]

A sequence of items. The first item in the list should be an interior file path, the last item should be an integer. [<hdf5 interior path> , <integer>]

dict{int: float}

A dictionary holding ineteger keys and float values. {int:float}

Description

Any string of text

Default

The default value of the parameter. For example, False, 0, 0.01. This default value must adhere to the correct data type, as specified by dtype.

iterations:
    default: 0.01

This can be filled in with further details if this parameter is reliant on another. For example, depending on the current value for the regularisation method, the default value for ‘iterations’ will change. When the method selected is ROF_TV, then the iterations value should be 1000.

iterations:
    default:
        regularisation_method:
            ROF_TV: 1000
            FGP_TV: 500
            PD_TV: 100
            SB_TV: 100
            LLT_ROF: 1000
            NDF: 1000
            DIFF4th: 1000

Optional fields:

Additional fields whaich may be included:

  • dependency

  • options

and additional fields which should be included within the description key are:

  • summary

  • verbose

  • range

  • options

Dependency

If the parameter is only applicable to certain other parameter choices you can make it only appear when a certain value is chosen.

For example, the NDF penalty should only be displayed when the regularisation method is NDF.

NDF_penalty:
    dependency:
        regularisation_method: NDF

Options

If you have a closed list of options for your parameter, you should enter them here. The case will matter.

regularisation_method:
    options: [ROF_TV, FGP_TV, PD_TV, SB_TV, LLT_ROF, NDF, Diff4th]

When you add options, you can include more fields within the description. The indentation level should be consistent.

NDF_penalty:
    description:
       summary: Penalty dtype
       verbose: Nonlinear/Linear Diffusion model (NDF) specific penalty
         type.
       options:
         Huber: Huber
         Perona: Perona-Malik model
         Tukey: Tukey

Description

If you are giving a more detailed description, then you can extend the description key. Summary, verbose, range, and options descriptions can be included here.

Summary

The summary holds a short description of your parameter.

data_Huber_thresh:
    description:
        summary: Threshold parameter for Huber data fidelity.
Verbose

Verbose is for a more in depth explanation of your parameter.

data_Huber_thresh:
    description:
        summary: Threshold parameter for Huber data fidelity.
        verbose: Parameter which controls the level of suppression
         of outliers in the data
Range

If you have a value which must be within a certain range, describe this range here.

Range is a descriptor key at the moment, and it does not perform a validation check.

iteration:
    description:
        range: Between 10 and 100

Citation Text

The citations are included inside the method define_citations. You should write a docstring which will contain the citation details. The text should be in a yaml format.

from savu.plugins.plugin_tools import PluginTools

class NoProcessTools(PluginTools):
    """The base class from which all plugins should inherit.
    """
    def define_parameters(self):
        """
        """
    def citation(self):
        """
        Short description
        bibtex:
                Bibtex block of text.
        endnote:
                Endnote block of text.
        doi: doi link
        """
    def citation2(self):
        """
        Short description
        bibtex:
                Bibtex block of text.
        endnote:
                Endnote block of text.
        doi: doi link
        """

Below is a longer example of the yaml text.

The CCPi-Regularisation toolkit provides a set of
variational regularisers (denoisers) which can be embedded in
a plug-and-play fashion into proximal splitting methods for
image reconstruction. CCPi-RGL comes with algorithms that can
satisfy various prior expectations of the reconstructed object,
for example being piecewise-constant or piecewise-smooth nature.
bibtex:
        @article{kazantsev2019ccpi,
        title={Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms},
        author={Kazantsev, Daniil and Pasca, Edoardo and Turner, Martin J and Withers, Philip J},
        journal={SoftwareX},
        volume={9},
        pages={317--323},
        year={2019},
        publisher={Elsevier}
        }
endnote:
        %0 Journal Article
        %T Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms
        %A Kazantsev, Daniil
        %A Pasca, Edoardo
        %A Turner, Martin J
        %A Withers, Philip J
        %J SoftwareX
        %V 9
        %P 317-323
        %@ 2352-7110
        %D 2019
        %I Elsevier
short_name_article: ccpi regularisation toolkit for CT
doi: "10.1016/j.softx.2019.04.003"

Description

This is a string describing the citation.

description: A string to describe the citation

Bibtex

The bibtex text.

bibtex:
        @article{kazantsev2019ccpi,
        title={Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms},
        author={Kazantsev, Daniil and Pasca, Edoardo and Turner, Martin J and Withers, Philip J},
        journal={SoftwareX},
        volume={9},
        pages={317--323},
        year={2019},
        publisher={Elsevier}
        }

Endnote

The endnote text.

endnote:
        @article{kazantsev2019ccpi,
        title={Ccpi-regularisation toolkit for computed tomographic image reconstruction with proximal splitting algorithms},
        author={Kazantsev, Daniil and Pasca, Edoardo and Turner, Martin J and Withers, Philip J},
        journal={SoftwareX},
        volume={9},
        pages={317--323},
        year={2019},
        publisher={Elsevier}
        }

Digital Object Identifier

The DOI identifies a journal or article. It should remain fixed over the lifetime of the journal. It is a character string divided into two parts, a prefix and a suffix, separated by a slash.

doi: "10.1016/0167-2789(92)90242-F"

Dependency

If the citation is only applicable to certain other parameter choices you can make it only appear when a certain value is chosen.

For example, the ROF_TV citation should only be displayed when the method is ROF_TV.

dependency:
    method: ROF_TV

Document your plugin in restructured text

If you are creating your plugin with the ‘savu_plugin_generator’ command, then the restructured text file will be created automatically for you and the link to this file will be printed to the terminal window.

You will need to open the linked file and write down instructions about how to use your plugin. The language this file should be written in is reStructured text. This is described in more detail here.

An example would be:

.. ::process_list:: test_data/process_lists/vo_centering_process.nxs

The Plugin Module
###################

A text description including different steps to follow when using the plugin.

* Item 1
* Item 2
* Item 3

This is an example image:

.. figure:: plugins/corrections/tomo_68067_00000.png
  :figwidth: 50 %
  :align: center
  :figclass: align-center

Equations can be included as:

.. math::

    \frac{1}{2}\pi

A section heading
===================

Section content.

Example piece of code
=======================

>>> add plugin_module
>>> mod 1.1 'New value'
>>> mod 1.5 4.5
>>> mod 1.8 True

Process List
=============

If you would like to load and use a process list within your example
code, you can specify the file path at the start of this rst file.

The included process list file will be checked for every sphinx documentation
build when the directive '.. ::process_list:: <file_path>' is included at
the top of your rst file.

For example '.. ::process_list:: test_data/process_lists/vo_centering_process.nxs'

Plugin Documentation

Below is a list of the current plugins grouped by type. You may also use the search bar on the left to find a specific one.

3. How to create a test

Testing something else in line with text example:

In order to submit a new plugin to Savu on Github, you MUST provide a test for your new plugin. To create a test follow the steps below:

  1. Choose a test template

  2. Choose a test dataset

  3. Amend the parameters r1,…,r8 in the file.

  4. Save the file.

  5. Add the file to your local repository.

Test templates

If your plugin is not dependent on any others (i.e. it can be run on its own on raw or corrected data), then download the sample test WITHOUT a process list. This will test the plugin with default parameters.

If your plugin is dependent on other plugins, you will need to create a process list create a process list link and download the sample test WITH a process list.

Test data

List of test data available. What to do if you require different test data. You can submit a new test dataset to Savu, with the requirement that it is less than 10MB in size.

Amending the parameters

See the real test modules:
  1. median_filter_test.py tests the median_filter_plugin.py plugin WITH NO PROCESS LIST.

  2. median_filter_test.py tests the median_filter_plugin.py plugin WITH NO PROCESS LIST.

Save the file as “your_module_name.py”

Warning

Ensure the test file name has the same name as the module name (r1)

Note

Have a look at the real test for the median_filter_plugin.py module.