Savu Developer Information

Developing in locally installed Savu (Lite)

Once Savu-lite has been installed into conda environment you can test the changes you make by simply reinstalling conda in your environment by running:

>>> python setup.py install

This will reinstall Savu with your changes in your conda location, e.g. here: “miniconda3/envs/savu/lib/python3.7/site-packages/savu-4.0-py3.7.egg”. One can also work directly in conda environment to see the immediate changes or for debugging purposes.

Release Strategy

When a new release is required the following process should be done. Make a new branch for convinience if fixes are needed, it should be called vX.Y-release i.e v0.1-release. After the branch is made, move to it and then create a release tag for it called vX.Y.Z i.e v0.1.0, where Z starts at 0, and future minor release verisons can be made with v0.1.1 etc later on down the branch.

Once this is done, Zenodo.org will automatically create and archive an artefact.

How to develop on a feature branch

Make a new Branch for development and move to it

git checkout -b new_branch

Make modifications and commit as normal

git commit -m “test commit”

push this branch up to github

git push –set-upstream origin new_branch

get all the latest data from github on all branches

git fetch

to keep up to date, merge recent changes from master into the development branch and fix issues if there are any

git merge origin/master

Continue working with the branch untill you are happy with the new feature, merge master into it as shown before and fix up issues, then merge the branch into master

move to master

git checkout master

update

git pull

merge in the new branch

git merge new_branch

there should be no problems if you have merge master in first, so just push back up

git push

How to test a new plugin using DAWN

DAWN can be downloaded from http://www.dawnsci.org/ and general user tutorials are found at https://www.youtube.com/user/DAWNScience

Using the Debug perspective, create a new test, e.g. “tomopy_recon_test.py” to test your plugin in “/Savu/savu/test/”, in this case the “example_filter_back_projection.py” plugin for reconstructing data, setting the self.plugin_name appropriately. After saving the file, right-click on it in the PyDev Package Explorer window and Run As a Python unit-test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# -*- coding: utf-8 -*-
# Copyright 2014 Diamond Light Source Ltd.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


"""
.. module:: tomopy_recon_test
   :platform: Unix
   :synopsis: unittest for the tomopy 'gridrec' reconstruction

.. moduleauthor:: Nicola Wadeson <scientificsoftware@diamond.ac.uk>

"""

import unittest
import savu.test.test_utils as tu
from savu.test.travis.framework_tests.plugin_runner_test import \
        run_protected_plugin_runner

class TomopyReconTest(unittest.TestCase):
    def setUp(self):
        self.data_file = 'tomo_standard.nxs'
        self.experiment = 'tomo'

    def test_tomopy_gridrec(self):
        process_list = 'reconstruction/tomopy_test.nxs'
        options = tu.initialise_options(self.data_file, self.experiment, process_list)
        run_protected_plugin_runner(options)
        tu.cleanup(options)

if __name__ == "__main__":
    unittest.main()

This runs a series of tests and produces an output file with the result of the plugin, whether it be a filter or a reconstruction, allowing for visualisation of the data, providing a check of whether the process has worked successfully.

The output file is saved in a tmp directory as a .h5 file, e.g. “/tmp/tmp32bexK.h5”. This can be viewed in DAWN.

Adding C/C++ extensions to a plugin

There are numerous ways to create python bindings to external C/C++ libraries, which may be useful to recycle existing code or to improve performance. Two different approaches have currently been tested: Cython (to link to external C code) and Boost.Python (to link to external C++ code). Cython is essentially python with C-types and requires a C-API, a python wrapper and a makefile, whilst Boost.Python is a wrapper for the Python/C API and requires a wrapper and a makefile. By building the makefile a shared library (*.so) file is created and can be added to the \lib directory in the Savu framework and imported as a python module.

Cython Example

http://docs.cython.org/src/tutorial/clibraries.html

  1. A C interface: A *.pxd file, which is similar to a C header file, providing C function definitions required in the python code. For example, cdexing.pxd:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
cdef extern from "./options.h":
   ctypedef struct Options:
      unsigned char versionflag
      unsigned char f_call_num
      size_t cropwd
      unsigned int nlines
      float outlier_mu
      unsigned char returnflag
      unsigned int npad

cdef extern from "./timestamp.h":
   void timestamp_open(const char * const logname)
   void timestamp_close()
   void timestamp_init()
   void timestamp(const char * const stampmsg)

cdef extern from "./dezing_functions.h":
   void runDezing(Options * ctrlp, unsigned int  thisbatch,unsigned char * inbuf, unsigned char * outbuf )

  1. A python wrapper: A *.pyx file that must have a different name to the *.pyd file above. For example, dezing.pyx:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import numpy as np
cimport numpy as np
cimport cdezing


cdef cdezing.Options ctrl
cdef unsigned int batchsize

def getversion():
   global ctrl
   ctrl.versionflag=1
   cdezing.runDezing(&ctrl,0,NULL,NULL)


def setup_size(array_size,outlier_mu,npad,logfile="dezing.log",versionflag=0): #,bytes summary):
   global ctrl
   global batchsize

   /* ... some other python code here ... */

   cdezing.timestamp( "calling c function setup" )
   cdezing.runDezing(&ctrl,batchsize, NULL, NULL)
   pass

def setup(np.ndarray[np.uint16_t,ndim=3,mode="c"] inarray,np.ndarray[np.uint16_t,ndim=3,mode="c"] outarray,outlier_mu,npad,logfile="dezing.log",versionflag=0): #,bytes summary):

   /* ... some other python code here ... */

   cdezing.timestamp( "calling c function setup" )
   cdezing.runDezing(&ctrl,batchsize, inbuf, outbuf)
   pass

def run(np.ndarray[np.uint16_t,ndim=3,mode="c"] inarray,np.ndarray[np.uint16_t,ndim=3,mode="c"] outarray): #,bytes summary):

   /* ... some other python code here ... */

   cdezing.runDezing(&ctrl,batchsize, <unsigned char *>np.PyArray_DATA(inarray), <unsigned char *> np.PyArray_DATA(outarray))
   pass


def cleanup(): #,bytes summary):

   /* ... some other python code here ... */

   cdezing.runDezing(&ctrl,batchsize, NULL,NULL)
   cdezing.timestamp_close()

  1. Makefile: In python this is a setup.py file. For example, setup.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
      cmdclass={'build_ext':build_ext},
      ext_modules=[
         Extension("dezing",["dezing.pyx"],
            libraries=["dezing"])
         ]
      )

Compile this file, passing appropriate C compiler flags if necessary, to obtain a *.so file.

e.g. export CFLAGS=”-I . $CFLAGS” export LDFLAGS=”-L . $LDFLAGS” python setup.py build_ext -i

The output file for this example is a dezing.so file. Transfer this file to lib and import as a python module, e.g. import dezing

Boost.Python Example

http://www.boost.org/doc/libs/1_58_0/libs/python/doc/

Boost.python aims to expose C++ classes/functions to python, without changing the original code.

  1. A python wrapper: Create the python module and define the external function names. For example, example_wrapper.cpp

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include "include/numpy_boost_python.hpp"

#include "base_types.hpp"
/* ...other relevant header files... */

using namespace boost::python;

#include "example.hpp"
#include "example.cpp"

BOOST_PYTHON_MODULE(example)
{
  import_array();
  numpy_boost_python_register_type<float, 1>();
  numpy_boost_python_register_type<float, 3>();
  def("cpp_function1", example_function1);
  def("cpp_function2", example_function2);
  def("cpp_function3", example_function3);
}

  1. A makefile: A standard C++ makefile, incorporating Boost.Python path, to build a shared object library (*.so). For example, example_makefile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
CXX = g++
PYTHONINC = -I/usr/include/python2.6 -I/usr/lib64/python2.6/site-packages/numpy/core/include
PYTHONLIB = -lboost_python

CXXFLAGS = -g -O2 -fPIC -ansi -Wall -Wno-long-long -DUSE_TIMER=true -fopenmp -I../src $(PYTHONINC)

LD = $(CXX)
LDFLAGS = $(CXXFLAGS) -shared
LIBS = $(PYTHONLIB)

PYOBJS = example_wrapper.o
OBJECTS = # ...lots of .o files... #

example.so: $(PYOBJS) $(OBJECTS)
	$(LD) $(LDFLAGS) -o $@ $(PYOBJS) $(OBJECTS) $(LIBS)

# ...more code here... #

The output file for this example is a example.so file. Transfer this file to lib and import as a python module, e.g. import example, then simply access a function from within your python code as example.example_function1(…).

The class file example.cpp (below) along with example_wrapper.cpp, illustrate how to incorporate numpy arrays into the extension.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
void example_function1(numpy_boost<float, 3> &pixels, const real param_n,
			const real param_r, const int num_series)
{
  /* ... some other c++ code here ... */
}

numpy_boost<float, 3> example_function2(const numpy_boost<float, 3> &pixels,
				       const numpy_boost<float, 1> &angles,
				       double rotation_centre, int resolution,
				       int niterations, int nthreads)
{
  /* ... some other c++ code here ... */  
}

void example_function3()
{
  /* ... some other c++ code here ... */  
}