Savu Developer Information¶
Developing in locally installed Savu (Lite)¶
Once Savu-lite has been installed into conda environment you can test the changes you make by simply reinstalling conda in your environment by running:
>>> python setup.py install
This will reinstall Savu with your changes in your conda location, e.g. here: “miniconda3/envs/savu/lib/python3.7/site-packages/savu-4.0-py3.7.egg”. One can also work directly in conda environment to see the immediate changes or for debugging purposes.
Release Strategy¶
When a new release is required the following process should be done. Make a new branch for convinience if fixes are needed, it should be called vX.Y-release i.e v0.1-release. After the branch is made, move to it and then create a release tag for it called vX.Y.Z i.e v0.1.0, where Z starts at 0, and future minor release verisons can be made with v0.1.1 etc later on down the branch.
Once this is done, Zenodo.org will automatically create and archive an artefact.
How to develop on a feature branch¶
- Make a new Branch for development and move to it
git checkout -b new_branch
- Make modifications and commit as normal
git commit -m “test commit”
- push this branch up to github
git push –set-upstream origin new_branch
- get all the latest data from github on all branches
git fetch
- to keep up to date, merge recent changes from master into the development branch and fix issues if there are any
git merge origin/master
Continue working with the branch untill you are happy with the new feature, merge master into it as shown before and fix up issues, then merge the branch into master
- move to master
git checkout master
- update
git pull
- merge in the new branch
git merge new_branch
- there should be no problems if you have merge master in first, so just push back up
git push
How to test a new plugin using DAWN¶
DAWN can be downloaded from http://www.dawnsci.org/ and general user tutorials are found at https://www.youtube.com/user/DAWNScience
Using the Debug perspective, create a new test, e.g. “tomopy_recon_test.py” to test your plugin in “/Savu/savu/test/”, in this case the “example_filter_back_projection.py” plugin for reconstructing data, setting the self.plugin_name appropriately. After saving the file, right-click on it in the PyDev Package Explorer window and Run As a Python unit-test
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 | # -*- coding: utf-8 -*- # Copyright 2014 Diamond Light Source Ltd. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """ .. module:: tomopy_recon_test :platform: Unix :synopsis: unittest for the tomopy 'gridrec' reconstruction .. moduleauthor:: Nicola Wadeson <scientificsoftware@diamond.ac.uk> """ import unittest import savu.test.test_utils as tu from savu.test.travis.framework_tests.plugin_runner_test import \ run_protected_plugin_runner class TomopyReconTest(unittest.TestCase): def setUp(self): self.data_file = 'tomo_standard.nxs' self.experiment = 'tomo' def test_tomopy_gridrec(self): process_list = 'reconstruction/tomopy_test.nxs' options = tu.initialise_options(self.data_file, self.experiment, process_list) run_protected_plugin_runner(options) tu.cleanup(options) if __name__ == "__main__": unittest.main() |
This runs a series of tests and produces an output file with the result of the plugin, whether it be a filter or a reconstruction, allowing for visualisation of the data, providing a check of whether the process has worked successfully.
The output file is saved in a tmp directory as a .h5 file, e.g. “/tmp/tmp32bexK.h5”. This can be viewed in DAWN.
Adding C/C++ extensions to a plugin¶
There are numerous ways to create python bindings to external C/C++ libraries, which may be useful to recycle existing code or to improve performance. Two different approaches have currently been tested: Cython (to link to external C code) and Boost.Python (to link to external C++ code). Cython is essentially python with C-types and requires a C-API, a python wrapper and a makefile, whilst Boost.Python is a wrapper for the Python/C API and requires a wrapper and a makefile. By building the makefile a shared library (*.so) file is created and can be added to the \lib directory in the Savu framework and imported as a python module.
Cython Example¶
http://docs.cython.org/src/tutorial/clibraries.html
A C interface: A *.pxd file, which is similar to a C header file, providing C function definitions required in the python code. For example, cdexing.pxd:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | cdef extern from "./options.h": ctypedef struct Options: unsigned char versionflag unsigned char f_call_num size_t cropwd unsigned int nlines float outlier_mu unsigned char returnflag unsigned int npad cdef extern from "./timestamp.h": void timestamp_open(const char * const logname) void timestamp_close() void timestamp_init() void timestamp(const char * const stampmsg) cdef extern from "./dezing_functions.h": void runDezing(Options * ctrlp, unsigned int thisbatch,unsigned char * inbuf, unsigned char * outbuf ) |
A python wrapper: A *.pyx file that must have a different name to the *.pyd file above. For example, dezing.pyx:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | import numpy as np cimport numpy as np cimport cdezing cdef cdezing.Options ctrl cdef unsigned int batchsize def getversion(): global ctrl ctrl.versionflag=1 cdezing.runDezing(&ctrl,0,NULL,NULL) def setup_size(array_size,outlier_mu,npad,logfile="dezing.log",versionflag=0): #,bytes summary): global ctrl global batchsize /* ... some other python code here ... */ cdezing.timestamp( "calling c function setup" ) cdezing.runDezing(&ctrl,batchsize, NULL, NULL) pass def setup(np.ndarray[np.uint16_t,ndim=3,mode="c"] inarray,np.ndarray[np.uint16_t,ndim=3,mode="c"] outarray,outlier_mu,npad,logfile="dezing.log",versionflag=0): #,bytes summary): /* ... some other python code here ... */ cdezing.timestamp( "calling c function setup" ) cdezing.runDezing(&ctrl,batchsize, inbuf, outbuf) pass def run(np.ndarray[np.uint16_t,ndim=3,mode="c"] inarray,np.ndarray[np.uint16_t,ndim=3,mode="c"] outarray): #,bytes summary): /* ... some other python code here ... */ cdezing.runDezing(&ctrl,batchsize, <unsigned char *>np.PyArray_DATA(inarray), <unsigned char *> np.PyArray_DATA(outarray)) pass def cleanup(): #,bytes summary): /* ... some other python code here ... */ cdezing.runDezing(&ctrl,batchsize, NULL,NULL) cdezing.timestamp_close() |
Makefile: In python this is a setup.py file. For example, setup.py:
1 2 3 4 5 6 7 8 9 10 11 | from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext setup( cmdclass={'build_ext':build_ext}, ext_modules=[ Extension("dezing",["dezing.pyx"], libraries=["dezing"]) ] ) |
Compile this file, passing appropriate C compiler flags if necessary, to obtain a *.so file.
e.g. export CFLAGS=”-I . $CFLAGS” export LDFLAGS=”-L . $LDFLAGS” python setup.py build_ext -i
The output file for this example is a dezing.so file. Transfer this file to lib and import as a python module, e.g. import dezing
Boost.Python Example¶
http://www.boost.org/doc/libs/1_58_0/libs/python/doc/
Boost.python aims to expose C++ classes/functions to python, without changing the original code.
A python wrapper: Create the python module and define the external function names. For example, example_wrapper.cpp
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION #include "include/numpy_boost_python.hpp" #include "base_types.hpp" /* ...other relevant header files... */ using namespace boost::python; #include "example.hpp" #include "example.cpp" BOOST_PYTHON_MODULE(example) { import_array(); numpy_boost_python_register_type<float, 1>(); numpy_boost_python_register_type<float, 3>(); def("cpp_function1", example_function1); def("cpp_function2", example_function2); def("cpp_function3", example_function3); } |
A makefile: A standard C++ makefile, incorporating Boost.Python path, to build a shared object library (*.so). For example, example_makefile
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | CXX = g++ PYTHONINC = -I/usr/include/python2.6 -I/usr/lib64/python2.6/site-packages/numpy/core/include PYTHONLIB = -lboost_python CXXFLAGS = -g -O2 -fPIC -ansi -Wall -Wno-long-long -DUSE_TIMER=true -fopenmp -I../src $(PYTHONINC) LD = $(CXX) LDFLAGS = $(CXXFLAGS) -shared LIBS = $(PYTHONLIB) PYOBJS = example_wrapper.o OBJECTS = # ...lots of .o files... # example.so: $(PYOBJS) $(OBJECTS) $(LD) $(LDFLAGS) -o $@ $(PYOBJS) $(OBJECTS) $(LIBS) # ...more code here... # |
The output file for this example is a example.so file. Transfer this file to lib and import as a python module, e.g. import example, then simply access a function from within your python code as example.example_function1(…).
The class file example.cpp (below) along with example_wrapper.cpp, illustrate how to incorporate numpy arrays into the extension.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | void example_function1(numpy_boost<float, 3> &pixels, const real param_n, const real param_r, const int num_series) { /* ... some other c++ code here ... */ } numpy_boost<float, 3> example_function2(const numpy_boost<float, 3> &pixels, const numpy_boost<float, 1> &angles, double rotation_centre, int resolution, int niterations, int nthreads) { /* ... some other c++ code here ... */ } void example_function3() { /* ... some other c++ code here ... */ } |