== HJCfit in QUB == HJCfit is a program for HMM analysis of ion channels, developed in the lab of David Colquhoun, featuring the best (exact) missed events correction. QUB is a set of user interfaces and algorithms for preprocessing and HMM analysis of ion channels. We will combine the two, making the HJC likelihood function available in a QUB interface, and using QUB's optimizers to find the maximum likelihood solution for a model. QUB Express implements several core functions as plugins, so we will use the plugin that implements MIL as a starting point. Some limitations: 1. While QUB's interface allows any number of conductance classes, HJCfit is implemented for only two (open and closed). We will reorder two-class models to match HJCfit's partitioned Q matrix format, and disallow models with more than two classes. 2. As demonstrated by (Sivilotti et al 2016), there is a better approach than maximum likelihood fitting. As their algorithm would have to be hand-translated from Matlab, it is beyond our scope today. Thanks to the researchers at University College, London, and at SUNY Buffalo, for making both programs available under compatible open-source licenses. Our procedure, briefly, is to: 0. set up the build environment 1. build HJCfit as a shared library 2. make a copy of the QUB Express "Single Molecule" plugin, rename it, and strip out QUB-specific algorithms 3. write any helper functions necessary to call the HJC likelihood function with a c-linked function of basic scalar and pointer types 4. write a python/ctypes adapter for the likelihood function 5. replace the MIL likelihood function with HJC 6. build and package for win32 7. build and package for macos 8. build and package for JavaScript (browser and node.js) == Set up the build environment == (using Ubuntu 16.04) # qub-express prerequisites $ sudo apt-get install libboost-dev libgsl0-dev python-gtk2 python-scipy git $ git clone https://www.qub.buffalo.edu/qub-express.git # HJC prerequisites $ sudo apt-get install cmake-qt-gui libeigen3-dev $ git clone https://github.com/DCPROGS/HJCFIT.git == Build HJCfit == $ cd HJCFIT $ cmake-gui & * choose the current directory (HJCFIT/) for both source and binaries * click "Configure"; it stops on an error (missing swig). We'll skip the python bindings for now. * un-check "compile docs", "python bindings", and "tests" * click "Configure" * click "Generate" $ make # copy the shared library into qub-express root $ cp likelihood/liblikelihood.so ../qub-express == QUB Express plugin == We'll copy the plugin containing MIL, at qub-express/qubx_single_molecule and start adapting it to our needs: $ cd ../qub-express $ cp -a qubx_single_molecule qubx_hjcfit $ cd qubx_hjcfit $ rm *.pas *.opencl $ rm max_inter* max_subi* qubx_ideal* qubx_dur* $ mv max_subi_ll.py qubxhjc.py $ mv maxill.h qubxhjc.h $ mv qubx_single_molecule.nsi qubx_hjcfit.nsi $ mv Makefile_qubx_single_molecule.in Makefile_qubx_hjcfit.in $ mv maxill.sln qubxhjc.sln $ mv maxill.vcproj qubxhjc.vcproj $ mv qubx_single qubx_hjc $ rm qubx_hjc/dur_hist.py qubx_hjc/idealize.py And we link in the plugin for testing: $ cd ../Plugins $ ln -s ../qubx_hjcfit . $ cd .. We'll comb through the remaining files, replacing the .cpp and .h with new stuff, and rewriting the rest as necessary. We'll also add qubx_hjcfit to qubx-express/Makefile, following the example of qubx_single_molecule. == Helper functions == liblikelihood.so exports a C++ interface. This is problematic because C++ shared linkage is not defined by any standard. We can only expect that another library, compiled with the same compiler, will be able to access the C++ interface correctly, so we will make a small adapter library which exports a C interface and internally calls HJCFIT's C++ API. We need not wrap the entire API in C; for our purpose we need only a few functions (in qubxhjc.cpp): qub_hjc_bursts_alloc() qub_hjc_bursts_append(bursts, durations[], count) qub_hjc_bursts_free(bursts) qub_hjc_likelihood(bursts, Q[], numStates, numOpen, tres, tcrit, useCHS) == Python/ctypes adapter == Even though QUB Express is written largely in Python, we are avoiding the Swig interface to keep things as simple as possible and not complicate the cross-platform build. With Python's ctypes and numpy packages, we can load and call c-linkage functions from a shared library without external help. Later, we'll also be able to use these c-linkage functions from JavaScript. In qubxhjc.py, we load the shared library and provide a basic interface. It filters events with the dead time, and re-orders the Q matrix to put all open states first. It enforces the rules about burst geometry and no sub-levels. == HJC likelihood == Then we change the rate optimizer panel in qubx_hjcfit/qubx_hjc/rates.py to use the interface from qubxhjc.py. We omit the gradient function entirely, and let QUB Express use finite differences. To test: $ make run You should see simulated single-channel data. Go to the Simulation tab and "pause", so data won't be re-simulated when the model changes. Then go to the Modeling:HJCFIT tab and try "Get LL" and "Optimize Rates". If only python sources have changed, we can re-load the plugin in the Admin:Plugins tab. If C++ sources have changed, we must close QUB Express, then `make run`. == For Windows (32 bit) == Using my trusty old Windows XP VM. I don't have a new enough Visual Studio, so we'll try MinGW. Alas, the MinGW installer is crashing on XP, so we'll try MinGW under linux, cross-compiling for win32. $ sudo apt-get install mingw-w64 $ cd ../HJCFIT $ cmake-gui& check "Advanced", then edit these entries: CMAKE_CXX_COMPILER=/usr/bin/i686-w64-mingw32-g++ CMAKE_C_COMPILER=/usr/bin/i686-w64-mingw32-gcc CMAKE_LINKER=/usr/bin/i686-w64-mingw32-g++ and add CMAKE_SYSTEM_NAME=Windows ... That didn't work. It complained the compiler can't handle -rdynamic. Same problem using cmake -DCMAKE_TOOLCHAIN_FILE=... Instead we'll try a primitive approach in qubx_hjcfit/Makefile, directly naming the cross-compiler, HJCFIT/likelihood/*.cc, and qubxhjc.cpp. This works. We copy the resulting qubxhjc.dll along with two runtime dlls from mingw in /usr/lib/gcc/i686-w64-mingw32/5.3-win32/: libstdc++-6.dll libgcc_s_sjlj-1.dll and add them to the NSIS install script. == For MacOS == We've been building for MacOS on Mavericks (10.9) using homebrew and its g++-4.9 $ brew install cmake $ git clone https://github.com/DCPROGS/HJCFIT.git $ cd HJCFIT $ cmake -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-4.9 -DCMAKE_LINKER=/usr/local/bin/g++-4.9 -Dtests=off -DpythonBindings=off -DcompileDocs=off . $ make $ cp likelihood/liblikelihood.dylib ../qub-express/liblikelihood.so $ cd ../qub-express $ cd Plugins $ ln -s ../qubx_hjcfit . $ cd .. $ make run It works! Now we add the libraries to the py2app setup script (add to extra_options['options']['py2app']['frameworks']) and can successfully build an app bundle. == For JavaScript == Back to the primitive Makefile in qubx_hjcfit/Makefile. We define an alternate compile target using emcc, the emscripten transpiler. This generates a JavaScript module, js/qubxhjc.js with its own memory space and special calling conventions. We write a small adapter in js/hjcfit.js which specifies function names and argument types, and handles the library's memory transfers and object pointers. We are using the "asmffi" tools we developed for QUB Online (work in progress at https://www.qub.buffalo.edu/online), which consist of C++ and JS sources. The C++ source defines std::ostream &console (prints to the browser console), and provides JS with c-linked basic memory allocation and access functions. The JS source defines a require.js-based loader which can be used either in the browser, where it loads the output from emscripten, or in node.js, where it can preferentially load a traditional .so shared library using node's 'ffi' module. We anticipate that quick calculations might stay in the user's browser (emscripten), while larger efforts might be farmed out to cloud instances (ffi).