Cytomine is designed to enable multidisciplinary and distributed collaboration. In this Section we describe the internals of the four main modules of our software illustrated by the figure below : Cytomine-Core, Cytomine-IMS, Cytomine-WebUI, and Cytomine-DataMining. Our software follows the representational state transfer (REST) architecture and it is based on abstract communication interfaces so that main components (and users) can be at different physical locations (they are referenced by uniform resource locator (URL)).
Shortly, the full Cytomine architecture is based on HTTP and Rest API (Part 1) .
- The Cytomine core server (Part 2) is a MVC web application written in Groovy/Java with the Grails framework. It provides a Rest API to manage all the data except for some specific image manipulation.
- Most of the data are store in SQL or NoSQL engine (Part 3). We store images on file system.
- There are some specific client in Java (Part 5) and Python (Part 6). They provides a easy way to get/update a domain instance inside a external application/script (abstract HTTP call, JSON message, HTTP status code,...).
- Image operations (image upload and get thumb, crop, tiles,...) are done thanks to the IMS server (Part 7). Like Cytomine server, its a MVC web application written with same languages/framework.
- Image servers (Part 8) are light c++ application that read image on disk and extract some information.
- The retrieval server (Part 9) is an external application providing Content-based image retrieval.
More precisely, the Cytomine core server (Cytomine-Core) (https://github.com/cytomine/Cytomine-core) is a web application backend written in Groovy/Java with the Grails framework. It seamlessly integrates several libraries (see Section 2.2). It is based on model-view-controller (MVC) design patterns. While most of the data are actually stored in spatial SQL and NoSQL engines (See Supplementary note 2 for a description of the data model), the application provides RESTful web services to manage all these data (it is documented here http://demo.cytomine.be/restApiDoc/?doc_url=http://demo.cytomine.be/restApiDoc/api#) and relies mostly on the Hibernate framework for object-relational mapping. NoSQL engines were choosen to log effectively user activities in settings with thousands of users (see Supplementary Note 5.1 about our teaching server instance).
Image operations (image upload and extraction of image areas and annotation masks) are performed through the Cytomine Image Management System (Cytomine-IMS) (https://github.com/cytomine/Cytomine-IMS). Like Cytomine-Core, it is a MVC web backend application written in a collection of languages and frameworks. By combining lightweight image servers (namely IIPImage) and libraries (namely VIPS and OpenSlide), it currently supports standard 2D image formats (Pyramidal or non-pyramidal TIFF/BigTiff, jpeg, png, gif) and major 2D whole-slide scanner image formats (including Hamamatsu Ndpi, Aperio svs, 3DHistech mrxs, Leica SCN, Philips Tiff, Ventana Bif, Huron Tiff, Zeiss CZI once converted to Tiff). Pyramidal TIFF and whole-slide scanner image formats do not require to store many independant tiles on disk. They are designed to enable fast random access to small image areas from the whole image file (without loading the whole image into memory). Using domain sharding, we enable parallelization of image tile delivery by running multiple instances of Cytomine-IMS and by associating (in our data model) a single image to multiple image servers URLs. Cytomine-Core then redirects seamlessly and randomly each image tile HTTP request to one of these Cytomine-IMS, distributed, server. This trick overcomes the limitation of simultaneous connections to a single host in web browsers, resulting in a faster user experience.
Our system also supports many microscopy 5D image sequences (x,y,channel,slice,timeframe) formats through seamless conversion to 2D OME-TIFF image sequences during uploading phase. Extension to novel image formats or optimizations could be implemented either as extension of current image server instances (through extension of currently used libraries, namely VIPS, Openslide and BioFormats) or by developing connectors to other image servers e.g. for 3D images using  or molecular MALDI imaging using .
The analysis modules (Cytomine-DataMining) are relying on Java and Python reusable clients (https://github.com/cytomine/Cytomine-java-client and https://github.com/cytomine/Cytomine-python-client) to extract/import data from Cytomine-Core and Cytomine-IMS. These clients provide an easy way to get/update Cytomine-Core/IMS data using an external application/script through abstract HTTP calls, JSON messages, and HTTP status codes. Simple examples are provided to manipulate data (Java: https://github.com/cytomine/Cytomine-java-client/tree/master/src/main/java/be/cytomine/client/sample and Python: https://github.com/cytomine/Cytomine-python-client/tree/master/client/examples). Developing clients in other languages is possible provided these implement basic HTTP operations. In terms of currently integrated algorithms, we first integrate an unsupervised image recognition library (https://github.com/loic911/CBIRetrieval) that is a multi-threaded implementation in Java of our unsupervised content-based image retrieval algorithm. The retrieval server is directly called by Cytomine-WebUI once a user selects or draws an annotation.The image recognition modules are multi-threaded implementations of our supervised algorithms for image classification, semantic segmentation, and landmark (interest point) detection. These modules are implemented in Python and rely on its stack of numerical and machine learning libraries (numpy, scikit-learn). Algorithms and examples of how to run them are provided (https://github.com/cytomine/Cytomine-python-datamining/tree/master/cytomine-applications). As other modules, they communicate with Cytomine-Core and Cytomine-IMS through HTTP requests. These modules can be launched through command-lines on remote servers, or executed manually from the Cytomine-WebUI. We use a message broker (RabbitMQ) implementing AMQP (Advanced Message Queuing Protocol) to invoke Cytomine-DataMining modules from Cytomine-WebUI and Cytomine-Core.
Although Cytomine is a complex software with tens of thousands of code lines and integrating many libraries, we use software containerization techniques (with Docker) to fully automate its deployment on regular desktops or servers. Scripts for installation and update of a Cytomine instance are provided: https://github.com/cytomine/Cytomine-bootstrap and the full installation procedure is described here: http://doc.cytomine.be/x/goCj
In addition to our main source code, we integrated many other open-source libraries, as illustrated by the figure below:
Table of contents: