Page tree
Skip to end of metadata
Go to start of metadata

For installation procedure of V1.0, see this page.

This guide is designed for an installation on Ubuntu 16.04 LTS.

In this page we assume you are interested to develop new modules for Cytomine-Core (e.g. new web services, new entity in the data model,...) and/or new modules for Cytomine-IMS (e.g. support of a new image format, new image web service, ...).

If you are only interested to plug a new script of image analysis algorithm, we recommend instead to look at the [DOC] Interoperability (data & algorithms) documentation page.

We will explain all the necessary steps to configure a development environment on your computer. The idea is to install a Cytomine instance on your localhost and all the development tools/libraries required so that you can develop and test on your own computer. 

Step 1 - Install the pre-requirements

Cytomine-Core and Cytomine-IMS mainly rely on Java, Groovy and the Grails framework.

sudo apt-get update -y
sudo apt-get install -y git curl software-properties-common wget unzip build-essential locate apt-transport-https
 
sudo echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
sudo echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update -y
sudo apt-get install -y oracle-java8-installer
echo 'export JAVA_HOME="/usr/lib/jvm/java-8-oracle"' >> .bashrc
 
cd /tmp/ && curl http://get.sdkman.io | bash
sdk install grails 2.4.4
sdk install groovy 2.3.7


Then, check Java and Groovy version:

java -version #(1.8)
groovy -v #(2.3.7)

 

Restart a terminal in order the environment variable $GRAILS_HOME is effective. Fix Grails 2.4.4 to work with Java 8:

wget repo.spring.io/libs-snapshot-local/org/springframework/springloaded/1.2.3.BUILD-SNAPSHOT/springloaded-1.2.3.BUILD-SNAPSHOT.jar -O $GRAILS_HOME/lib/org.springframework/springloaded/jars/springloaded-1.2.1.RELEASE.jar


It may be also usefull to install a modern web browser such as Google Chrome, to navigate into the web interface.

Step 2 - Retrieve source code

Download our source code in a local directory: 

mkdir Cytomine/
cd Cytomine/
 
git clone https://github.com/Cytomine-ULiege/Cytomine-bootstrap.git

git clone https://github.com/Cytomine-ULiege/Cytomine-core.git
 
# if you want to develop on IMS:
git clone https://github.com/Cytomine-ULiege/Cytomine-IMS.git

Step 3 - Create a Dockerized development environment.

One of the main interest of Docker is to have the same environment between developers and final users.

In order to do this and to avoid installation of an additional local machine, we will setup a Dockerized development environment. Follow the step 2 "Install Docker" and step 3 and 3.1 "Configure Cytomine Installation" from Automatic Installation with Docker.

In addition, into your configuration.sh, change

ADMIN_PWD=$(cat /proc/sys/kernel/random/uuid)
ADMIN_PUB_KEY=$(cat /proc/sys/kernel/random/uuid)
ADMIN_PRIV_KEY=$(cat /proc/sys/kernel/random/uuid)
SUPERADMIN_PUB_KEY=$(cat /proc/sys/kernel/random/uuid)
SUPERADMIN_PRIV_KEY=$(cat /proc/sys/kernel/random/uuid)
RABBITMQ_PUB_KEY=$(cat /proc/sys/kernel/random/uuid)
RABBITMQ_PRIV_KEY=$(cat /proc/sys/kernel/random/uuid)
IMS_PUB_KEY=$(cat /proc/sys/kernel/random/uuid)
IMS_PRIV_KEY=$(cat /proc/sys/kernel/random/uuid)

by

ADMIN_PWD="admin"
ADMIN_PUB_KEY="JKL"
ADMIN_PRIV_KEY="GHI"
SUPERADMIN_PUB_KEY="PQR"
SUPERADMIN_PRIV_KEY="MNO"
RABBITMQ_PUB_KEY="VWX"
RABBITMQ_PRIV_KEY="STU"
IMS_PUB_KEY="DEF"
IMS_PRIV_KEY="ABC"

This procedure breaks security mechanisms. It allows to install the development environment faster as public and private keys of servers don't need to be copied in your local instance of Cytomine-core and/or Cytomine-IMS.

Obviously, DON'T DO THAT IN PRODUCTION !


Step 4 - Install an integrated development 
environment

We recommend to use Intellij IDEA Ultimate Edition. The following considers it is the case.

Download Intellij IDEA Ultimate Edition on their website: https://www.jetbrains.com/idea/download/#section=linux

Install it by following the instructions on their website: https://www.jetbrains.com/help/idea/install-and-set-up-intellij-idea.html

If you prefer you can visualize the installation with screenshots here.

Step 5 - Install & run core locally

Step 5.1 - Update your /etc/hosts

 If you deploy on local host. If you kept the default configuration, update your /etc/hosts with:

127.0.0.1   localhost-core
127.0.0.1   localhost-ims
127.0.0.1   localhost-ims2
127.0.0.1   localhost-upload
127.0.0.1   localhost-retrieval
127.0.0.1   rabbitmq


Step 5.2 - Configure the project in IntelliJ

Import the Cytomine-Core source project into Intellij, by choosing Maven as external model to import the projects.

Choose the right JDK version.

If needed, right click on the project (left menu) and choose "Add Framework support" to select Groovy.

If Grails SDK is not found (see Event Log), click on "Configure SDK Home". The path is given by

echo $GRAILS_HOME

If the source root of the project is not detected, in the Project panel on left, right click on grails-app folder and then Mark Directory as > Source root.

For more detailed importation of projects if you are not familiar with Intellij, see the screenshots on this page.

In your Cytomine-Core project, in the Cytomine-core/grails-app/conf/Config.groovy file of your Cytomine-core project, in "devlopment" part, replace the $XXX variables with the ones from your configuration.sh:

...
development {
    grails.serverURL = "http://$CORE_URL:8080"
    grails.uploadURL = "http://$UPLOAD_URL"
    grails.imageServerURL = ["$IMS_URLS[0]"]
    grails.retrievalServerURL = ["$RETRIEVAL_URL"]
    grails.converters.default.pretty.print = true
    grails.plugin.springsecurity.useBasicAuth = false
    grails.resources.adhoc.patterns = ['/images/*', '/css/jsondoc/*', '/js/*', '/plugins/*']
    grails.readOnlyProjectsByDefault = true
    grails.adminPassword="admin"
    grails.ImageServerPrivateKey="$IMS_PRIV_KEY"
    grails.ImageServerPublicKey="$IMS_PUB_KEY"
    grails.adminPrivateKey="$ADMIN_PRIV_KEY"
    grails.adminPublicKey="$ADMIN_PUB_KEY"
    grails.superAdminPrivateKey="$SUPERADMIN_PRIV_KEY"
    grails.superAdminPublicKey="$SUPERADMIN_PUB_KEY"
    grails.rabbitMQPrivateKey="$RABBITMQ_PRIV_KEY"
    grails.rabbitMQPublicKey="$RABBITMQ_PUB_KEY"
    grails.retrievalUsername="cytomine"
    grails.retrievalPassword="retrieval_default"
}
...


In the DataSource.groovy file into the Core project, update the values in development part:

In IntelliJ, in the "Run" menu toolbar, Go to "Edit configurations" edit the Command Line by 

-reloading -Dserver.port=8080 run-app -echoOut --stacktrace -nocoverage --verbose

 

Step 6 - Install & run IMS locally

If DEV_IMS=true in your configuration.sh, follow these steps.

Step 6.1 - Install the IMS required libs

If you want develop on IMS, the IMS will run on your machine and you will need all the dependencies of IMS : tiffinfo, identify, vips. You can find into the Dockerfile of IMS the way to install these dependancies. However, the following commands should be enough to get the related libs:

sudo apt-get install -y imagemagick libvips libvips-dev libvips-tools libtiff-tools openslide-tools autoconf
sudo chmod -R 777 /data
 
# Install Openslide-Java bindings
export CFLAGS="-I/usr/lib/jvm/java-8-oracle" 
cd /tmp 
git clone https://github.com/cytomine/openslide-java
cd /tmp/openslide-java/
autoreconf -i
./configure
make
sudo make install

 

Step 6.2 - Update your /etc/hosts

 If you deploy on local host. If you kept the default configuration, append to your /etc/hosts:

127.0.0.1   localhost-iip-base
127.0.0.1   localhost-iip-jp2000
127.0.0.1   localhost-iip-cyto


Step 6.3 - Configure the project in IntelliJ

Import the Cytomine-IMS source project into Intellij, by choosing Maven as external model to import the projects.

Choose the right JDK version.

If needed, right click on the project (left menu) and choose "Add Framework support" to select Groovy.

If Grails SDK is not found (see Event Log), click on "Configure SDK Home". The path is given by

If the source root of the project is not detected, in the Project panel on left, right click on grails-app folder and then Mark Directory as > Source root.

For more detailed importation of projects if you are not familiar with Intellij, see the screenshots on this page.

In your Cytomine-IMS project, in the Cytomine-IMS/grails-app/conf/Config.groovy file of your Cytomine-IMS project, replace the $XXX variables with the ones from your configuration.sh:

...
cytomine.imageServerPrivateKey="$IMS_PRIV_KEY"
cytomine.imageServerPublicKey="$IMS_PUB_KEY"

//image manipulation executable
cytomine.vips = "vips"
cytomine.tiffinfo = "tiffinfo"
cytomine.identify = "identify"
cytomine.vipsthumbnail = "vipsthumbnail"

...

 

If you still develop with a verison that has Background plugin, in order to avoid the error "org.codehaus.groovy.grails.commons.ConfigurationHolder: class not found" when compiling Cytomine-IMS, edit the file ~/.grails/2.4.4/projects/IMS/plugins/background-thread-1.6/BackgroundThreadGrailsPlugin.groovy, by replacing 

import org.codehaus.groovy.grails.commons.ConfigurationHolder as CH

by 

import grails.util.Holders as CH

 

In IntelliJ, in the "Run" menu toolbar, Go to "Edit configurations" edit the Command Line by 

-reloading -Dserver.port=9080 run-app -echoOut --stacktrace -nocoverage --verbose


Step 7 - Start Docker containers and develop

In Cytomine-boostrap directory, you can now run the script that will generate the development environment installation file and deploy the Docker containers (except Core and/or IMS that you will run locally):

sudo bash init.sh
sudo bash dev_core_start_deploy.sh

If DEV_CORE=true, in IntelliJ, you can now run the Cytomine-core project that will simulate a server at the grails.serverURL mentionned above. You can now access the core development server at http://CORE_URL:8080 to test your core dev environment. It will rely on other containers installed by Docker but use your own core development environment.

The first time, it will create some user accounts and an admin account (username: admin, password: admin).

If DEV_IMS=true, in IntelliJ, you can now run the Cytomine-IMS project that will simulate the IMS server mentionned above. You can now access the IMS development server at http://IMS_URL to test your IMS dev environment. It will rely on other containers installed by Docker but use your own IMS development environment.

Obvisouly, you can combine DEV_CORE=true and DEV_IMS=true to work on booth servers at the same time.

 

To restart the docker containers while keeping data, use the same script:

sudo sh ./dev_core_restart.sh

 

Troubleshooting

  • If you encounter a "java.lang.OutOfMememoryError: PermGen space" error, add the option -XX:+CMSClassUnloadingEnabled to the JVM, when running Cytomine-Core. In Intellij, go to Run > Edit Configurations and add it in the VM options field.

  • As nginx will try to contact your machine, you may need to allow docker containers to contact port 8080 & 9080. Here is the command with UFW

    sudo ufw allow from 172.17.0.1/27

 

 

  • No labels