anchor.diff:  image:https://api.codacy.com/project/badge/Grade/d1d650940b1f4a6286607c1d06573090["Codacy code quality", link="https://www.codacy.com/app/danielvaughan/biosamples-v4?utm_source=github.com&utm_medium=referral&utm_content=EBIBioSamples/biosamples-v4&utm_campaign=Badge_Grade"]
   image:https://quay.io/repository/ebibiosamples/biosamples-v4/status["Docker Repository on Quay", link="https://quay.io/repository/ebibiosamples/biosamples-v4"]
   
  -[[quickstart]]
  -Quickstart
  -----------
  -
  -Install Maven and JDK 8 Install docker-compose
  -https://docs.docker.com/compose/
  -
  -MacOS: Install Docker for Mac from https://www.docker.com/docker-mac
  -
  -`mvn package`
  += BioSamples
  +
  +BioSamples https://www.ebi.ac.uk/biosamples/ stores and supplies descriptions and metadata about biological samples used in research and development by academia and industry. Samples are either 'reference' samples (e.g. from 1000 Genomes, HipSci, FAANG) or have been used in an assay database such as the European Nucleotide Archive (ENA) or ArrayExpress. 
  +
  +BioSamples also synchronize data with the NCBI BioSample database and import data from ENA.
  +
  +This document provides information about the local installation, development environment setup  instructions of BioSamples database. 
  +
  +== Table of contents
  +
  +<<Softwares>>
  +
  +<<Setup>>
  +
  +<<Data import>>
  +
  +<<Developing>>
  +
  +<<Client useage>>
  +
  +<<Issues and troubleshooting>>
  +
  +<<License>>
  +
  +== Softwares
  +
  +* Git 2.17.1
  +* Java 8
  +* JDK 8
  +* Maven 3.6.0
  +* Docker 18.6
  +
  +== Setup
  +
  +[arabic]
  +. Run this in terminal to install the dependent softwares.
  ++
  +[source,sh]
  +----
  +sudo apt-get update
  +sudo apt-get install openjdk-8-jdk maven git docker
  +----
  +. Please make sure the software versions are correct.
  ++
  +[source,sh]
  +----
  +mvn -v
  +# Output: 
  +# Apache Maven 3.6.0
  +
  +docker -v 
  +# Output
  +# Docker version 18.06.1-ce
  +
  +java -version
  +# Output
  +# openjdk version "1.8.0_222"
  +----
  +. Install BioSamples on your computer.
  ++
  +This process sets up a local compiled version of all biosamples tools.
  +It requires a large download of Spring dependencies and uses up to two
  +threads per core of your machine. The installation might take several
  +minutes.
  ++
  +[source,sh]
  +----
  +git clone https://github.com/EBIBioSamples/biosamples-v4.git
  +cd biosamples-v4
  +mvn -T 2C package
  +----
  +. Start Biosamples on your own machine
  ++
  +[source,sh]
  +----
  +docker-compose up
  +----
  ++
  +If it returns
  +`+ERROR: Couldn't connect to Docker daemon - you might need to run docker-machine start default+`.
  +Please run `+sudo docker-compose up+` instead.
  +. Now you can access the public interface at
  +http://localhost:8081/biosamples/. So far, there is no data in the local
  +sample.
  +
  +== Data import
  +
  +An example of the JSON format that can be sent by POST to http://localhost:8081/biosamples/beta/samples is at https://github.com/EBIBioSamples/biosamples-v4/blob/master/models/core/src/test/resources/TEST1.json
  +
  +=== NCBI
  +
  +Download the XML dump (~400Mb) to the current directory:
  +
  +`wget http://ftp.ncbi.nih.gov/biosample/biosample_set.xml.gz`
  +
  +Run the pipeline to send the data to the submission API via REST
  +
  +`docker-compose up biosamples-pipelines-ncbi`
  +
  +Note: You will need to mount the location that the XML dump was
  +downloaded to within the docker container. A docker-compose.override.yml
  +file is the easiest way to do that.
  +
  +=== ENA
  +
  +You can run the pipelines-ena to import ENA samples.
  +In order to do that you will need to add some security settings to maven 
  +to get access to oracle private driver repository.
  +
  +You can read more instructions about this at https://blogs.oracle.com/dev2dev/get-oracle-jdbc-drivers-and-ucp-from-oracle-maven-repository-without-ides
  +
  +=== MongoDB notes
  +
  +Cross-platform easy to use mongodb management tool
  +http://www.mongoclient.com
  +
  +== Developing
  +
  +Docker can be run from within a virtual machine e.g VirtualBox. This is
  +useful if it causes any problems for your machine or if you have an OS
  +that is not supported.
  +
  +You might want to mount the virtual machines directory with the host, so
  +you can work in a standard IDE outside of the VM. VirtualBox supports
  +this.
  +
  +If you ware using a virtual machine, you might also want to configure
  +docker-compose to start by default.
  +
  +As you make changes to the code, you can recompile it via Maven with:
  +
  +`mvn -T 2C package`
  +
  +And to get the new packages into the docker containers you will need to
  +rebuild containers with:
   
   `docker-compose build`
   
  -`docker-compose up biosamples-webapps-core`
  -
  -Note: this will download around 1GB of docker containers
  -
  -public interface at http://localhost:8081/biosamples/beta
  -
  -An example of the JSON format that can be sent by POST to
  -http://localhost:8081/biosamples/beta/samples is at
  -https://github.com/EBIBioSamples/biosamples-v4/blob/master/models/core/src/test/resources/TEST1.json
  -
  -[[client-useage]]
  -Client useage
  --------------
  +If needed, you can rebuild just a single container by specifying its
  +name e.g.
  +
  +`docker-compose build biosamples-pipelines`
  +
  +To start a service, using docker compose will also start and dependent
  +services it requires e.g.
  +
  +`docker-compose up biosamples-webapp-api`
  +
  +will also start solr, neo4j, mongo, and rabbitmq
  +
  +To run an executable file in a docker container, and start its
  +dependencies first use something like:
  +
  +`docker-compose run --service-ports biosamples-pipelines`
  +
  +If you want to add command line arguments note that these will entirely
  +replace the executable in the docker-compose.yml file. So you need to do
  +something like:
  +
  +`docker-compose run --service-ports biosamples-pipelines java -jar pipelines-4.0.0-SNAPSHOT.jar --debug`
  +
  +If you want to connect debugging tools to the java applications running
  +inside docker containers, see instructions at
  +http://www.jamasoftware.com/blog/monitoring-java-applications/
  +
  +Note that you can bring maven and docker together into a single
  +commandline like:
  +
  +`mvn -T 2C package && docker-compose build && docker-compose run --service-ports biosamples-pipelines`
  +
  +Beware, Docker tar's and copies all the files on the filesystem from the
  +location of docker-compose down. If you have data files there (e.g.
  +downloads from ncbi, docker volumes, logs) then that process can take so
  +long that it makes using Docker impractical.
  +
  +As docker-compose creates new volumes each time, you may fill the disk
  +docker is working on. To delete all docker volumes use:
  +
  +`docker volume ls -q | xargs -r docker volume rm`
  +
  +To delete all docker images use:
  +
  +`docker images -q | xargs -r docker rmi`
  +
  +NOTE: this will remove everything not just things for this project
  +
  +== Client useage
   
   There is a spring client, and a spring-boot starter module, for use with
   BioSamples. To use these in a maven project, add the following to the
  
positive.code: including biosamples.client.uri to specify the base URI of the
  BioSamples instance to use.
  
  [[development-getting-started]]
  Development getting started
  ---------------------------
  
  To checkout and compile the code, you will need Git, Maven, and a JDK 8.
  On ubuntu-based Linux distributions (16.04 or higher) you can do this
  with:
  
  `sudo apt-get install maven git default-jdk`
  
  Then you can check out and compile the code with:
  
  `git clone https://github.com/EBIBioSamples/biosamples-v4 biosamples`
  `cd biosamples` `mvn -T 2C package`
  
  Note: This will require a large download of Spring dependencies. Note:
  This uses up to two threads per core of your machine.
  
  At that point, you will have a local compiled version of all the
  biosamples tools.
  
  To start a copy running on the local machine (e.g. to test any changes
  you have made) you can use Docker and Docker-compose.
  https://docs.docker.com/compose/
  
  You can use `docker-compose up` to start all the services, or you can
  bring them up individually. See docker-compose.yml file for more
  information on service names and dependencies.
  
  To update the code you are running you will need to 
  1. `git pull` the latest version
  2. Recompile the java classes by running `docker-webapp.sh`
  
  By default, the pipelines will not be run. They can be manually
  triggered as follows:
  
  [[ncbi]]
  NCBI
  ~~~~
  
  Download the XML dump (~400Mb) to the current directory:
  
  `wget http://ftp.ncbi.nih.gov/biosample/biosample_set.xml.gz`
  
  Run the pipeline to send the data to the submission API via REST
  
  `docker-compose up biosamples-pipelines-ncbi`
  
  Note: You will need to mount the location that the XML dump was
  downloaded to within the docker container. A docker-compose.override.yml
  file is the easiest way to do that.
  
  [[ena]]
  ENA
  ~~~~
  You can run the pipelines-ena to import ENA samples.
  In order to do that you will need to add some security settings to maven 
  to get access to oracle private driver repository.
  
  You can read more instructions about this at https://blogs.oracle.com/dev2dev/get-oracle-jdbc-drivers-and-ucp-from-oracle-maven-repository-without-ides
  
  
  [[developing]]
  Developing
  ----------
  
  Docker can be run from within a virtual machine e.g VirtualBox. This is
  useful if it causes any problems for your machine or if you have an OS
  that is not supported.
  
  You might want to mount the virtual machines directory with the host, so
  you can work in a standard IDE outside of the VM. VirtualBox supports
  this.
  
  If you ware using a virtual machine, you might also want to configure
  docker-compose to start by default.
  
  As you make changes to the code, you can recompile it via Maven with:
  
  `mvn -T 2C package`
  
  And to get the new packages into the docker containers you will need to
  rebuild containers with:
  
  `docker-compose build`
  
  If needed, you can rebuild just a single container by specifying its
  name e.g.
  
  `docker-compose build biosamples-pipelines`
  
  To start a service, using docker compose will also start and dependent
  services it requires e.g.
  
  `docker-compose up biosamples-webapp-api`
  
  will also start solr, neo4j, mongo, and rabbitmq
  
  To run an executable file in a docker container, and start its
  dependencies first use something like:
  
  `docker-compose run --service-ports biosamples-pipelines`
  
  If you want to add command line arguments note that these will entirely
  replace the executable in the docker-compose.yml file. So you need to do
  something like:
  
  `docker-compose run --service-ports biosamples-pipelines java -jar pipelines-4.0.0-SNAPSHOT.jar --debug`
  
  If you want to connect debugging tools to the java applications running
  inside docker containers, see instructions at
  http://www.jamasoftware.com/blog/monitoring-java-applications/
  
  Note that you can bring maven and docker together into a single
  commandline like:
  
  `mvn -T 2C package && docker-compose build && docker-compose run --service-ports biosamples-pipelines`
  
  Beware, Docker tar's and copies all the files on the filesystem from the
  location of docker-compose down. If you have data files there (e.g.
  downloads from ncbi, docker volumes, logs) then that process can take so
  long that it makes using Docker impractical.
  
  As docker-compose creates new volumes each time, you may fill the disk
  docker is working on. To delete all docker volumes use:
  
  `docker volume ls -q | xargs -r docker volume rm`
  
  To delete all docker images use:
  
  `docker images -q | xargs -r docker rmi`
  
  NOTE: this will remove everything not just things for this project
  
  [[mongodb-notes]]
  MongoDB notes
  -------------
  
  Cross-platform easy to use mongodb management tool
  http://www.mongoclient.com
  
  [[problems-with-spring-data-rest]]
  Problems with spring-data-rest
  ------------------------------
  
  This was originally using spring-data-rest to expose rest API for the
  repositories. But there are a number of problems with this (see below)
  
negative.code: ordering changes. Maybe this can be remedied by using map of attribute
  types instead?
  
  === Known issues
  
  Solr has a limit on the field size (technically the term vector).
  Therefore the attribute values over 255 characters are not indexed in
  solr.
  
  === License
  
  link:https://github.com/EBIBioSamples/biosamples-v4}/LICENSE[Apache 2.0]
  