3. Starting Scorpio via docker-compose

3.1. Start commands to copy

Looking for the easiest way to start Scorpio? This is it.

curl https://raw.githubusercontent.com/ScorpioBroker/ScorpioBroker/development/docker-compose-aaio.yml
sudo docker-compose -f docker-compose-aaio.yml up

3.2. Introduction

The easiest way to start Scorpio is to use docker-compose. We provide 2 main docker-compose files which rely on dockerhub. docker-compose-aaio.yml and docker-compose-dist.yml. You can use this files directly as they are to start Scorpio When you want to run Scorpio in the distributed variant exchange the yml file in the command above.

3.3. docker-compose-aaio.yml

AAIO here stands for almost all in one. In this variant the core components of Scorpio and the Spring Cloud components are started within one container. Additional containers are only Kafka and Postgres. For testing and small to medium size deployments this is most likely what you want to use.

3.4. docker-compose-dist.yml

In this variant each Scorpio component is started in a different container. This makes it highly flexible and allows you to replace individual components or to start new instances of some core components.

3.5. Configure docker image via environment variables

There are multiple ways to enter environment variables into docker. We will not got through all of them but only through the docker-compose files. However the scorpio relevant parts apply to all these variants. Configuration of Scorpio is done via the Spring Cloud configuration system. For a complete overview of the used parameters and the default values have a look at the application.yml for the AllInOneRunner here, https://github.com/ScorpioBroker/ScorpioBroker/blob/development/AllInOneRunner/src/main/resources/application-aaio.yml. To provide a new setting you can provide those via an environment entry in the docker-compose file. The variable we want to set is called spring_args. Since we only want to set this option for the Scorpio container we make it a sub part of the Scorpio Container entry like this

scorpio:
  image: scorpiobroker/scorpio:scorpio-aaio_1.0.0
  ports:
    - "9090:9090"
  depends_on:
    - kafka
    - postgres
  environment:
    spring_args: --maxLimit=1000

With this we would set the maximum limit for a query reply to 1000 instead of the default 500.

3.6. Be quit! docker

Some docker containers can be quite noisy and you don’t want all of that output. The easy solution is to add this

  logging:
driver: none

in the docker-compose file to respective container config. E.g. to make Kafka quite.

kafka:
  image: wurstmeister/kafka
  hostname: kafka
  ports:
    - "9092"
  environment:
    KAFKA_ADVERTISED_HOST_NAME: kafka
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_PORT: 9092
    KAFKA_LOG_RETENTION_MS: 10000
    KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 5000
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  depends_on:
    - zookeeper
  logging:
    driver: none

4. Configuration Parameters

Scorpio uses the Spring Cloud/Boot configuration system. This is done via the application.yml files in the corresponding folders. The AllInOneRunner has a complete set of all available configuration options in them.

Those can be overwriten via the command line or in the docker case as described above.

Config Option Description Default Value
atcontext.url the url to be used for the internal context server http://localhost:9090/ngsi-ld/contextes/
bootstrap.servers the host and port of the internal kafka kafka:9092 (default used for docker)
broker.id a unique id for the broker. needed for federation Broker1
broker.parent. location.url url for the parent broker in a federation setup SELF (meaning no federation)
broker. geoCoverage GeoJSON description of the coverage. used for registration in a federation setup. empty
defaultLimit The default limit for a query if no limit is provided 50
maxLimit The maximum number of results in a query 500
reader.datasource .password If you change the postgres setup here you set the password ngb
reader.datasource .url JDBC URL to postgres jdbc:postgresql://postgres:5432/ngb?ApplicationName=ngb_storagemanager_reader
reader.datasource .username username for the postgres db ngb
writer.datasource .password If you change the postgres setup here you set the password ngb
writer.datasource .url JDBC URL to postgres jdbc:postgresql://postgres:5432/ngb?ApplicationName=ngb_storagemanager_writer
writer.datasource .username username for the postgres db ngb
spring.datasource .password Same as above but used by flyway for database migration ngb
spring.datasource .url Same as above but used by flyway for database migration jdbc:postgresql://postgres:5432/ngb?ApplicationName=ngb_storagemanager_writer
spring.datasource .username Same as above but used by flyway for database migration ngb

5. Building Scorpio from source

Scorpio is developed in Java using SpringCloud as microservice framework and Apache Maven as build tool. Some of the tests require a running Apache Kafka messagebus (further instruction are in the Setup chapter). If you want to skip those tests you can run mvn clean package -DskipTests to just build the individual microservices.

5.1. General Remarks on Building

Further down this document you will get exact build commands/arguments for the different flavors. This part will give you an overview on how the different arguments work.

5.1.1. Maven Profiles

There currently three available Maven build profiles

5.1.1.1. Default

If you provide no -P argument Maven will produce individual jar files for the microservices and the AllInOneRunner with each “full” microservice packaged (this will result in ca. 500 MB size for the AllInOneRunner)

5.1.1.2. docker

This will trigger the Maven to build docker containers for each microservice.

5.1.1.3. docker-aaio

This will trigger the Maven to build one docker container, containing the AllInOneRunner and the spring cloud components (eureka, configserver and gateway)

5.1.1.4. Maven arguments

These arguments are provided via -D in the command line.

5.1.1.5. skipTests

Generally recommended if you want to speed up the build or you don’t have a kafka instance running, which is required by some of the tests.

5.1.1.6. skipDefault

This is a special argument for the Scorpio build. This argument will disable springs repacking for the individual microservices and will allow for a smaller AllInOneRunner jar file. This argument shoulnd ONLY be used in combination with the docker-aaio profile.

5.1.2. Spring Profiles

Spring supports also profiles which can be activated when launching a jar file. Currently there 3 profiles actively used in Scorpio. The default profiles assume the default setup to be a individual microservices. The exception is the AllInOneRunner which as default assumes to be running in the docker-aaio setup.

Currently you should be able to run everything with a default profile except the gateway in combination with the AllInOneRunner. In order to use these two together you need to start the gateway with the aaio spring profile. This can be done by attaching this to your start command -Dspring.profiles.active=aaio.

Additonally some components have a dev profile available which is purely meant for development purposes and should only be used for such.

5.2. Setup

Scorpio requires two components to be installed.

5.2.1. Postgres

Please download the Postgres DB and the Postgis extension and follow the instructions on the websites to set them up.

Scorpio has been tested and developed with Postgres 10.

The default username and password which Scorpio uses is “ngb”. If you want to use a different username or password you need to provide them as parameter when starting the StorageManager and the RegistryManager.

e.g.

java -jar Storage/StorageManager/target/StorageManager-<VERSIONNUMBER>-SNAPSHOT.jar --reader.datasource.username=funkyusername --reader.datasource.password=funkypassword

OR

java -jar Registry/RegistryManager/target/RegistryManager-<VERSIONNUMBER>-SNAPSHOT.jar --spring.datasource.username=funkyusername --spring.datasource.password=funkypassword

Don’t forget to create the corresponding user (“ngb” or the different username you chose) in postgres. It will be used by the SpringCloud services for database connection. While in terminal, log in to the psql console as postgres user:

sudo -u postgres psql

Then create a database “ngb”:

postgres=# create database ngb;

Create a user “ngb” and make him a superuser:

postgres=# create user ngb with encrypted password 'ngb';
postgres=# alter user ngb with superuser;

Grant privileges on database:

postgres=# grant all privileges on database ngb to ngb;

Also create an own database/schema for the Postgis extension:

postgres=# CREATE DATABASE gisdb;
postgres=# \connect gisdb;
postgres=# CREATE SCHEMA postgis;
postgres=# ALTER DATABASE gisdb SET search_path=public, postgis, contrib;
postgres=# \connect gisdb;
postgres=# CREATE EXTENSION postgis SCHEMA postgis;

5.2.2. Apache Kafka

Scorpio uses Apache Kafka for the communication between the microservices.

Scorpio has been tested and developed with Kafka version 2.12-2.1.0

Please download Apache Kafka and follow the instructions on the website.

In order to start kafka you need to start two components: Start zookeeper with

<kafkafolder>/bin/[Windows]/zookeeper-server-start.[bat|sh] <kafkafolder>/config/zookeeper.properties

Start kafkaserver with

<kafkafolder>/bin/[Windows]/kafka-server-start.[bat|sh] <kafkafolder>/config/server.properties

For more details please visit the Kafka website.

5.2.2.1. Getting a docker container

The current maven build supports two types of docker container generations from the build using maven profiles to trigger it.

The first profile is called ‘docker’ and can be called like this

sudo mvn clean package -DskipTests -Pdocker

this will generate individual docker containers for each micro service. The corresponding docker-compose file is docker-compose-dist.yml

The second profile is called ‘docker-aaio’ (for almost all in one). This will generate one single docker container for all components the broker except the kafka message bus and the postgres database.

To get the aaio version run the maven build like this

sudo mvn clean package -DskipTests -DskipDefault -Pdocker-aaio

The corresponding docker-compose file is docker-compose-aaio.yml

5.2.2.2. Starting the docker container

To start the docker container please use the corresponding docker-compose files. I.e.

sudo docker-composer -f docker-compose-aaio.yml up

to stop the container properly execute

sudo docker-composer -f docker-compose-aaio.yml down

5.2.2.3. General remark for the Kafka docker image and docker-compose

The Kafka docker container requires you to provide the environment variable KAFKA_ADVERTISED_HOST_NAME. This has to be changed in the docker-compose files to match your docker host IP. You can use 127.0.0.1 however this will disallow you to run Kafka in a cluster mode.

For further details please refer to https://hub.docker.com/r/wurstmeister/kafka

5.2.2.4. Running docker build outside of Maven

If you want to have the build of the jars separated from the docker build you need to provide certain VARS to docker. The following list shows all the vars and their intended value if you run docker build from the root dir

  • BUILD_DIR_ACS = Core/AtContextServer
  • BUILD_DIR_SCS = SpringCloudModules/config-server
  • BUILD_DIR_SES = SpringCloudModules/eureka
  • BUILD_DIR_SGW = SpringCloudModules/gateway
  • BUILD_DIR_HMG = History/HistoryManager
  • BUILD_DIR_QMG = Core/QueryManager
  • BUILD_DIR_RMG = Registry/RegistryManager
  • BUILD_DIR_EMG = Core/EntityManager
  • BUILD_DIR_STRMG = Storage/StorageManager
  • BUILD_DIR_SUBMG = Core/SubscriptionManager
  • JAR_FILE_BUILD_ACS = AtContextServer-${project.version}.jar
  • JAR_FILE_BUILD_SCS = config-server-${project.version}.jar
  • JAR_FILE_BUILD_SES = eureka-server-${project.version}.jar
  • JAR_FILE_BUILD_SGW = gateway-${project.version}.jar
  • JAR_FILE_BUILD_HMG = HistoryManager-${project.version}.jar
  • JAR_FILE_BUILD_QMG = QueryManager-${project.version}.jar
  • JAR_FILE_BUILD_RMG = RegistryManager-${project.version}.jar
  • JAR_FILE_BUILD_EMG = EntityManager-${project.version}.jar
  • JAR_FILE_BUILD_STRMG = StorageManager-${project.version}.jar
  • JAR_FILE_BUILD_SUBMG = SubscriptionManager-${project.version}.jar
  • JAR_FILE_RUN_ACS = AtContextServer.jar
  • JAR_FILE_RUN_SCS = config-server.jar
  • JAR_FILE_RUN_SES = eureka-server.jar
  • JAR_FILE_RUN_SGW = gateway.jar
  • JAR_FILE_RUN_HMG = HistoryManager.jar
  • JAR_FILE_RUN_QMG = QueryManager.jar
  • JAR_FILE_RUN_RMG = RegistryManager.jar
  • JAR_FILE_RUN_EMG = EntityManager.jar
  • JAR_FILE_RUN_STRMG = StorageManager.jar
  • JAR_FILE_RUN_SUBMG = SubscriptionManager.jar

5.3. Starting of the components

After the build start the individual components as normal Jar files.

Start the SpringCloud services by running

java -jar SpringCloudModules/eureka/target/eureka-server-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar SpringCloudModules/gateway/target/gateway-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar SpringCloudModules/config-server/target/config-server-<VERSIONNUMBER>-SNAPSHOT.jar

Start the broker components

java -jar Storage/StorageManager/target/StorageManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/QueryManager/target/QueryManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Registry/RegistryManager/target/RegistryManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/EntityManager/target/EntityManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar History/HistoryManager/target/HistoryManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/SubscriptionManager/target/SubscriptionManager-<VERSIONNUMBER>-SNAPSHOT.jar
java -jar Core/AtContextServer/target/AtContextServer-<VERSIONNUMBER>-SNAPSHOT.jar

5.3.1. Changing config

All configurable options are present in application.properties files. In order to change those you have two options. Either change the properties before the build or you can override configs by add --<OPTION_NAME>=<OPTION_VALUE) e.g.

java -jar Storage/StorageManager/target/StorageManager-<VERSIONNUMBER>-SNAPSHOT.jar --reader.datasource.username=funkyusername --reader.datasource.password=funkypassword`

5.3.2. Enable CORS support

You can enable cors support in the gateway by providing these configuration options - gateway.enablecors - default is False. Set to true for general enabling - gateway.enablecors.allowall - default is False. Set to true to enable CORS from all origins, allow all headers and all methods. Not secure but still very often used. - gateway.enablecors.allowedorigin - A comma separated list of allowed origins - gateway.enablecors.allowedheader - A comma separated list of allowed headers - gateway.enablecors.allowedmethods - A comma separated list of allowed methods - gateway.enablecors.allowallmethods - default is False. Set to true to allow all methods. If set to true it will override the allowmethods entry

5.4. Troubleshooting

5.4.1. Missing JAXB dependencies

When starting the eureka-server you may facing the

java.lang.TypeNotPresentException: Type javax.xml.bind.JAXBContext not present exception. It’s very likely that you are running Java 11 on your machine then. Starting from Java 9 package javax.xml.bind has been marked deprecated and was finally completely removed in Java 11.

In order to fix this issue and get eureka-server running you need to manually add below JAXB Maven dependencies to ScorpioBroker/SpringCloudModules/eureka/pom.xml before starting:

...
<dependencies>
        ...
        <dependency>
                <groupId>com.sun.xml.bind</groupId>
                <artifactId>jaxb-core</artifactId>
                <version>2.3.0.1</version>
        </dependency>
        <dependency>
                <groupId>javax.xml.bind</groupId>
                <artifactId>jaxb-api</artifactId>
                <version>2.3.1</version>
        </dependency>
        <dependency>
                <groupId>com.sun.xml.bind</groupId>
                <artifactId>jaxb-impl</artifactId>
                <version>2.3.1</version>
        </dependency>
        ...
</dependencies>
...

This should be fixed now using conditional dependencies.