setup.rst

.. _setup:

Setup
=====

In this chapter the setup of a new VS stack is detailed. Before this step can
be done, the configuration and environment files need to be present. These
files can be added manually or be created as described in the
:ref:`initialization` step.


Docker
------

In order to deploy the Docker Swarm stack to the target machine, Docker and
its facilities need to be installed. This step depends on the systems
architecture. On a Debian based system this may look like this:

.. code-block:: bash

    sudo apt-get install \
        apt-transport-https \
        ca-certificates \
        curl \
        gnupg-agent \
        software-properties-common

    curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

    # add the apt repository
    sudo add-apt-repository \
        "deb [arch=amd64] https://download.docker.com/linux/debian \
        $(lsb_release -cs) \
        stable"

    # fetch the package index and install Docker
    sudo apt-get update
    sudo apt-get install docker-ce docker-ce-cli containerd.io

.. ## TODO: RHEL?

Docker Swarm
------------

Now that Docker is installed, the machine can either create a new swarm or join
an existing one.

To create a new Swarm, the following command is used:

.. code-block:: bash

    docker swarm init


Alternatively, an existing Swarm can be joined. The easiest way to do this, is
to obtain a ``join-token``. On an existing Swarm manager (where a Swarm was
initialized or already joined as manager) run this command:


.. code-block:: bash

    docker swarm join-token worker


This prints out a command that can be run on a machine to join the swarm:

.. code-block:: bash

    docker swarm join --token <obtained token>

It is possible to dedicate certain workers for example to contribute to ingestion exclusively, while others can take care only for rendering. This setup has benefits, when a mixed setup of nodes with different parameters is available.

In order to set a node for example as `external`, to contribute in rendering only, one can simply run: 

.. code-block:: bash

    docker node update --label-add type=external <node-id>
    
Additionally, it is necessary to modify `placement` parameter in the docker compose file.

.. code-block:: yaml

  renderer:
    deploy:
      placement:
        constraints:
          - node.labels.type == external

Additional information for swarm management can be obtained in the official
`documentation of the project
<https://docs.docker.com/engine/reference/commandline/swarm/>`_.


Docker Image Retrieval
----------------------

Before the Docker images can be used, they have to be retrieved first. With
images from the default repository, this happens automatically. When private
repositories are used, they need to be configured beforehand.
Currently, all images used in VS that are not off-the-shelf are hosted on the
``registry.gitlab.eox.at`` registry. It can be configured to be used with this
command with the correct ``username`` and ``password`` filled in:

.. code-block:: bash

    docker login -u <username> -p <password> registry.gitlab.eox.at

Now the relevant images can be pulled:

.. code-block:: bash

    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_core
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_cache
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor
    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_client

.. # TODO: ingestor image?

Stack Deployment
----------------

Now that a Docker Swarm is established, it is time to deploy the VS as a stack.
This is done using the created Docker Compose configuration files. In order to
enhance the re-usability, these files are split into multiple parts to be used
for both development and final service deployment.

For a development deployment one would do (replace ``name`` with the actual
service identifier:

.. code-block:: bash

    docker stack deploy -c docker-compose.<name>.yml -c docker-compose.<name>.dev.yml <stack-name>


This command actually performs a variety of tasks. First off, it obtains any
missing images, such as the image for the reverse proxy, the database, or the
redis key-value store.

When all relevant images are pulled from their respective repository the
services of the stack are initialized. In the default setting, each service is
represented by a single container of its respective service type. When starting
for the first time, the startup procedure takes some time, as everything needs
to be initialized. This includes the creation of the database, user,
required tables, and the Django instance.

That process can be supervised using the ``docker service ls`` command, which
lists all available services and their respective status.

Continue to the next section :ref:`configuration` to read how a running VS
stack can be configured.