configuration.rst

.. Configuration:

Configuration
=============

This chapter details how a running VS service can be configured. And what steps
are necessary to deploy the configuration.

In the following sections, it is indicated, that for an update of the
configuration to take effect some steps need to be taken.

As will be further described, for some configurations it is only required to
"re-deploy" the stack and re-start the services. This is done using the
following commands:

.. code-block:: bash

  # scale all services to 0 replicas, effectively pausing the stack
  docker service scale ...

  # re-deploy the stack (update its configuration)
  docker stack deploy -c ...

  # scale the services to their original replica count
  docker service scale

.. warning::

  When calling the ``docker stack deploy`` command, it is vital to use the
  command with that the stack was originally created.


In some cases this is not enough, as the configuration was used to for a
metarialized instance which needs to be reverted. The easiest way to do this is
to delete the volume in question. If, for example, the renderer/registrar
configuration was updated, the ``instance-data`` volume needs to be re-created.

First, the stack needs to be shut down. This is done using the following
command:

.. code-block:: bash

  docker stack rm <stack-name>


When that command has completed (it is advisable to wait for some time until
all containers have actually stopped) the next step is to delete the
``instance-data`` volume:

.. code-block:: bash

  docker volume rm <stack-name>_instance-data

.. note::

  It is possible that this command fails, with the error message that the
  volume is still in use. In this case, it is advisable to wait for a minute
  and to try the deletion again.

Now that the volume was deleted, the stack can be re-deployed, which will
trigger the automatic re-creation and initialization of the volume. For the
``instance-data``, it means that the instance will be re-created and all
database models with it.


Docker Compose Settings
-----------------------

These configurations are altering the behavior of the stack itself and its
contained services. A complete reference of the configuration file structure
can be found in `Docker Compose documentation
<https://docs.docker.com/compose/compose-file/>`_.

Environment Variables
---------------------

These variables are passed to their respective containers environment and
change the behavior of certain functionality. They can be declared in the
Docker Compose configuration file directly, but typically they are bundled by
field of interest and then placed into ``.env`` files and then passed to the
containers. So for example, there will be a ``<stack-name>_obs.env`` file
to store the access parameters for the object storage.
All those files are placed in the ``env/`` directory in the instances
directory.

Environment variables and ``.env`` files are passed to the services via the
``docker-compose.yml`` directives. The following example shows how to pass
``.env`` files and direct environment variables:

.. code-block:: yaml

  services:
    # ....
    registrar:
      env_file:
        - env/stack.env
        - env/stack_db.env
        - env/stack_obs.env
        - env/stack_redis.env
      environment:
        INSTANCE_ID: "prism-view-server_registrar"
        INSTALL_DIR: "/var/www/pvs/dev/"
        SCALEFACTOR: "1"
        IN_MEMORY: "false"
        INIT_SCRIPTS: "/configure.sh /init-db.sh"
      # ...

The following ``.env`` files are typically used:

  * ``<stack-name>.env``: The general ``.env`` file used for all services
  * ``<stack-name>_db.env``: The database access credentials, for all services
    interacting with the database.
  * ``<stack-name>_django.env``: This env files defines the credentials for the
    django admin user to be used with the admin GUI.
  * ``<stack-name>_obs.env``: This contains access parameters for the object
    storage(s).
  * ``<stack-name>_preprocessor.env``: Preprocessor related environment
    variables
  * ``<stack-name>_redis.env``: Redis access credentials and queue names


Groups of environment variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

GDAL environment variables
  This group of environment variables controls the intricacies of GDAL. They
  control how GDAL interacts with its supported files. As GDAL supports a
  variety of formats and backend access, most of the full `list of env
  variables <https://gdal.org/user/configoptions.html>`_ are not applicable and
  only a handful are actually relevant for VS.

  ``GDAL_DISABLE_READDIR_ON_OPEN``
    Especially when using an Object Storage backend with a very large number of
    files, it is vital to activate this setting (``=TRUE``) in order to
    suppress to read the whole directory contents which is very slow for some
    OBS backends.
  ``CPL_VSIL_CURL_ALLOWED_EXTENSIONS``
    This limits the file extensions to disable the lookup of so called sidecar
    files which are not used for VS. By default this value is used:
    ``=.TIF,.tif,.xml``.

OpenStack Swift environment variables
  These variables define the access coordinates and credentials for the
  OpenStack Swift Object storage backend.

  ``ST_AUTH_VERSION``
  ``OS_AUTH_URL_SHORT``
  ``OS_AUTH_URL``
  ``OS_USERNAME``
  ``OS_PASSWORD``
  ``OS_TENANT_NAME``
  ``OS_TENANT_ID``
  ``OS_REGION_NAME``
  ``OS_USER_DOMAIN_NAME``
    This set of variables define the credentials for the object storage to
    place the preprocessed results.


  ``OS_USERNAME_DOWNLOAD``
  ``OS_PASSWORD_DOWNLOAD``
  ``OS_TENANT_NAME_DOWNLOAD``
  ``OS_TENANT_ID_DOWNLOAD``
  ``OS_REGION_NAME_DOWNLOAD``
  ``OS_AUTH_URL_DOWNLOAD``
  ``ST_AUTH_VERSION_DOWNLOAD``
    This set of variables define the credentials for the object storage to
    retrieve the original product files.

VS environment variables
  These environment variables are used by VS itself to configure various parts.

  .. note::
    These variables are used during the initial stack setup. When these
    variables are changed, they will not be reflected unless the instance
    volume is re-created.

  ``COLLECTION``
    This defines the main collections name. This is used in various parts of
    the VS and serves as the layer base name.
  ``UPLOAD_CONTAINER``
    This controls the bucket name where the preprocessed images are uploaded
    to.
  ``DJANGO_USER``, ``DJANGO_MAIL``, ``DJANGO_PASSWORD``
    The Django admin user account credentials to use the Admin GUI.

  .. note::
    These variables are used during the initial stack setup. When these
    variables are changed, they will not be reflected unless the database
    volume is re-created.

  ``POSTGRES_USER``
  ``POSTGRES_PASSWORD``
  ``POSTGRES_DB``
  ``DB``
  ``DB_USER``
  ``DB_PW``
  ``DB_HOST``
  ``DB_PORT``
  ``DB_NAME``
    These are the internal access credentials for the database.


Configuration Files
-------------------

Such files are passed to the containers in a similar way as environment
variables, but usually contain more settings at once and are placed at a
specific path in the container at runtime.

Configuration files are passed into the containers using the ``configs``
section of the ``docker-compose.yaml`` file. The following example shows how
such a configuration file is defined and the used in a service:


.. code-block:: yaml

  # ...
  configs:
    my-config:
      file: ./config/example.cfg
  # ...
  services:
    myservice:
      # ...
      configs:
      - source: my-config
        target: /example.cfg


The following configuration files are used throughout the VS:

  * ``<stack-name>_init-db.sh``: This shell script files purpose is to set up
    the EOxServer instance used by both the renderer and registrar.
  * ``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``: The
    clients main HTML page, containing various client settings. The ``dev`` one
    is used for development only, whereas the ``ops`` one is used for operational
    deployment.
  * ``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``: The
    configuration file for MapCache, the software powering the cache service.
    Similarly to the client configuration files, the ``dev`` and ``ops`` files
    used for development and operational usage respectively. Further
    documentation can be found at `the official site
    <https://mapserver.org/mapcache/config.html>`_.