configuration.rst

.. _configuration:

Configuration
=============

This chapter details how a running VS stack can be configured. And what steps
are necessary to deploy the configuration.

In order for these configuration changes to be picked up by a running VS
stack and to take effect some steps need to be performed. These steps are
either a "re-deploy" of the running stack or a complete re-creation of it.

Stack Re-deploy
---------------

As will be further described, for some configurations it is sufficient to
"re-deploy" the stack which automatically re-starts any service with changed
configuration. This is done re-using the stack deployment command:

.. code-block:: bash

  docker stack deploy -c docker-compose.<name>.yml -c docker-compose.<name>.dev.yml <stack-name>

.. warning::

  When calling the ``docker stack deploy`` command, it is vital to use the
  command with the same files and name the stack was originally created.

Stack Re-creation
-----------------

In some cases a stack re-redeploy is not enough, as the configuration was used
for a materialized instance which needs to be reverted. The easiest way to do
this is to delete the volume in question. If, for example, the
renderer/registrar configuration was updated, the ``instance-data`` volume
needs to be re-created.

First, the stack needs to be shut down. This is done using the following
command:

.. code-block:: bash

  docker stack rm <stack-name>


When that command has completed (it is advisable to wait for some time until
all containers have actually stopped) the next step is to delete the
``instance-data`` volume:

.. code-block:: bash

  docker volume rm <stack-name>_instance-data

.. note::

  It is possible that this command fails, with the error message that the
  volume is still in use. In this case, it is advisable to wait for a minute
  and to try the deletion again.

Now that the volume was deleted, the stack can be re-deployed as described
above, which will trigger the automatic re-creation and initialization of the
volume. For the ``instance-data``, it means that the instance will be
re-created and all database models with it.


Docker Compose Settings
-----------------------

These configurations are altering the behavior of the stack itself and its
contained services. A complete reference of the configuration file structure
can be found in the `Docker Compose documentation
<https://docs.docker.com/compose/compose-file/>`_.

Environment Variables
---------------------

These variables are passed to their respective containers environment and
change the behavior of certain functionality. They can be declared in the
Docker Compose configuration file directly, but typically they are bundled by
field of interest and then placed into ``.env`` files and then passed to the
containers. So for example, there will be a ``<stack-name>_obs.env`` file
to store the access parameters for the object storage.
All those files are placed in the ``env/`` directory in the instances
directory.

Environment variables and ``.env`` files are passed to the services via the
``docker-compose.yml`` directives. The following example shows how to pass
``.env`` files and direct environment variables:

.. code-block:: yaml

  services:
    # ....
    registrar:
      env_file:
        - env/stack.env
        - env/stack_db.env
        - env/stack_obs.env
        - env/stack_redis.env
      environment:
        INSTANCE_ID: "prism-view-server_registrar"
        INSTALL_DIR: "/var/www/pvs/dev/"
        SCALEFACTOR: "1"
        IN_MEMORY: "false"
        INIT_SCRIPTS: "/configure.sh /init-db.sh"
      # ...

``.env`` Files
~~~~~~~~~~~~~~

The following ``.env`` files are typically used:

* ``<stack-name>.env``: The general ``.env`` file used for all services
* ``<stack-name>_db.env``: The database access credentials, for all services
  interacting with the database.
* ``<stack-name>_django.env``: This env files defines the credentials for the
  django admin user to be used with the admin GUI.
* ``<stack-name>_obs.env``: This contains access parameters for the object
  storage(s).
* ``<stack-name>_preprocessor.env``: Preprocessor related environment
  variables
* ``<stack-name>_redis.env``: Redis access credentials and queue names


Groups of Environment Variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

GDAL Environment Variables
^^^^^^^^^^^^^^^^^^^^^^^^^^

This group of environment variables controls the intricacies of GDAL. They
control how GDAL interacts with its supported files. As GDAL supports a
variety of formats and backend access, most of the full `list of env
variables <https://gdal.org/user/configoptions.html>`_ are not applicable and
only a handful are actually relevant for the VS.

* ``GDAL_DISABLE_READDIR_ON_OPEN`` -
  Especially when using an Object Storage backend with a very large number of
  files, it is vital to activate this setting (``=TRUE``) in order to
  suppress to read the whole directory contents which is very slow for some
  OBS backends.
* ``CPL_VSIL_CURL_ALLOWED_EXTENSIONS`` -
  This limits the file extensions to disable the lookup of so called sidecar
  files which are not used for VS. By default this value is used:
  ``=.TIF,.tif,.xml``.

OpenStack Swift Environment Variables
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These variables define the access coordinates and credentials for the
OpenStack Swift Object storage backend.

This set of variables define the credentials for the object storage to
place the preprocessed results:

* ``ST_AUTH_VERSION``
* ``OS_AUTH_URL_SHORT``
* ``OS_AUTH_URL``
* ``OS_USERNAME``
* ``OS_PASSWORD``
* ``OS_TENANT_NAME``
* ``OS_TENANT_ID``
* ``OS_REGION_NAME``
* ``OS_USER_DOMAIN_NAME``

This set of variables define the credentials for the object storage to
retrieve the original product files:

* ``OS_USERNAME_DOWNLOAD``
* ``OS_PASSWORD_DOWNLOAD``
* ``OS_TENANT_NAME_DOWNLOAD``
* ``OS_TENANT_ID_DOWNLOAD``
* ``OS_REGION_NAME_DOWNLOAD``
* ``OS_AUTH_URL_DOWNLOAD``
* ``ST_AUTH_VERSION_DOWNLOAD``

VS Environment Variables
^^^^^^^^^^^^^^^^^^^^^^^^

These environment variables are used by the VS itself to configure various
parts.

.. note::
  These variables are used during the initial stack setup. When these
  variables are changed, they will not be reflected unless the instance
  volume is re-created.

* ``COLLECTION`` -
  This defines the main collections name. This is used in various parts of
  the VS and serves as the layer base name.
* ``UPLOAD_CONTAINER`` -
  This controls the bucket name where the preprocessed images are uploaded
  to.
* ``DJANGO_USER``, ``DJANGO_MAIL``, ``DJANGO_PASSWORD`` -
  The Django admin user account credentials to use the Admin GUI.

.. note::
  These variables are used during the initial stack setup. When these
  variables are changed, they will not be reflected unless the database
  volume is re-created.

These are the internal access credentials for the database:

* ``POSTGRES_USER``
* ``POSTGRES_PASSWORD``
* ``POSTGRES_DB``
* ``DB``
* ``DB_USER``
* ``DB_PW``
* ``DB_HOST``
* ``DB_PORT``
* ``DB_NAME``

Configuration Files
-------------------

Such files are passed to the containers in a similar way as environment
variables, but usually contain more settings at once and are placed at a
specific path in the container at runtime.

Configuration files are passed into the containers using the ``configs``
section of the ``docker-compose.yaml`` file. The following example shows how
such a configuration file is defined and the used in a service:


.. code-block:: yaml

  # ...
  configs:
    my-config:
      file: ./config/example.cfg
  # ...
  services:
    myservice:
      # ...
      configs:
      - source: my-config
        target: /example.cfg


The following configuration files are used throughout the VS:

* ``<stack-name>_init-db.sh``: This shell script file's purpose is to set up
  the EOxServer instance used by both the renderer and registrar.
* ``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``: The
  clients main HTML page, containing various client settings. The ``dev`` one
  is used for development only, whereas the ``ops`` one is used for operational
  deployment.
* ``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``: The
  configuration file for MapCache, the software powering the cache service.
  Similarly to the client configuration files, the ``dev`` and ``ops`` files
  used for development and operational usage respectively. Further
  documentation can be found at `the official site
  <https://mapserver.org/mapcache/config.html>`_.

The next section :ref:`management` describes how an operator interacts with a
deployed VS stack.