From 6b291e1b5aa08f54ea8e024bef5c74041da7edf7 Mon Sep 17 00:00:00 2001 From: Fabian Schindler <fabian.schindler.strauss@gmail.com> Date: Mon, 29 Jun 2020 16:01:23 +0200 Subject: [PATCH] Updating configuration section --- .../operator-guide/configuration.rst | 310 +++++++++++++----- 1 file changed, 221 insertions(+), 89 deletions(-) diff --git a/documentation/operator-guide/configuration.rst b/documentation/operator-guide/configuration.rst index 6e34ca9f..4d40c695 100644 --- a/documentation/operator-guide/configuration.rst +++ b/documentation/operator-guide/configuration.rst @@ -6,108 +6,240 @@ Configuration This chapter details how a running VS service can be configured. And what steps are necessary to deploy the configuration. -Concepts --------- -This section explains the various kinds of configuration items and where to -apply them. +In the following sections, it is indicated, that for an update of the +configuration to take effect some steps need to be taken. + +As will be further described, for some configurations it is only required to +"re-deploy" the stack and re-start the services. This is done using the +following commands: + +.. code-block:: bash + + # scale all services to 0 replicas, effectively pausing the stack + docker service scale ... + + # re-deploy the stack (update its configuration) + docker stack deploy -c ... + + # scale the services to their original replica count + docker service scale + +.. warning:: + + When calling the ``docker stack deploy`` command, it is vital to use the + command with that the stack was originally created. + + +In some cases this is not enough, as the configuration was used to for a +metarialized instance which needs to be reverted. The easiest way to do this is +to delete the volume in question. If, for example, the renderer/registrar +configuration was updated, the ``instance-data`` volume needs to be re-created. + +First, the stack needs to be shut down. This is done using the following +command: + +.. code-block:: bash + + docker stack rm <stack-name> + + +When that command has completed (it is advisable to wait for some time until +all containers have actually stopped) the next step is to delete the +``instance-data`` volume: + +.. code-block:: bash + + docker volume rm <stack-name>_instance-data + +.. note:: + + It is possible that this command fails, with the error message that the + volume is still in use. In this case, it is advisable to wait for a minute + and to try the deletion again. + +Now that the volume was deleted, the stack can be re-deployed, which will +trigger the automatic re-creation and initialization of the volume. For the +``instance-data``, it means that the instance will be re-created and all +database models with it. + Docker Compose Settings - These configurations are altering the behavior of the stack itself and its - contained services. A complete reference of the configuration file structure - can be found in `Docker Compose documentation - <https://docs.docker.com/compose/compose-file/>`_. +----------------------- + +These configurations are altering the behavior of the stack itself and its +contained services. A complete reference of the configuration file structure +can be found in `Docker Compose documentation +<https://docs.docker.com/compose/compose-file/>`_. Environment Variables - These variables are passed to their respective containers environment and - change the behavior of certain functionality. They can be declared in the - Docker Compose configuration file directly, but typically they are bundled by - field of interest and then placed into ``.env`` files and then passed to the - containers. So for example, there will be a ``<stack-name>_obs.env`` file - to store the access parameters for the object storage. - All those files are placed in the ``env/`` directory in the instances - directory. - - Environment variables and ``.env`` files are passed to the services via the - ``docker-compose.yml`` directives. The following example shows how to pass - ``.env`` files and direct environment variables: - - .. code-block:: yaml - - services: - # .... - registrar: - env_file: - - env/stack.env - - env/stack_db.env - - env/stack_obs.env - - env/stack_redis.env - environment: - INSTANCE_ID: "prism-view-server_registrar" - INSTALL_DIR: "/var/www/pvs/dev/" - SCALEFACTOR: "1" - IN_MEMORY: "false" - INIT_SCRIPTS: "/configure.sh /init-db.sh" - # ... - - - The following ``.env`` files are typically used: - - * ``<stack-name>.env``: The general ``.env`` file used for all services - * ``<stack-name>_db.env``: The database access credentials, for all services - interacting with the database. - * ``<stack-name>_django.env``: This env files defines the credentials for the - django admin user to be used with the admin GUI. - * ``<stack-name>_obs.env``: This contains access parameters for the object - storage(s). - * ``<stack-name>_preprocessor.env``: Preprocessor related environment - variables - * ``<stack-name>_redis.env``: Redis access credentials and queue names - - .. Usually, it is not possible to override the values in running containers. - .. So whenever such a value is changed, the stack has to be re-deployed in - .. order to push the changes to the services. +--------------------- + +These variables are passed to their respective containers environment and +change the behavior of certain functionality. They can be declared in the +Docker Compose configuration file directly, but typically they are bundled by +field of interest and then placed into ``.env`` files and then passed to the +containers. So for example, there will be a ``<stack-name>_obs.env`` file +to store the access parameters for the object storage. +All those files are placed in the ``env/`` directory in the instances +directory. + +Environment variables and ``.env`` files are passed to the services via the +``docker-compose.yml`` directives. The following example shows how to pass +``.env`` files and direct environment variables: + +.. code-block:: yaml + + services: + # .... + registrar: + env_file: + - env/stack.env + - env/stack_db.env + - env/stack_obs.env + - env/stack_redis.env + environment: + INSTANCE_ID: "prism-view-server_registrar" + INSTALL_DIR: "/var/www/pvs/dev/" + SCALEFACTOR: "1" + IN_MEMORY: "false" + INIT_SCRIPTS: "/configure.sh /init-db.sh" + # ... + +The following ``.env`` files are typically used: + + * ``<stack-name>.env``: The general ``.env`` file used for all services + * ``<stack-name>_db.env``: The database access credentials, for all services + interacting with the database. + * ``<stack-name>_django.env``: This env files defines the credentials for the + django admin user to be used with the admin GUI. + * ``<stack-name>_obs.env``: This contains access parameters for the object + storage(s). + * ``<stack-name>_preprocessor.env``: Preprocessor related environment + variables + * ``<stack-name>_redis.env``: Redis access credentials and queue names + + +Groups of environment variables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +GDAL environment variables + This group of environment variables controls the intricacies of GDAL. They + control how GDAL interacts with its supported files. As GDAL supports a + variety of formats and backend access, most of the full `list of env + variables <https://gdal.org/user/configoptions.html>`_ are not applicable and + only a handful are actually relevant for VS. + + ``GDAL_DISABLE_READDIR_ON_OPEN`` + Especially when using an Object Storage backend with a very large number of + files, it is vital to activate this setting (``=TRUE``) in order to + suppress to read the whole directory contents which is very slow for some + OBS backends. + ``CPL_VSIL_CURL_ALLOWED_EXTENSIONS`` + This limits the file extensions to disable the lookup of so called sidecar + files which are not used for VS. By default this value is used: + ``=.TIF,.tif,.xml``. + +OpenStack Swift environment variables + These variables define the access coordinates and credentials for the + OpenStack Swift Object storage backend. + + ``ST_AUTH_VERSION`` + ``OS_AUTH_URL_SHORT`` + ``OS_AUTH_URL`` + ``OS_USERNAME`` + ``OS_PASSWORD`` + ``OS_TENANT_NAME`` + ``OS_TENANT_ID`` + ``OS_REGION_NAME`` + ``OS_USER_DOMAIN_NAME`` + This set of variables define the credentials for the object storage to + place the preprocessed results. + + + ``OS_USERNAME_DOWNLOAD`` + ``OS_PASSWORD_DOWNLOAD`` + ``OS_TENANT_NAME_DOWNLOAD`` + ``OS_TENANT_ID_DOWNLOAD`` + ``OS_REGION_NAME_DOWNLOAD`` + ``OS_AUTH_URL_DOWNLOAD`` + ``ST_AUTH_VERSION_DOWNLOAD`` + This set of variables define the credentials for the object storage to + retrieve the original product files. + +VS environment variables + These environment variables are used by VS itself to configure various parts. + + .. note:: + These variables are used during the initial stack setup. When these + variables are changed, they will not be reflected unless the instance + volume is re-created. + + ``COLLECTION`` + This defines the main collections name. This is used in various parts of + the VS and serves as the layer base name. + ``UPLOAD_CONTAINER`` + This controls the bucket name where the preprocessed images are uploaded + to. + ``DJANGO_USER``, ``DJANGO_MAIL``, ``DJANGO_PASSWORD`` + The Django admin user account credentials to use the Admin GUI. + + .. note:: + These variables are used during the initial stack setup. When these + variables are changed, they will not be reflected unless the database + volume is re-created. + + ``POSTGRES_USER`` + ``POSTGRES_PASSWORD`` + ``POSTGRES_DB`` + ``DB`` + ``DB_USER`` + ``DB_PW`` + ``DB_HOST`` + ``DB_PORT`` + ``DB_NAME`` + These are the internal access credentials for the database. -Configuration Files - Such files are passed to the containers in a similar way as environment - variables, but usually contain more settings at once and are placed at a - specific path in the container at runtime. - Configuration files are passed into the containers using the ``configs`` - section of the ``docker-compose.yaml`` file. The following example shows how - such a configuration file is defined and the used in a service: +Configuration Files +------------------- +Such files are passed to the containers in a similar way as environment +variables, but usually contain more settings at once and are placed at a +specific path in the container at runtime. - .. code-block:: yaml +Configuration files are passed into the containers using the ``configs`` +section of the ``docker-compose.yaml`` file. The following example shows how +such a configuration file is defined and the used in a service: - # ... - configs: - my-config: - file: ./config/example.cfg - # ... - services: - myservice: - # ... - configs: - - source: my-config - target: /example.cfg +.. code-block:: yaml - The following configuration files are used throughout the VS: + # ... + configs: + my-config: + file: ./config/example.cfg + # ... + services: + myservice: + # ... + configs: + - source: my-config + target: /example.cfg - * ``<stack-name>_init-db.sh``: This shell script files purpose is to set up - the EOxServer instance used by both the renderer and registrar. - * ``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``: The - clients main HTML page, containing various client settings. The ``dev`` one - is used for development only, whereas the ``ops`` one is used for operational - deployment. - * ``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``: The - configuration file for MapCache, the software powering the cache service. - Similarly to the client configuration files, the ``dev`` and ``ops`` files - used for development and operational usage respectively. Further - documentation can be found at `the official site - <https://mapserver.org/mapcache/config.html>`_. +The following configuration files are used throughout the VS: + * ``<stack-name>_init-db.sh``: This shell script files purpose is to set up + the EOxServer instance used by both the renderer and registrar. + * ``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``: The + clients main HTML page, containing various client settings. The ``dev`` one + is used for development only, whereas the ``ops`` one is used for operational + deployment. + * ``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``: The + configuration file for MapCache, the software powering the cache service. + Similarly to the client configuration files, the ``dev`` and ``ops`` files + used for development and operational usage respectively. Further + documentation can be found at `the official site + <https://mapserver.org/mapcache/config.html>`_. -https://gdal.org/user/configoptions.html \ No newline at end of file -- GitLab