From 5e596feec169d6c9a6ae4b2fb4c520c7d24a133f Mon Sep 17 00:00:00 2001
From: Fabian Schindler <fabian.schindler.strauss@gmail.com>
Date: Mon, 29 Jun 2020 11:34:32 +0200
Subject: [PATCH] Improving docs
---
.gitignore | 2 +
.../operator-guide/configuration.rst | 113 ++++++++++++++++++
documentation/operator-guide/index.rst | 1 -
documentation/operator-guide/ingestion.rst | 66 ++++++----
.../operator-guide/initialization.rst | 10 +-
documentation/operator-guide/intro.rst | 9 +-
.../operator-guide/operator-guide.rst | 1 +
7 files changed, 168 insertions(+), 34 deletions(-)
create mode 100644 documentation/operator-guide/configuration.rst
diff --git a/.gitignore b/.gitignore
index 3af0ccb6..907583ae 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1 +1,3 @@
/data
+__pycache__
+*.pyc
diff --git a/documentation/operator-guide/configuration.rst b/documentation/operator-guide/configuration.rst
new file mode 100644
index 00000000..6e34ca9f
--- /dev/null
+++ b/documentation/operator-guide/configuration.rst
@@ -0,0 +1,113 @@
+.. Configuration:
+
+Configuration
+=============
+
+This chapter details how a running VS service can be configured. And what steps
+are necessary to deploy the configuration.
+
+Concepts
+--------
+This section explains the various kinds of configuration items and where to
+apply them.
+
+Docker Compose Settings
+ These configurations are altering the behavior of the stack itself and its
+ contained services. A complete reference of the configuration file structure
+ can be found in `Docker Compose documentation
+ <https://docs.docker.com/compose/compose-file/>`_.
+
+Environment Variables
+ These variables are passed to their respective containers environment and
+ change the behavior of certain functionality. They can be declared in the
+ Docker Compose configuration file directly, but typically they are bundled by
+ field of interest and then placed into ``.env`` files and then passed to the
+ containers. So for example, there will be a ``<stack-name>_obs.env`` file
+ to store the access parameters for the object storage.
+ All those files are placed in the ``env/`` directory in the instances
+ directory.
+
+ Environment variables and ``.env`` files are passed to the services via the
+ ``docker-compose.yml`` directives. The following example shows how to pass
+ ``.env`` files and direct environment variables:
+
+ .. code-block:: yaml
+
+ services:
+ # ....
+ registrar:
+ env_file:
+ - env/stack.env
+ - env/stack_db.env
+ - env/stack_obs.env
+ - env/stack_redis.env
+ environment:
+ INSTANCE_ID: "prism-view-server_registrar"
+ INSTALL_DIR: "/var/www/pvs/dev/"
+ SCALEFACTOR: "1"
+ IN_MEMORY: "false"
+ INIT_SCRIPTS: "/configure.sh /init-db.sh"
+ # ...
+
+
+ The following ``.env`` files are typically used:
+
+ * ``<stack-name>.env``: The general ``.env`` file used for all services
+ * ``<stack-name>_db.env``: The database access credentials, for all services
+ interacting with the database.
+ * ``<stack-name>_django.env``: This env files defines the credentials for the
+ django admin user to be used with the admin GUI.
+ * ``<stack-name>_obs.env``: This contains access parameters for the object
+ storage(s).
+ * ``<stack-name>_preprocessor.env``: Preprocessor related environment
+ variables
+ * ``<stack-name>_redis.env``: Redis access credentials and queue names
+
+ .. Usually, it is not possible to override the values in running containers.
+ .. So whenever such a value is changed, the stack has to be re-deployed in
+ .. order to push the changes to the services.
+
+Configuration Files
+ Such files are passed to the containers in a similar way as environment
+ variables, but usually contain more settings at once and are placed at a
+ specific path in the container at runtime.
+
+ Configuration files are passed into the containers using the ``configs``
+ section of the ``docker-compose.yaml`` file. The following example shows how
+ such a configuration file is defined and the used in a service:
+
+
+ .. code-block:: yaml
+
+ # ...
+ configs:
+ my-config:
+ file: ./config/example.cfg
+ # ...
+ services:
+ myservice:
+ # ...
+ configs:
+ - source: my-config
+ target: /example.cfg
+
+
+ The following configuration files are used throughout the VS:
+
+ * ``<stack-name>_init-db.sh``: This shell script files purpose is to set up
+ the EOxServer instance used by both the renderer and registrar.
+ * ``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``: The
+ clients main HTML page, containing various client settings. The ``dev`` one
+ is used for development only, whereas the ``ops`` one is used for operational
+ deployment.
+ * ``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``: The
+ configuration file for MapCache, the software powering the cache service.
+ Similarly to the client configuration files, the ``dev`` and ``ops`` files
+ used for development and operational usage respectively. Further
+ documentation can be found at `the official site
+ <https://mapserver.org/mapcache/config.html>`_.
+
+
+
+
+https://gdal.org/user/configoptions.html
\ No newline at end of file
diff --git a/documentation/operator-guide/index.rst b/documentation/operator-guide/index.rst
index 8060b25b..4dd75baf 100644
--- a/documentation/operator-guide/index.rst
+++ b/documentation/operator-guide/index.rst
@@ -10,7 +10,6 @@ View Server - Operator Guide
initialization
setup
-
operator-guide
..
diff --git a/documentation/operator-guide/ingestion.rst b/documentation/operator-guide/ingestion.rst
index 2a50654a..58c49949 100644
--- a/documentation/operator-guide/ingestion.rst
+++ b/documentation/operator-guide/ingestion.rst
@@ -9,14 +9,11 @@ Redis queues
------------
The central synchronization component in the VS is the redis key-value store.
-It provides various queues, which are listened on by the services. For example,
-the ``preprocessor`` service instances listen on the ``preprocess_queue`` on
-the Redis. Whenever an item is added to the queue, it is eventually consumed by
-one of the ``preprocessor`` replicas, which performs the preprocessing. When
-completed, it pushes the processed item into the ``register_queue`` which
-itself is listened by the ``registrar``.
-
-So via the Redis, the ingestion can be triggered and observed. In order to
+It provides various queues, which are listened on by the services. For
+operators it provides a high-level interface through wich data products can be
+registered and managed.
+
+Via the Redis, the ingestion can be triggered and observed. In order to
eventually start the preprocessing of a product, its path on the configured
object storage has to be pushed onto the ``preprocess_queue``, as will be
explained in detail in this chapter.
@@ -131,12 +128,48 @@ added to the ``registering_set``, afterwards the path is placed to either the
``register-success_set`` or ``register-failure_set``. Again, these queues or
sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands.
-Data Management
----------------
+Direct Data Management
+----------------------
Sometimes it is necessary to directly interact with the registrar/renderer. The
following section shows what tasks on the registrar can be accomplished.
+.. warning::
+
+ This approach is not recommended for everyday use, as it circumvents the
+ Redis sets to track what products have been registered and where the
+ registration failed.
+
+
+Preprocessing
+~~~~~~~~~~~~~
+
+In this section all command examples are assumed to be run from a running
+preprocessor container. To open a shell on a preprocessor, the following
+command can be used.
+
+.. code-block:: bash
+
+ docker exec -it $(docker ps -qf "name=<stack-name>_preprocessor") bash
+
+The preprocessor can be used in two modes. The first (and default mode when
+used as a service) is to be run as a daemon: it listens to the Redis queue for
+new items, which will be preprocessed one by one. The second mode is to run the
+preprocessor in a "one-off" mode: instead of pulling an item from the queue,
+it is passed as a command line argument, which is then processed normally.
+
+.. code-block:: bash
+
+ python preprocessor.py ... TODO
+
+In this mode, the item will not be placed in the resulting set
+(``preprocessing_set``, ``preprocess-success_set``, and
+``preprocess-failure_set``).
+
+
+Registration Handling
+~~~~~~~~~~~~~~~~~~~~~
+
For all intents and purposes in this section it is assumed, that the operator
is logged into a shell on the ``registrar`` service. This can be achieved via
the following command (assuming at least one registrar replica is running):
@@ -154,19 +187,6 @@ alias is assumed:
alias manage.py='python3 ..../manage.py' # TODO
-Manual Data Registration
-------------------------
-
-.. warning::
-
- This approach is not recommended for production use, as it circumvents the
- Redis sets to track what products have been registered and where the
- registration failed.
-
-
-Collection Management
----------------------
-
A collection is a grouping of earth observation products, accessible as a
single entity via various service endpoints. Depending on the configuration,
multiple collections are created when the service is set up. They can be listed
diff --git a/documentation/operator-guide/initialization.rst b/documentation/operator-guide/initialization.rst
index 9477411b..6b1418fb 100644
--- a/documentation/operator-guide/initialization.rst
+++ b/documentation/operator-guide/initialization.rst
@@ -22,12 +22,12 @@ This takes the initialization configuration ``config.yaml`` to generate
the structure in the ``out/`` directory.
-Configuration
--------------
+Initialization config
+---------------------
-The important part of the initialization is of course the configuration. The
-format is structured in YAML and will be detailed here. It contains the
-following sections:
+The important part of the initialization is the configuration. The format is
+structured in YAML and will be detailed here. It contains the following
+sections:
``database``
~~~~~~~~~~~~
diff --git a/documentation/operator-guide/intro.rst b/documentation/operator-guide/intro.rst
index 41a11281..f00806ca 100644
--- a/documentation/operator-guide/intro.rst
+++ b/documentation/operator-guide/intro.rst
@@ -10,7 +10,8 @@ Since the View Server is a Docker based software and all of its components are
distributed and executed in context of Docker images and containers, a
knowledge of Docker and Docker Swarm is required.
-## Components
+Components
+----------
The View Server consists of the following service components (with their
respective Docker image in parenthesis):
@@ -29,8 +30,8 @@ respective Docker image in parenthesis):
These services are bundled and managed together in a Docker Swarm via
Docker Compose configuration files.
-Images
-------
+Docker Images
+-------------
The software is distributed as Docker images, which can be instantiated
and run in their intended role. Some images are hosted on the docker hub,
@@ -65,5 +66,3 @@ Setup
In order to help with the initial setup of a VS, the ``pvs_starter`` package
allows to quickly establish the required structure of configuration files.
-
-It is run from a configuration file and sets up
diff --git a/documentation/operator-guide/operator-guide.rst b/documentation/operator-guide/operator-guide.rst
index e7dab9fb..1378fcfd 100644
--- a/documentation/operator-guide/operator-guide.rst
+++ b/documentation/operator-guide/operator-guide.rst
@@ -6,6 +6,7 @@ Operator Guide
.. toctree::
:maxdepth: 3
+ configuration
management
ingestion
--
GitLab