Improving docs

.gitignore

 /data
+__pycache__
+*.pyc

documentation/operator-guide/configuration.rst

+.. Configuration:
+
+Configuration
+=============
+
+This chapter details how a running VS service can be configured. And what steps
+are necessary to deploy the configuration.
+
+Concepts
+--------
+This section explains the various kinds of configuration items and where to
+apply them.
+
+Docker Compose Settings
+  These configurations are altering the behavior of the stack itself and its
+  contained services. A complete reference of the configuration file structure
+  can be found in `Docker Compose documentation
+  <https://docs.docker.com/compose/compose-file/>`_.
+
+Environment Variables
+  These variables are passed to their respective containers environment and
+  change the behavior of certain functionality. They can be declared in the
+  Docker Compose configuration file directly, but typically they are bundled by
+  field of interest and then placed into ``.env`` files and then passed to the
+  containers. So for example, there will be a ``<stack-name>_obs.env`` file
+  to store the access parameters for the object storage.
+  All those files are placed in the ``env/`` directory in the instances
+  directory.
+
+  Environment variables and ``.env`` files are passed to the services via the
+  ``docker-compose.yml`` directives. The following example shows how to pass
+  ``.env`` files and direct environment variables:
+
+  .. code-block:: yaml
+
+    services:
+      # ....
+      registrar:
+        env_file:
+          - env/stack.env
+          - env/stack_db.env
+          - env/stack_obs.env
+          - env/stack_redis.env
+        environment:
+          INSTANCE_ID: "prism-view-server_registrar"
+          INSTALL_DIR: "/var/www/pvs/dev/"
+          SCALEFACTOR: "1"
+          IN_MEMORY: "false"
+          INIT_SCRIPTS: "/configure.sh /init-db.sh"
+        # ...
+
+
+  The following ``.env`` files are typically used:
+
+    * ``<stack-name>.env``: The general ``.env`` file used for all services
+    * ``<stack-name>_db.env``: The database access credentials, for all services
+      interacting with the database.
+    * ``<stack-name>_django.env``: This env files defines the credentials for the
+      django admin user to be used with the admin GUI.
+    * ``<stack-name>_obs.env``: This contains access parameters for the object
+      storage(s).
+    * ``<stack-name>_preprocessor.env``: Preprocessor related environment
+      variables
+    * ``<stack-name>_redis.env``: Redis access credentials and queue names
+
+  .. Usually, it is not possible to override the values in running containers.
+  .. So whenever such a value is changed, the stack has to be re-deployed in
+  .. order to push the changes to the services.
+
+Configuration Files
+  Such files are passed to the containers in a similar way as environment
+  variables, but usually contain more settings at once and are placed at a
+  specific path in the container at runtime.
+
+  Configuration files are passed into the containers using the ``configs``
+  section of the ``docker-compose.yaml`` file. The following example shows how
+  such a configuration file is defined and the used in a service:
+
+
+  .. code-block:: yaml
+
+    # ...
+    configs:
+      my-config:
+        file: ./config/example.cfg
+    # ...
+    services:
+      myservice:
+        # ...
+        configs:
+        - source: my-config
+          target: /example.cfg
+
+
+  The following configuration files are used throughout the VS:
+
+    * ``<stack-name>_init-db.sh``: This shell script files purpose is to set up
+      the EOxServer instance used by both the renderer and registrar.
+    * ``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``: The
+      clients main HTML page, containing various client settings. The ``dev`` one
+      is used for development only, whereas the ``ops`` one is used for operational
+      deployment.
+    * ``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``: The
+      configuration file for MapCache, the software powering the cache service.
+      Similarly to the client configuration files, the ``dev`` and ``ops`` files
+      used for development and operational usage respectively. Further
+      documentation can be found at `the official site
+      <https://mapserver.org/mapcache/config.html>`_.
+
+
+
+
+https://gdal.org/user/configoptions.html
\ No newline at end of file

documentation/operator-guide/index.rst

@@ -10,7 +10,6 @@ View Server - Operator Guide
    initialization
    setup
 
-
    operator-guide
 
 ..

documentation/operator-guide/ingestion.rst

@@ -9,14 +9,11 @@ Redis queues
 ------------
 
 The central synchronization component in the VS is the redis key-value store.
-It provides various queues, which are listened on by the services. For example,
-the ``preprocessor`` service instances listen on the ``preprocess_queue`` on
-the Redis. Whenever an item is added to the queue, it is eventually consumed by
-one of the ``preprocessor`` replicas, which performs the preprocessing. When
-completed, it pushes the processed item into the ``register_queue`` which
-itself is listened by the ``registrar``.
-
-So via the Redis, the ingestion can be triggered and observed. In order to
+It provides various queues, which are listened on by the services. For
+operators it provides a high-level interface through wich data products can be
+registered and managed.
+
+Via the Redis, the ingestion can be triggered and observed. In order to
 eventually start the preprocessing of a product, its path on the configured
 object storage has to be pushed onto the ``preprocess_queue``, as will be
 explained in detail in this chapter.
@@ -131,12 +128,48 @@ added to the ``registering_set``, afterwards the path is placed to either the
 ``register-success_set`` or ``register-failure_set``. Again, these queues or
 sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands.
 
-Data Management
----------------
+Direct Data Management
+----------------------
 
 Sometimes it is necessary to directly interact with the registrar/renderer. The
 following section shows what tasks on the registrar can be accomplished.
 
+.. warning::
+
+    This approach is not recommended for everyday use, as it circumvents the
+    Redis sets to track what products have been registered and where the
+    registration failed.
+
+
+Preprocessing
+~~~~~~~~~~~~~
+
+In this section all command examples are assumed to be run from a running
+preprocessor container. To open a shell on a preprocessor, the following
+command can be used.
+
+.. code-block:: bash
+
+    docker exec -it $(docker ps -qf "name=<stack-name>_preprocessor") bash
+
+The preprocessor can be used in two modes. The first (and default mode when
+used as a service) is to be run as a daemon: it listens to the Redis queue for
+new items, which will be preprocessed one by one. The second mode is to run the
+preprocessor in a "one-off" mode: instead of pulling an item from the queue,
+it is passed as a command line argument, which is then processed normally.
+
+.. code-block:: bash
+
+    python preprocessor.py ... TODO
+
+In this mode, the item will not be placed in the resulting set
+(``preprocessing_set``, ``preprocess-success_set``, and
+``preprocess-failure_set``).
+
+
+Registration Handling
+~~~~~~~~~~~~~~~~~~~~~
+
 For all intents and purposes in this section it is assumed, that the operator
 is logged into a shell on the ``registrar`` service. This can be achieved via
 the following command (assuming at least one registrar replica is running):
@@ -154,19 +187,6 @@ alias is assumed:
     alias manage.py='python3 ..../manage.py' # TODO
 
 
-Manual Data Registration
-------------------------
-
-.. warning::
-
-    This approach is not recommended for production use, as it circumvents the
-    Redis sets to track what products have been registered and where the
-    registration failed.
-
-
-Collection Management
----------------------
-
 A collection is a grouping of earth observation products, accessible as a
 single entity via various service endpoints. Depending on the configuration,
 multiple collections are created when the service is set up. They can be listed

documentation/operator-guide/initialization.rst

@@ -22,12 +22,12 @@ This takes the initialization configuration ``config.yaml`` to generate
 the structure in the ``out/`` directory.
 
 
-Configuration
--------------
+Initialization config
+---------------------
 
-The important part of the initialization is of course the configuration. The
-format is structured in YAML and will be detailed here. It contains the
-following sections:
+The important part of the initialization is the configuration. The format is
+structured in YAML and will be detailed here. It contains the following
+sections:
 
 ``database``
 ~~~~~~~~~~~~

documentation/operator-guide/intro.rst

@@ -10,7 +10,8 @@ Since the View Server is a Docker based software and all of its components are
 distributed and executed in context of Docker images and containers, a
 knowledge of Docker and Docker Swarm is required.
 
-## Components
+Components
+----------
 
 The View Server consists of the following service components (with their
 respective Docker image in parenthesis):
@@ -29,8 +30,8 @@ respective Docker image in parenthesis):
 These services are bundled and managed together in a Docker Swarm via
 Docker Compose configuration files.
 
-Images
-------
+Docker Images
+-------------
 
 The software is distributed as Docker images, which can be instantiated
 and run in their intended role. Some images are hosted on the docker hub,
@@ -65,5 +66,3 @@ Setup
 
 In order to help with the initial setup of a VS, the ``pvs_starter`` package
 allows to quickly establish the required structure of configuration files.
-
-It is run from a configuration file and sets up

documentation/operator-guide/operator-guide.rst

@@ -6,6 +6,7 @@ Operator Guide
 .. toctree::
    :maxdepth: 3
 
+   configuration
    management
    ingestion