Adding additional documentation

documentation/operator-guide/conf.py

@@ -27,8 +27,6 @@ author = u'EOX IT Services GmbH'
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'recommonmark',
-
 ]
 
 # Add any paths that contain templates here, relative to this directory.

documentation/operator-guide/ingestion.rst

+.. _ingestion:
+
+Data Ingestion
+==============
+
+This section details the data ingestion and later management in the VS.
+
+Redis queues
+------------
+
+The central synchronization component in the VS is the redis key-value store.
+It provides various queues, which are listened on by the services. For example,
+the ``preprocessor`` service instances listen on the ``preprocess_queue`` on
+the Redis. Whenever an item is added to the queue, it is eventually consumed by
+one of the ``preprocessor`` replicas, which performs the preprocessing. When
+completed, it pushes the processed item into the ``register_queue`` which
+itself is listened by the ``registrar``.
+
+So via the Redis, the ingestion can be triggered and observed. In order to
+eventually start the preprocessing of a product, its path on the configured
+object storage has to be pushed onto the ``preprocess_queue``, as will be
+explained in detail in this chapter.
+
+As the Redis store is not publicly accessible from outside of the stack. So to
+interact with it, the operator has to run a command from one of the services.
+Conveniently, the service running Redis also has the ``redis-cli`` tool
+installed that lets users interact with the store.
+
+When doing one off commands, it is maybe more convenient to execute it on a
+running service. For this, the ``docker ps`` command can be used to select the
+identifier of the running docker container of the redis service.
+
+.. code-block:: bash
+
+    container_id=$(docker ps -qf "name=<stack-name>_redis")
+
+With this identifier, a command can be issued:
+
+.. code-block:: bash
+
+    docker exec -it $container_id redis-cli ...
+
+When performing more than one command, it can be simpler to open a shell on the
+service instead:
+
+.. code-block:: bash
+
+    docker exec -it $container_id bash
+
+As the container ID may change (for example when the replica is restarted) it
+is better to retrieve it for every command instead of relying on a variable:
+
+.. code-block:: bash
+
+    docker exec -it $(docker ps -qf "name=<stack-name>_redis")
+
+For the sake of brevity, the next commands in this chaptere are using either of
+the above techniques and will just print the final commands inside the redis
+container.
+
+.. note::
+
+    For the VS, only the ``List``  and ``Set`` `Redis data types
+    <https://redis.io/topics/data-types>`_ are really used. ``Sets`` are an
+    unordered collection of string elements. In VS it is used to denote that an
+    element is part of a particular group, e.g: being preprocessed, or having
+    failed registration.
+
+    ``Lists`` are used as a task queue. It is possible to add items to eithre
+    end of the queue, but by convention items are pushed on the "left" and
+    popped from the "right" end of the list resulting in a last-in-first-out
+    (LIFO) queue. It is entirely possible to push elements to the "right" end
+    as-well, and an operator may want to do so in order to add an element to be
+    processed as soon as possible instead of waiting before all other elements
+    before it are processed.
+
+    The full list of available commands can be found for both `Lists
+    <https://redis.io/commands#list>`_ and `Sets
+    <https://redis.io/commands#set>`_.
+
+For a more concrete example: the following command finds the container ID of
+the redis service replica, and executes a ``redis-cli lpush`` command to add a
+new path of an object to preprocess on the ``preprocess_queue``:
+
+.. code-block:: bash
+
+    redis-cli lpush preprocess_queue "/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar"
+
+Usually, with a preprocessor service running and no other items in the
+``preprocess_queue`` this value will be immediatly popped from the list and
+processed. For the sake of demonstration this command would print the contents
+of the ``preprocess_queue``:
+
+.. code-block:: bash
+
+    $ redis-cli lrange preprocess_queue 0 -1
+    /data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar
+
+Now that the product is beeing preprocessed, it should be visible in the
+``preprocessing_set``. As the name indicates, this is using the ``Set``
+datatype, thus requiring the ``SMEMBERS`` subcommand to list:
+
+.. code-block:: bash
+
+    $ redis-cli smembers preprocessing_set 0 -1
+    /data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar
+
+Once the preprocessing of the product is finished, the preprocessor will remove
+the currently worked on path from the ``preprocessing_set`` and add it either
+to the ``preprocess-success_set`` or the ``preprocess-failure_set`` depending
+on whether the processing succeeded or not. They can be inspected using the
+same ``SMEMBERS`` subcommand but either name as parameter.
+
+Additionally, upon success, the preprocessor places the same product path on
+the ``register_queue``, where it can be inspected with the following command.
+
+.. code-block:: bash
+
+    $ redis-cli lrange register_queue 0 -1
+    /data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar
+
+If an operator wants to trigger the re-registration of a product only the
+product path needs to be pushed to this queue:
+
+.. code-block:: bash
+
+    redis-cli lpush register_queue "/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar"
+
+Very similar to the preprocessing, during the registration the product path is
+added to the ``registering_set``, afterwards the path is placed to either the
+``register-success_set`` or ``register-failure_set``. Again, these queues or
+sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands.
+
+Data Management
+---------------
+
+Sometimes it is necessary to directly interact with the registrar/renderer. The
+following section shows what tasks on the registrar can be accomplished.
+
+For all intents and purposes in this section it is assumed, that the operator
+is logged into a shell on the ``registrar`` service. This can be achieved via
+the following command (assuming at least one registrar replica is running):
+
+.. code-block:: bash
+
+    docker exec -it $(docker ps -qf "name=<stack-name>_registrar") bash
+
+The contents of the shared registrar/renderer database can be managed using
+the registrars instance ``manage.py`` script. For brevity, the following bash
+alias is assumed:
+
+.. code-block:: bash
+
+    alias manage.py='python3 ..../manage.py' # TODO
+
+
+Manual Data Registration
+------------------------
+
+.. warning::
+
+    This approach is not recommended for production use, as it circumvents the
+    Redis sets to track what products have been registered and where the
+    registration failed.
+
+
+Collection Management
+---------------------
+
+A collection is a grouping of earth observation products, accessible as a
+single entity via various service endpoints. Depending on the configuration,
+multiple collections are created when the service is set up. They can be listed
+using the ``collection list`` command.
+
+New collections can be created using the ``collection create`` command. This
+can refer to a ``Collection Type``, which will restrict the collection in terms
+of insertable products: only products of an allowed ``Product Type`` can be
+added. Detailed information about the available Collection management commands
+can be found in the `CLI documentation <https://docs.eoxserver.org/en/master/users/coverages.html#command-line-interfaces>`__.
+
+Collections can be deleted, without affecting the contained products.
+
+.. warning::
+
+    Since the other services have fixed configuration and depend on specific
+    collection, deleting said collections without a replacement can lead to
+    service disruptions.
+
+In certain scenarios it may be useful, to add specific products to or exclude
+them from a collection. For this, the Product identifier needs to be known. To
+find out the Product identifier, either the OpenSearch on an existing
+collection or the CLI command ``id list`` can be used.
+
+When the identifier is obtained, the following management command inserts a
+product into a collection:
+
+.. code-block:: bash
+
+    manage.py collection insert <collection-id> <product-id>
+
+Multiple products can be inserted in one pass by providing more than one
+identifier.
+
+The reverse command excludes a product from a collection:
+
+.. code-block:: bash
+
+    manage.py collection exclude <collection-id> <product-id>
+
+Again, multiple products can be excluded in a single call.
+
+
+

documentation/operator-guide/management.rst

+.. _management:
+
+Service Management
+==================
+
+This section shows how a deployed VS stack can be interacted with.
+
+
+Scaling
+-------
+
+Scaling is a handy tool to ensure stable performance, even when dealing with
+higher usage on either service. For example, the preprocessor and registrar can
+be scaled to a higher replica count to enable a better throughput when
+ingesting data into the VS.
+
+The following command scales the ``renderer`` service to 5 replicas:
+
+.. code-block:: bash
+
+    docker service scale <stack-name>_renderer=5
+
+A service can also be scaled to zero replicas, effectively disabling the
+service.
+
+.. warning::
+
+    The ``redis`` and ``database`` should never be scaled (their replica count
+    should remain 1) as this can lead to service disruptions and corrupted data.
+
+
+Updating Images
+---------------
+
+Updating the service software is done using previously established tools. To
+update the service in question, it needs to be scaled to zero replicas. Then
+the new image can be pulled, and the service can be scaled back to its original
+value. This forces the start of the service from the newly fetched image.

documentation/operator-guide/operator-guide.rst

 Operator Guide
 ==============
 
-TODO
+
+
+.. toctree::
+   :maxdepth: 3
+
+   management
+   ingestion
+
 
 
 Admin

documentation/operator-guide/setup.rst

@@ -72,3 +72,59 @@ Additional information for swarm management can be obtained in the official
 documentation of the project:
 https://docs.docker.com/engine/reference/commandline/swarm/
 
+
+Image retrieval
+---------------
+
+Before the Docker images can be used, they have to be retrieved first. With
+images from the default repository, this happens automatically. When private
+repositories are used, they need to be configured beforehand.
+Currently, all images used in VS that are not off-the-shelf are hosted on the
+``registry.gitlab.eox.at`` registry. It can be configured to be used with this
+command with the correct username and password filled in:
+
+.. code-block:: bash
+
+    docker login -u <username> -p <password> registry.gitlab.eox.at
+
+Now the relevant images can be pulled:
+
+.. code-block:: bash
+
+    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_core
+    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_cache
+    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor
+    docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_client
+
+# TODO: ingestor image?
+
+
+Stack Deployment
+----------------
+
+Now that a Docker Swarm is established, it is time to deploy the VS as a stack.
+This is done using the created Docker Compose configuration files. In order to
+enhance the re-usability, these files are split into multiple parts to be used
+for both development and final service deployment.
+
+For a development deployment one would do (replace ``name`` with the actual
+service identifier:
+
+.. code-block:: bash
+
+    docker stack deploy -c docker-compose.<name>.yml -c docker-compose.<name>.dev.yml <name>-pdas
+
+
+This command actually performs a variety of tasks. First off, it obtains any
+missing images, such as the image for the reverse proxy, the database or the
+redis key-value store.
+
+When all relevant images are pulled from their respective repository the
+services of the stack are initialized. In the default setting, each service is
+represented by a single container of its respective service type. When starting
+for the first time, the startup procedure takes some time, as everything needs
+to be initialized. This includes the creation of the database, user,
+required tables and the Django instance.
+
+That process can be supervised using the ``docker service ls`` command, which
+lists all available services and their respective status.