ingestion.rst

.. _ingestion:

Data Ingestion
==============

This section details the data ingestion and later management in the VS.

Redis Queues
------------

The central synchronization component in the VS is the ``redis`` key-value
store. It provides various queues, which the services are listening to. For
operators it provides a high-level interface through which data products can be
registered and managed.

Via the Redis, the ingestion can be triggered and observed. In order to
eventually start the preprocessing of a product, its path on the configured
object storage has to be pushed onto the ``preprocess_queue``, as will be
explained in detail in this chapter.

As the Redis store is not publicly accessible from outside of the stack. So to
interact with it, the operator has to run a command from one of the services.
Conveniently, the service running Redis also has the ``redis-cli`` tool
installed that lets users interact with the store.

When doing one off commands, it is maybe more convenient to execute it on a
running service. For this, the ``docker ps`` command can be used to select the
identifier of the running docker container of the redis service.

.. code-block:: bash

    container_id=$(docker ps -qf "name=<stack-name>_redis")

With this identifier, a command can be issued:

.. code-block:: bash

    docker exec -it $container_id redis-cli ...

When performing more than one command, it can be simpler to open a shell on the
service instead:

.. code-block:: bash

    docker exec -it $container_id bash

As the container ID may change (for example when the replica is restarted) it
is better to retrieve it for every command instead of relying on a variable:

.. code-block:: bash

    docker exec -it $(docker ps -qf "name=<stack-name>_redis")

For the sake of brevity, the subsequent commands in this chapter can be used
with either of the above techniques and will just print the final commands that
are run inside the redis container.

.. note::

    For the VS, only the ``List``  and ``Set`` `Redis data types
    <https://redis.io/topics/data-types>`_ are really used.

    ``Sets`` are an unordered collection of string elements. In the VS it is
    used to denote that an element is part of a particular group, e.g: being
    preprocessed, or having failed registration.

    ``Lists`` are used as a task queue. It is possible to add items to either
    end of the queue, but by convention items are pushed on the "left" and
    popped from the "right" end of the list resulting in a first-in-first-out
    (FIFO) queue. It is entirely possible to push elements to the "right" end
    as-well, and an operator may want to do so in order to add an element to be
    processed as soon as possible instead of waiting before all other elements
    before it are processed.

    The full list of available commands can be found for both `Lists
    <https://redis.io/commands#list>`_ and `Sets
    <https://redis.io/commands#set>`_.

For a more concrete example the following command executes a
``redis-cli lpush`` command to add a new path of an object to preprocess on the
``preprocess_queue``:

.. code-block:: bash

    redis-cli lpush preprocess_queue "/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar"

Usually, with a preprocessor service running and no other items in the
``preprocess_queue`` this value will be immediately popped from the list and
processed. For the sake of demonstration this command would print the contents
of the ``preprocess_queue``:

.. code-block:: bash

    $ redis-cli lrange preprocess_queue 0 -1
    /data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar

Now that the product is being preprocessed, it should be visible in the
``preprocessing_set``. As the name indicates, this is using the ``Set``
datatype, thus requiring the ``SMEMBERS`` subcommand to list:

.. code-block:: bash

    $ redis-cli smembers preprocessing_set 0 -1
    /data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar

Once the preprocessing of the product is finished, the preprocessor will remove
the currently worked on path from the ``preprocessing_set`` and add it either
to the ``preprocess-success_set`` or the ``preprocess-failure_set`` depending
on whether the processing succeeded or not. They can be inspected using the
same ``SMEMBERS`` subcommand with one of set names as a parameter.

Additionally, upon success, the preprocessor places the same product path on
the ``register_queue``, where it can be inspected with the following command.

.. code-block:: bash

    $ redis-cli lrange register_queue 0 -1
    /data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar

If an operator wants to trigger only the re-registration of a product without
preprocessing the product path needs to be pushed to this queue:

.. code-block:: bash

    redis-cli lpush register_queue "/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar"

Very similar to the preprocessing, during the registration the product path is
added to the ``registering_set``, afterwards the path is placed to either the
``register-success_set`` or ``register-failure_set``. Again, these queues or
sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands
respectively.

Direct Data Management
----------------------

Sometimes it is necessary to directly interact with the preprocessor or
registrar. The following section shows what tasks on the preprocessor and
registrar can be accomplished.

.. warning::

    This approach is not recommended for everyday use, as it circumvents the
    Redis sets to track what products have been registered and where the
    registration failed.

Preprocessing
~~~~~~~~~~~~~

In this section all command examples are assumed to be run from within a
running preprocessor container. To open a shell on a preprocessor, the
following command can be used.

.. code-block:: bash

    docker exec -it $(docker ps -qf "name=<stack-name>_preprocessor") bash

The preprocessor can be used in two modes. The first (and default mode when
used as a service) is to be run as a daemon: it listens to the Redis queue for
new items, which will be preprocessed one by one. The second mode is to run the
preprocessor in a "one-off" mode: instead of pulling an item from the queue,
it is passed as a command line argument, which is then processed normally.

.. code-block:: bash

    python3 /preprocessor.py \
        --mode standard \
        --replace \
        --tar-object-path /data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar

In this mode, the item will not be placed in the resulting set
(``preprocessing_set``, ``preprocess-success_set``, and
``preprocess-failure_set``).


Registration Handling
~~~~~~~~~~~~~~~~~~~~~

For all intents and purposes in this section it is assumed, that the operator
is logged into a shell on the ``registrar`` service. This can be achieved via
the following command (assuming at least one registrar replica is running):

.. code-block:: bash

    docker exec -it $(docker ps -qf "name=<stack-name>_registrar") bash

The contents of the shared registrar/renderer database can be managed using
the registrars instance ``manage.py`` script. For brevity, the following bash
alias is assumed:

.. code-block:: bash

    alias manage.py='python3 /var/www/pvs/dev/pvs_instance/manage.py'


A collection is a grouping of earth observation products, accessible as a
single entity via various service endpoints. Depending on the configuration,
multiple collections are created when the service is set up. They can be listed
using the ``collection list`` command.

New collections can be created using the ``collection create`` command. This
can refer to a ``Collection Type``, which will restrict the collection in terms
of insertable products: only products of an allowed ``Product Type`` can be
added. Detailed information about the available Collection management commands
can be found in the `CLI documentation <https://docs.eoxserver.org/en/master/users/coverages.html#command-line-interfaces>`__.

Collections can be deleted, without affecting the contained products.

.. warning::

    As some other services have fixed configurations and depend on specific
    collections, deleting said collections without a replacement can lead to
    configuration inconsistencies and ultimately service disruptions.

In certain scenarios it may be useful to add specific products to or exclude
them from a collection. For this, the Product identifier needs to be known. To
find out the Product identifier, either a query of the existing
collection via OpenSearch or the CLI command ``id list`` can be used.

When the identifier is obtained, the following management command inserts a
product into a collection:

.. code-block:: bash

    manage.py collection insert <collection-id> <product-id>

Multiple products can be inserted in one pass by providing more than one
identifier.

The reverse command excludes a product from a collection:

.. code-block:: bash

    manage.py collection exclude <collection-id> <product-id>

Again, multiple products can be excluded in a single call.


Product Handling
~~~~~~~~~~~~~~~~

Registration
  Products can be registered using the EOxServer CLI tools as well.

  .. code-block:: bash

    manage.py product register \
        --metadata-file data25 /OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar/metadata.xml \
        --print-identifier \
        --type PL00

  The identifier of the newly registered product is printed to the console and
  can be used to put it into a collection. Additionally, it is necessary to add
  a coverage to it, which can be registered like:

  .. code-block:: bash

    manage.py coverage register \
        -d data25 /OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar/some.tif \
        -m data25 /OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar/metadata.xml \
        --identifier "${product_id}_coverage" \
        --type RGBNir

Deregistration
  Products and coverages need to be derigestered when no longer in use. A
  product can be deregistered using its identifier:

  .. code-block:: bash

    manage.py product deregister "${product_id}"

  The contained coverage must also be deregistered manually:

  .. code-block:: bash

    manage.py coverage deregister "${product_id}_coverage"