EOX GitLab Instance

Skip to content
Snippets Groups Projects
Commit 719a6fe4 authored by Stephan's avatar Stephan
Browse files

some adjustements

parent d2487c46
Branches
Tags
No related merge requests found
......@@ -5,7 +5,7 @@ Data Ingestion
This section details the data ingestion and later management in the VS.
Redis queues
Redis Queues
------------
The central synchronization component in the VS is the ``redis`` key-value
......@@ -51,22 +51,23 @@ is better to retrieve it for every command instead of relying on a variable:
docker exec -it $(docker ps -qf "name=<stack-name>_redis")
For the sake of brevity, the next commands in this chapter are using either of
the above techniques and will just print the final commands inside the redis
container.
For the sake of brevity, the subsequent commands in this chapter can be used
with either of the above techniques and will just print the final commands that
are run inside the redis container.
.. note::
For the VS, only the ``List`` and ``Set`` `Redis data types
<https://redis.io/topics/data-types>`_ are really used. ``Sets`` are an
unordered collection of string elements. In VS it is used to denote that an
element is part of a particular group, e.g: being preprocessed, or having
failed registration.
<https://redis.io/topics/data-types>`_ are really used.
``Sets`` are an unordered collection of string elements. In the VS it is
used to denote that an element is part of a particular group, e.g: being
preprocessed, or having failed registration.
``Lists`` are used as a task queue. It is possible to add items to either
end of the queue, but by convention items are pushed on the "left" and
popped from the "right" end of the list resulting in a last-in-first-out
(LIFO) queue. It is entirely possible to push elements to the "right" end
popped from the "right" end of the list resulting in a first-in-first-out
(FIFO) queue. It is entirely possible to push elements to the "right" end
as-well, and an operator may want to do so in order to add an element to be
processed as soon as possible instead of waiting before all other elements
before it are processed.
......@@ -75,9 +76,9 @@ container.
<https://redis.io/commands#list>`_ and `Sets
<https://redis.io/commands#set>`_.
For a more concrete example: the following command finds the container ID of
the redis service replica, and executes a ``redis-cli lpush`` command to add a
new path of an object to preprocess on the ``preprocess_queue``:
For a more concrete example the following command executes a
``redis-cli lpush`` command to add a new path of an object to preprocess on the
``preprocess_queue``:
.. code-block:: bash
......@@ -116,8 +117,8 @@ the ``register_queue``, where it can be inspected with the following command.
$ redis-cli lrange register_queue 0 -1
/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar
If an operator wants to trigger the re-registration of a product only the
product path needs to be pushed to this queue:
If an operator wants to trigger only the re-registration of a product without
preprocessing the product path needs to be pushed to this queue:
.. code-block:: bash
......@@ -126,13 +127,15 @@ product path needs to be pushed to this queue:
Very similar to the preprocessing, during the registration the product path is
added to the ``registering_set``, afterwards the path is placed to either the
``register-success_set`` or ``register-failure_set``. Again, these queues or
sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands.
sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands
respectively.
Direct Data Management
----------------------
Sometimes it is necessary to directly interact with the registrar/renderer. The
following section shows what tasks on the registrar can be accomplished.
Sometimes it is necessary to directly interact with the preprocessor or
registrar. The following section shows what tasks on the preprocessor and
registrar can be accomplished.
.. warning::
......@@ -140,11 +143,10 @@ following section shows what tasks on the registrar can be accomplished.
Redis sets to track what products have been registered and where the
registration failed.
Preprocessing
~~~~~~~~~~~~~
In this section all command examples are assumed to be run from a running
In this section all command examples are assumed to be run from within a running
preprocessor container. To open a shell on a preprocessor, the following
command can be used.
......@@ -205,13 +207,13 @@ Collections can be deleted, without affecting the contained products.
.. warning::
Since the other services have fixed configuration and depend on specific
collection, deleting said collections without a replacement can lead to
service disruptions.
As some other services have fixed configurations and depend on specific
collections, deleting said collections without a replacement can lead to
configuration inconsistencies and ultimately service disruptions.
In certain scenarios it may be useful to add specific products to or exclude
them from a collection. For this, the Product identifier needs to be known. To
find out the Product identifier, either query of the existing
find out the Product identifier, either a query of the existing
collection via OpenSearch or the CLI command ``id list`` can be used.
When the identifier is obtained, the following management command inserts a
......@@ -231,6 +233,3 @@ The reverse command excludes a product from a collection:
manage.py collection exclude <collection-id> <product-id>
Again, multiple products can be excluded in a single call.
......@@ -59,4 +59,4 @@ A new deployment of the stack will use the updated configuration. The above
mentioned process necessarily involves a certain service downtime between
shutting down of the stack and new deployment.
The next section :ref:`ingestion` explains how to get data into the VS.
The final section :ref:`ingestion` explains how to get data into the VS.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment