EOX GitLab Instance
Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
V
VS
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Container registry
Model registry
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
ESA
PRISM
VS
Commits
719a6fe4
Commit
719a6fe4
authored
4 years ago
by
Stephan
Browse files
Options
Downloads
Patches
Plain Diff
some adjustements
parent
d2487c46
Branches
Branches containing commit
Tags
Tags containing commit
No related merge requests found
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
documentation/operator-guide/ingestion.rst
+26
-27
26 additions, 27 deletions
documentation/operator-guide/ingestion.rst
documentation/operator-guide/management.rst
+1
-1
1 addition, 1 deletion
documentation/operator-guide/management.rst
with
27 additions
and
28 deletions
documentation/operator-guide/ingestion.rst
+
26
−
27
View file @
719a6fe4
...
...
@@ -5,7 +5,7 @@ Data Ingestion
This section details the data ingestion and later management in the VS.
Redis
q
ueues
Redis
Q
ueues
------------
The central synchronization component in the VS is the ``redis`` key-value
...
...
@@ -51,22 +51,23 @@ is better to retrieve it for every command instead of relying on a variable:
docker exec -it $(docker ps -qf "name=<stack-name>_redis")
For the sake of brevity, the
nex
t commands in this chapter
are using either of
the above techniques and will just print the final commands
inside the redis
container.
For the sake of brevity, the
subsequen
t commands in this chapter
can be used
with either of
the above techniques and will just print the final commands
that
are run inside the redis
container.
.. note::
For the VS, only the ``List`` and ``Set`` `Redis data types
<https://redis.io/topics/data-types>`_ are really used. ``Sets`` are an
unordered collection of string elements. In VS it is used to denote that an
element is part of a particular group, e.g: being preprocessed, or having
failed registration.
<https://redis.io/topics/data-types>`_ are really used.
``Sets`` are an unordered collection of string elements. In the VS it is
used to denote that an element is part of a particular group, e.g: being
preprocessed, or having failed registration.
``Lists`` are used as a task queue. It is possible to add items to either
end of the queue, but by convention items are pushed on the "left" and
popped from the "right" end of the list resulting in a
la
st-in-first-out
(
L
IFO) queue. It is entirely possible to push elements to the "right" end
popped from the "right" end of the list resulting in a
fir
st-in-first-out
(
F
IFO) queue. It is entirely possible to push elements to the "right" end
as-well, and an operator may want to do so in order to add an element to be
processed as soon as possible instead of waiting before all other elements
before it are processed.
...
...
@@ -75,9 +76,9 @@ container.
<https://redis.io/commands#list>`_ and `Sets
<https://redis.io/commands#set>`_.
For a more concrete example
:
the following command
finds the container ID of
the redis service replica, and executes a ``redis-cli lpush`` command to add a
new path of an object to preprocess on the
``preprocess_queue``:
For a more concrete example the following command
executes a
``redis-cli lpush`` command to add a new path of an object to preprocess on the
``preprocess_queue``:
.. code-block:: bash
...
...
@@ -116,8 +117,8 @@ the ``register_queue``, where it can be inspected with the following command.
$ redis-cli lrange register_queue 0 -1
/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar
If an operator wants to trigger the re-registration of a product
only the
product path needs to be pushed to this queue:
If an operator wants to trigger
only
the re-registration of a product
without
preprocessing the
product path needs to be pushed to this queue:
.. code-block:: bash
...
...
@@ -126,13 +127,15 @@ product path needs to be pushed to this queue:
Very similar to the preprocessing, during the registration the product path is
added to the ``registering_set``, afterwards the path is placed to either the
``register-success_set`` or ``register-failure_set``. Again, these queues or
sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands.
sets can be inspected by the ``LRANGE`` or ``SMEMBERS`` subcommands
respectively.
Direct Data Management
----------------------
Sometimes it is necessary to directly interact with the registrar/renderer. The
following section shows what tasks on the registrar can be accomplished.
Sometimes it is necessary to directly interact with the preprocessor or
registrar. The following section shows what tasks on the preprocessor and
registrar can be accomplished.
.. warning::
...
...
@@ -140,11 +143,10 @@ following section shows what tasks on the registrar can be accomplished.
Redis sets to track what products have been registered and where the
registration failed.
Preprocessing
~~~~~~~~~~~~~
In this section all command examples are assumed to be run from a running
In this section all command examples are assumed to be run from
within
a running
preprocessor container. To open a shell on a preprocessor, the following
command can be used.
...
...
@@ -205,13 +207,13 @@ Collections can be deleted, without affecting the contained products.
.. warning::
Since th
e other services have fixed configuration and depend on specific
collection, deleting said collections without a replacement can lead to
service disruptions.
As som
e other services have fixed configuration
s
and depend on specific
collection
s
, deleting said collections without a replacement can lead to
configuration inconsistencies and ultimately
service disruptions.
In certain scenarios it may be useful to add specific products to or exclude
them from a collection. For this, the Product identifier needs to be known. To
find out the Product identifier, either query of the existing
find out the Product identifier, either
a
query of the existing
collection via OpenSearch or the CLI command ``id list`` can be used.
When the identifier is obtained, the following management command inserts a
...
...
@@ -231,6 +233,3 @@ The reverse command excludes a product from a collection:
manage.py collection exclude <collection-id> <product-id>
Again, multiple products can be excluded in a single call.
This diff is collapsed.
Click to expand it.
documentation/operator-guide/management.rst
+
1
−
1
View file @
719a6fe4
...
...
@@ -59,4 +59,4 @@ A new deployment of the stack will use the updated configuration. The above
mentioned process necessarily involves a certain service downtime between
shutting down of the stack and new deployment.
The
next
section :ref:`ingestion` explains how to get data into the VS.
The
final
section :ref:`ingestion` explains how to get data into the VS.
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment