ingestion.rst

Data Ingestion

This section details the data ingestion and later management in the VS.

Redis queues

The central synchronization component in the VS is the redis key-value store. It provides various queues, which are listened on by the services. For example, the preprocessor service instances listen on the preprocess_queue on the Redis. Whenever an item is added to the queue, it is eventually consumed by one of the preprocessor replicas, which performs the preprocessing. When completed, it pushes the processed item into the register_queue which itself is listened by the registrar.

So via the Redis, the ingestion can be triggered and observed. In order to eventually start the preprocessing of a product, its path on the configured object storage has to be pushed onto the preprocess_queue, as will be explained in detail in this chapter.

As the Redis store is not publicly accessible from outside of the stack. So to interact with it, the operator has to run a command from one of the services. Conveniently, the service running Redis also has the redis-cli tool installed that lets users interact with the store.

When doing one off commands, it is maybe more convenient to execute it on a running service. For this, the docker ps command can be used to select the identifier of the running docker container of the redis service.

container_id=$(docker ps -qf "name=<stack-name>_redis")

With this identifier, a command can be issued:

docker exec -it $container_id redis-cli ...

When performing more than one command, it can be simpler to open a shell on the service instead:

docker exec -it $container_id bash

As the container ID may change (for example when the replica is restarted) it is better to retrieve it for every command instead of relying on a variable:

docker exec -it $(docker ps -qf "name=<stack-name>_redis")

For the sake of brevity, the next commands in this chaptere are using either of the above techniques and will just print the final commands inside the redis container.

Note

For the VS, only the List and Set Redis data types are really used. Sets are an unordered collection of string elements. In VS it is used to denote that an element is part of a particular group, e.g: being preprocessed, or having failed registration.

Lists are used as a task queue. It is possible to add items to eithre end of the queue, but by convention items are pushed on the "left" and popped from the "right" end of the list resulting in a last-in-first-out (LIFO) queue. It is entirely possible to push elements to the "right" end as-well, and an operator may want to do so in order to add an element to be processed as soon as possible instead of waiting before all other elements before it are processed.

The full list of available commands can be found for both Lists and Sets.

For a more concrete example: the following command finds the container ID of the redis service replica, and executes a redis-cli lpush command to add a new path of an object to preprocess on the preprocess_queue:

redis-cli lpush preprocess_queue "/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar"

Usually, with a preprocessor service running and no other items in the preprocess_queue this value will be immediatly popped from the list and processed. For the sake of demonstration this command would print the contents of the preprocess_queue:

$ redis-cli lrange preprocess_queue 0 -1
/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar

Now that the product is beeing preprocessed, it should be visible in the preprocessing_set. As the name indicates, this is using the Set datatype, thus requiring the SMEMBERS subcommand to list:

$ redis-cli smembers preprocessing_set 0 -1
/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar

Once the preprocessing of the product is finished, the preprocessor will remove the currently worked on path from the preprocessing_set and add it either to the preprocess-success_set or the preprocess-failure_set depending on whether the processing succeeded or not. They can be inspected using the same SMEMBERS subcommand but either name as parameter.

Additionally, upon success, the preprocessor places the same product path on the register_queue, where it can be inspected with the following command.

$ redis-cli lrange register_queue 0 -1
/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar

If an operator wants to trigger the re-registration of a product only the product path needs to be pushed to this queue:

redis-cli lpush register_queue "/data25/OA/PL00/1.0/00/urn:eop:DOVE:MULTISPECTRAL_4m:20180811_081455_1054_3be7/0001/PL00_DOV_MS_L3A_20180811T081455_20180811T081455_TOU_1234_3be7.DIMA.tar"

Very similar to the preprocessing, during the registration the product path is added to the registering_set, afterwards the path is placed to either the register-success_set or register-failure_set. Again, these queues or sets can be inspected by the LRANGE or SMEMBERS subcommands.

Data Management

Sometimes it is necessary to directly interact with the registrar/renderer. The following section shows what tasks on the registrar can be accomplished.

For all intents and purposes in this section it is assumed, that the operator is logged into a shell on the registrar service. This can be achieved via the following command (assuming at least one registrar replica is running):

docker exec -it $(docker ps -qf "name=<stack-name>_registrar") bash

The contents of the shared registrar/renderer database can be managed using the registrars instance manage.py script. For brevity, the following bash alias is assumed:

alias manage.py='python3 ..../manage.py' # TODO

Manual Data Registration

Warning

This approach is not recommended for production use, as it circumvents the Redis sets to track what products have been registered and where the registration failed.

Collection Management

A collection is a grouping of earth observation products, accessible as a single entity via various service endpoints. Depending on the configuration, multiple collections are created when the service is set up. They can be listed using the collection list command.

New collections can be created using the collection create command. This can refer to a Collection Type, which will restrict the collection in terms of insertable products: only products of an allowed Product Type can be added. Detailed information about the available Collection management commands can be found in the CLI documentation.

Collections can be deleted, without affecting the contained products.

Warning

Since the other services have fixed configuration and depend on specific collection, deleting said collections without a replacement can lead to service disruptions.

In certain scenarios it may be useful, to add specific products to or exclude them from a collection. For this, the Product identifier needs to be known. To find out the Product identifier, either the OpenSearch on an existing collection or the CLI command id list can be used.

When the identifier is obtained, the following management command inserts a product into a collection:

manage.py collection insert <collection-id> <product-id>

Multiple products can be inserted in one pass by providing more than one identifier.

The reverse command excludes a product from a collection:

manage.py collection exclude <collection-id> <product-id>

Again, multiple products can be excluded in a single call.