@@ -139,6 +139,7 @@ Triggering preprocessing and registration via pushing to the redis queues is ver
- Default: Exposing a simple ``/`` endpoint, and listening for ``POST`` requests containing ``data`` with either a Browse Report XML or a string with path to the object storage with product to be ingested. It then parses this information and internally puts it into configured redis queue (preprocess or register).
- Alternative: Listening for newly added Browse Report or Availability Report files on a configured path on a file system via ``inotify``.
These Browse Report files need to be in an agreed XML schema to be correctly handled.
``Sftp`` service enables a secure access to a configured folder via sftp, while this folder can be mounted to other vs services. This way, ``Ingestor`` can listen for newly created files by the sftp access.
If the filedaemon alternative mode should be used, ``INOTIFY_WATCH_DIR`` environment variable needs to be set and a ``command`` used in the docker-compose.<stack>.ops.yml for ``ingestor`` service needs to be set to ``python3 filedaemon.py``:
...
...
@@ -315,4 +316,49 @@ The preprocessing step aims to ensure that cloud optimized GeoTIFF (COG) files a
- Also for cases, where preprocessing step would take too much time, direct registration allowing access to the metadata and catalog functions, while justifying slower rendering times can be preferred.
Monitoring ingestion
~~~~~~~~~~~~~~~~~~~~
Monitoring ingestion can be done on production system easily via Kibana using its query language KQL. Kibana in `Discover` mode shows time histogram of individual entries, which makes it easy to visually infer the ingestion progress in time. These queries can be saved for later use and more importantly to set up alerts and statistics on these saved queries.
In order to watch for successful registrations or preprocessing campaigns, simply search for
.. code-block::
"<stack-name>_registrar" AND "Successfully"
Example of such a query, filtering data for one day into the past from now:
`stack-name`, `kibana-url` and `elasticsearch-index-id` needs to be substituted with valid values.
For failures in preprocessing following search query can be used:
.. code-block::
"<stack-name>_preprocessor" AND "ERROR" AND NOT "Target.replace"
Preprocessor and registrar by default run in mode, where they skip already registered/preprocessed products. This KQL query does not list errors like "file is already in target storage".
For checking of the status of individual product ingestion (for example to find out why it failed), it can be searched for its `path` and then list `surrounding documents` and filter them by `docker container name`.
An example query would be:
.. code-block::
"emg-pdas_registrar" AND "data26/0000171398/PL00_DOV_MS_L3A_20190313T075450_20190313T075450_PLA_000000_D5E9.DIR.tar"
Then click on an arrow on left border of the individual log message (document) to display more details -> `View surrounding documents` link appears, which lists other logs close in time to this one (default 5 before and 5 after).
It is also advisable to filter the logs per container (showing only logs from that registrar/preprocessor container, that has selected surrounding documents).
Querying for `ingestor` logs allows to see if while using the ingestor push ingestion mode, the XML was parsed correctly.
.. code-block::
"<stack-name>_ingestor"
Next chapter :ref:`access` describes used authorization and authentication concepts and lines out how the external access to individual components and service as such is configured.
