VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2020-07-23T13:16:03+02:00https://gitlab.eox.at/esa/prism/vs/-/issues/46Extract preprocessor steps to allow for better re-use in VS2020-07-23T13:16:03+02:00Fabian SchindlerExtract preprocessor steps to allow for better re-use in VSCurrently the preprocessor in VHR18 is very specific for the use of the products/collections of that project. For the use in VS, the implementation needs to be generalized.
In order to generalize the preprocessor, the necessary steps ca...Currently the preprocessor in VHR18 is very specific for the use of the products/collections of that project. For the use in VS, the implementation needs to be generalized.
In order to generalize the preprocessor, the necessary steps can be extracted:
- Data retrieval: download from the swift object storage (maybe abstract this to allow other sources in the future as well)
- Unpacking: (recursively) unpack downloaded source files
- File selection: from the unpacked files, select the ones that actually have significance in VS (data files and metadata files)
- Data file merging: stacking bands if separated into multiple files, combine tiles into a single file, etc
- Output data file generation: COGs (possibly others)
- Metadata extraction
- Upload to output swift bucket (maybe other object storages in the future)https://gitlab.eox.at/esa/prism/vs/-/issues/145fix 2D-3D harmonization + style + attribution2021-11-10T22:21:58+01:00Mussab Abdallafix 2D-3D harmonization + style + attributionDEM First ReleaseMussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/120fix bump2version config for client package.json increment2021-11-09T14:30:22+01:00Lubomir Doležalfix bump2version config for client package.json increment```
[bumpversion:glob:client/package*.json]
search = "name": "VSClient",
"version": "{current_version}"
replace = "name": "VSClient",
"version": "{new_version}"
```
while running bumpversion - following part of config will have whites...```
[bumpversion:glob:client/package*.json]
search = "name": "VSClient",
"version": "{current_version}"
replace = "name": "VSClient",
"version": "{new_version}"
```
while running bumpversion - following part of config will have whitespace "spaces" changed to "tabs", which breaks the search and creates mangled package.json and package-lock.json
for now, version incremented manually via `npm --prefix client/ version 1.3.11`https://gitlab.eox.at/esa/prism/vs/-/issues/118Fix Mapcache WMTS in QGIS - type unknown error2021-11-09T14:30:56+01:00Lubomir DoležalFix Mapcache WMTS in QGIS - type unknown errorSuggested PR & linked issue https://github.com/MapServer/mapcache/pull/248 https://github.com/MapServer/mapcache/issues/247
Suggested solution https://github.com/MapServer/mapcache/pull/248#pullrequestreview-628959909 to be added to our ...Suggested PR & linked issue https://github.com/MapServer/mapcache/pull/248 https://github.com/MapServer/mapcache/issues/247
Suggested solution https://github.com/MapServer/mapcache/pull/248#pullrequestreview-628959909 to be added to our branch as patch or directly to the PR.
Related issue: https://gitlab.eox.at/esa/prism/vs/-/issues/88Fabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/70full prism functionality test2021-11-09T13:48:05+01:00Mussab Abdallafull prism functionality testA test which would verify the functionality of the entire prism stack should be added.
Currently a product is being ingested and the expected processed result is inspected, this test can be expanded so that the registration / reporting ...A test which would verify the functionality of the entire prism stack should be added.
Currently a product is being ingested and the expected processed result is inspected, this test can be expanded so that the registration / reporting / rendering of the ingested product can be tested.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/57Generalize registrar2020-10-12T11:19:06+02:00Lubomir DoležalGeneralize registrarIdeally have a simplified configuration as a yaml in a similar way as new preprocessor has it.
Mainly targetting the XML paths to get product / level / collection from
Currently it is hardcodedIdeally have a simplified configuration as a yaml in a similar way as new preprocessor has it.
Mainly targetting the XML paths to get product / level / collection from
Currently it is hardcodedBaseline evolution v2https://gitlab.eox.at/esa/prism/vs/-/issues/29GetCapabilities of a single product could show available dim_bands2021-11-09T13:55:24+01:00Lubomir DoležalGetCapabilities of a single product could show available dim_bandsvia cql=identifier=... or via layers= (with and without __coverage)via cql=identifier=... or via layers= (with and without __coverage)https://gitlab.eox.at/esa/prism/vs/-/issues/131GetEOCoverageSet requests2021-09-14T18:37:55+02:00Mussab AbdallaGetEOCoverageSet requestsGetEOCoverageSet requests returns an exception: `Missing required parameter 'format'`, even though `format` is not a mandatory parameter according to [OGC Standerds](https://docs.opengeospatial.org/is/10-140r2/10-140r2.html#geteocoverage...GetEOCoverageSet requests returns an exception: `Missing required parameter 'format'`, even though `format` is not a mandatory parameter according to [OGC Standerds](https://docs.opengeospatial.org/is/10-140r2/10-140r2.html#geteocoverageset_operation).
When provided the response returns that format is not supported (e.g: `'Format 'image/tiff' is not supported.'`) even if the format is supported- listed in element`wcs:formatSupported` in wcs `GetCapabilities` response -https://gitlab.eox.at/esa/prism/vs/-/issues/117gunicorn: configurable worker timeouts2021-11-09T14:32:44+01:00Fabian Schindlergunicorn: configurable worker timeoutsThis setting should be configurable via ENV/Chart
https://docs.gunicorn.org/en/latest/settings.html#timeoutThis setting should be configurable via ENV/Chart
https://docs.gunicorn.org/en/latest/settings.html#timeouthttps://gitlab.eox.at/esa/prism/vs/-/issues/30Idea: if gsc does not contain footprint, registrar/preprocessor could attempt...2021-11-09T13:22:18+01:00Lubomir DoležalIdea: if gsc does not contain footprint, registrar/preprocessor could attempt to recreate it at least approximately from image boundsI think preprocessor in ngeo already has some code for itI think preprocessor in ngeo already has some code for ithttps://gitlab.eox.at/esa/prism/vs/-/issues/129Implement file level metadata in registrar2021-11-09T12:19:44+01:00Nikola JankovicImplement file level metadata in registrarThe idea is to add a function to the source that would fetch file level metadata and use it in the scheme to register in the backendThe idea is to add a function to the source that would fetch file level metadata and use it in the scheme to register in the backendNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/108Implement Harvester2021-09-02T19:28:44+02:00Fabian SchindlerImplement HarvesterAccording to https://app.diagrams.net/#G1L8WXp2Ldp1kAqYZMDZWjs0r_g8UzB3hX and https://docs.google.com/document/d/1EWL3C0FrHqoJMMK1_1XisLBOokpPUpbwSaKteQrcYo0/edit#
the harvester should be capable of
- listening on a queue for new harve...According to https://app.diagrams.net/#G1L8WXp2Ldp1kAqYZMDZWjs0r_g8UzB3hX and https://docs.google.com/document/d/1EWL3C0FrHqoJMMK1_1XisLBOokpPUpbwSaKteQrcYo0/edit#
the harvester should be capable of
- listening on a queue for new harvesting jobs
- querying the configured backend with static/dynamic parameters
- resolve items to be pushed to a configurable queue (registration, preprocessing, deletion, ...)
Endpoints include:
- resolving hierarchies of STAC-Catalogues
- OpenSearch
- STAC API
- OGC API RecordsNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/109Implement Harvesting Scheduler2021-09-02T19:28:43+02:00Fabian SchindlerImplement Harvesting SchedulerThe harvesting scheduler shall allow to configure regular intervals where harvesting jobs are to be executedThe harvesting scheduler shall allow to configure regular intervals where harvesting jobs are to be executedNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/20Implement reporting interface2020-11-11T16:20:46+01:00Stephan Meißlstephan.meissl@eox.atImplement reporting interfaceBaseline evolutionFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/55Implement SFTP service2020-11-11T11:55:19+01:00Lubomir DoležalImplement SFTP serviceTarget is that on master node (role==manager), there is a service providing SFTP (FTP via SSH) access on an exposed port in internal network to a configured set of folders - currently:
- /data/from/fepd
- /data/to/panda
These folders sh...Target is that on master node (role==manager), there is a service providing SFTP (FTP via SSH) access on an exposed port in internal network to a configured set of folders - currently:
- /data/from/fepd
- /data/to/panda
These folders should be accessible as volumes for other components on master node (ingestor & optimally also registrar for export when new product registered)
A premade docker image should be used
#20 and #7 partly depend on itBaseline evolutionMussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/159implement WPS UI for DEM collection in the vs client2021-11-10T21:54:45+01:00Mussab Abdallaimplement WPS UI for DEM collection in the vs clientDEM Second Releasehttps://gitlab.eox.at/esa/prism/vs/-/issues/99Improve and properly document current operations2021-11-22T16:21:44+01:00Lubomir DoležalImprove and properly document current operations- properly document operations part of the deployed production and staging instance of prism
- target of the `prism` documentation should be a new entry on https://handbook.eox.at/
- Some parts of current readme / operators guide could ...- properly document operations part of the deployed production and staging instance of prism
- target of the `prism` documentation should be a new entry on https://handbook.eox.at/
- Some parts of current readme / operators guide could be moved there - review
- improve on ssh accessing on ever-growing multinode setup
- add some kibana alerts on production service similar as for mundi: renderer errors, cache errors rate over threshold, ingestion failure (above 0) - emg collections
- to document: proxy usage, deploy stack, tear stack, cleanup leftover preprocessing temp files after errors, pull newest images version ...Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/79Ingestor fails to save success/fail file if already exists.2021-02-02T17:14:55+01:00Lubomir DoležalIngestor fails to save success/fail file if already exists.Ingestor:
When it tries to save the "ingestor success" file when it is already present there. (ingested before already), the commands crashes because it already exists. Here are the logs from one such event on reference platform.
```
Feb...Ingestor:
When it tries to save the "ingestor success" file when it is already present there. (ingested before already), the commands crashes because it already exists. Here are the logs from one such event on reference platform.
```
Feb 2, 2021 @ 09:17:52.950 [2021-02-02 08:17:52,947][filedaemon] ERROR: Destination path '/var/ingestor/success/CDD_PDAS_20200925_090956239.xml' already exists
Feb 2, 2021 @ 09:17:52.950 Traceback (most recent call last):
Feb 2, 2021 @ 09:17:52.950 File "/filedaemon.py", line 101, in process_IN_CLOSE_WRITE
Feb 2, 2021 @ 09:17:52.950 save_mount_report(event.pathname, True)
Feb 2, 2021 @ 09:17:52.950 File "/ingestor/util.py", line 49, in save_mount_report
Feb 2, 2021 @ 09:17:52.950 shutil.move(data, save_dir)
Feb 2, 2021 @ 09:17:52.950 File "/usr/lib/python3.6/shutil.py", line 548, in move
Feb 2, 2021 @ 09:17:52.950 raise Error("Destination path '%s' already exists" % real_dst)
Feb 2, 2021 @ 09:17:52.950 shutil.Error: Destination path '/var/ingestor/success/CDD_PDAS_20200925_090956239.xml' already exists
Feb 2, 2021 @ 09:17:52.945 [2021-02-02 08:17:52,944][filedaemon] INFO: Parsing browse file: /mnt/data/CDD_PDAS_20200925_090956239.xml
Feb 2, 2021 @ 09:17:52.945 [2021-02-02 08:17:52,945][filedaemon] DEBUG: data26/0000120743/SP07_NAO_MS4__3_20180922T122315_20180922T122332_TOU_1234_aa94.DIMA.tar
Feb 2, 2021 @ 09:17:52.938 [2021-02-02 08:17:52,938][filedaemon] DEBUG: data26/0000120698/SP07_NAO_MS4__3_20190725T095121_20190725T095133_TOU_1234_d25e.DIMA.tar
Feb 2, 2021 @ 09:17:52.936 [2021-02-02 08:17:52,933][filedaemon] INFO: Parsing browse file: /mnt/data/CDD_PDAS_20200925_090742918.xml
```
It does not crash the whole ingestion, as this step happens after putting item to preprocess queue.
Solution: overwrite the file or fail silently.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/66ingestor listening in daemon mode2020-11-27T18:37:41+01:00Mussab Abdallaingestor listening in daemon modeThe ingestor should be listening to the shared volume with the sftp image `/mnt/data `. Once brows reports are created there, it starts the process of feeding reports into the preprocessor through redis queue.
However, when posting brows...The ingestor should be listening to the shared volume with the sftp image `/mnt/data `. Once brows reports are created there, it starts the process of feeding reports into the preprocessor through redis queue.
However, when posting browse reports, there is no response from the ingestor.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/116Ingestor test should not use sleep2021-11-09T14:28:39+01:00Lubomir DoležalIngestor test should not use sleep- `sleep 60` is non-deterministic and sometimes fails the CI pipeline when there is no change in code causing this failure
- it should wait for completion of preprocessing caused by ingestor by for example listening on redis set `prepro...- `sleep 60` is non-deterministic and sometimes fails the CI pipeline when there is no change in code causing this failure
- it should wait for completion of preprocessing caused by ingestor by for example listening on redis set `preprocess-success_set` and `preprocess-failure_set` until product_path appears there (after clearing the set first) and this should have a reasonable timeout (a few minutes)