VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2021-09-02T19:28:44+02:00https://gitlab.eox.at/esa/prism/vs/-/issues/108Implement Harvester2021-09-02T19:28:44+02:00Fabian SchindlerImplement HarvesterAccording to https://app.diagrams.net/#G1L8WXp2Ldp1kAqYZMDZWjs0r_g8UzB3hX and https://docs.google.com/document/d/1EWL3C0FrHqoJMMK1_1XisLBOokpPUpbwSaKteQrcYo0/edit#
the harvester should be capable of
- listening on a queue for new harve...According to https://app.diagrams.net/#G1L8WXp2Ldp1kAqYZMDZWjs0r_g8UzB3hX and https://docs.google.com/document/d/1EWL3C0FrHqoJMMK1_1XisLBOokpPUpbwSaKteQrcYo0/edit#
the harvester should be capable of
- listening on a queue for new harvesting jobs
- querying the configured backend with static/dynamic parameters
- resolve items to be pushed to a configurable queue (registration, preprocessing, deletion, ...)
Endpoints include:
- resolving hierarchies of STAC-Catalogues
- OpenSearch
- STAC API
- OGC API RecordsNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/107Transparency and NODATA in client2022-11-21T11:31:18+01:00Nikola JankovicTransparency and NODATA in clientThe client shows some data as transparent (NODATA) even though they have values different from NODATA. The issue is observed both in the metadata view and in the main view.
![image](/uploads/2c950d799c59a81d26c70a6ec4c219a7/image.png)
!...The client shows some data as transparent (NODATA) even though they have values different from NODATA. The issue is observed both in the metadata view and in the main view.
![image](/uploads/2c950d799c59a81d26c70a6ec4c219a7/image.png)
![image](/uploads/c7f1de4dfb7a0866e74164ee18550af5/image.png)https://gitlab.eox.at/esa/prism/vs/-/issues/106extend viewserver helmchart allowing to pass kubernetes secret name(s) for cr...2021-11-22T15:59:32+01:00Stefan Achtsnitextend viewserver helmchart allowing to pass kubernetes secret name(s) for credentials like data bucket, cache-bucket, database,...should avoid to duplicate all these secrets!should avoid to duplicate all these secrets!Stephan Meißlstephan.meissl@eox.atStephan Meißlstephan.meissl@eox.athttps://gitlab.eox.at/esa/prism/vs/-/issues/104Product level requirement update for single level products2021-04-01T12:36:34+02:00Nikola JankovicProduct level requirement update for single level productsThe `product_level` is currently mandatory even for single level collections. This causes two collections to be created all pointing to the same data eg.:
- EXP__TRUE_COLOR
- EXP_**Level-2**__TRUE_COLOR
Figure out a way to normalize thi...The `product_level` is currently mandatory even for single level collections. This causes two collections to be created all pointing to the same data eg.:
- EXP__TRUE_COLOR
- EXP_**Level-2**__TRUE_COLOR
Figure out a way to normalize this in the templates.Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/103Visualizing in the metadata view with cache loads unnecessary data at differe...2021-04-07T15:21:39+02:00Nikola JankovicVisualizing in the metadata view with cache loads unnecessary data at different zoom levelsWhen visualizing the data in the information view some additional tiles appear that don't seem to belong to the queried product. Best observed with products here:
* https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/?x=15.061055&y=48.704231...When visualizing the data in the information view some additional tiles appear that don't seem to belong to the queried product. Best observed with products here:
* https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/?x=15.061055&y=48.704231&z=9×tart=2020-03-31T10%3A07%3A06Z&timeend=2020-03-31T10%3A07%3A17Z.
The request loads (when zooming out in the information view) the overlapping tile between the two products on the same day but different time:
* https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/cache/ows/wmts?layer=S2L2A__FALSE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=8&TileCol=277&TileRow=58&time=2020-03-31T10%3A07%3A06Z%2F2020-03-31T10%3A07%3A06Z
Not really sure where the fault lies, maybe the cache config?
Also observed with @stephan.meissl that for some requests that the client seems to send requests to the server that have temporal values for the full month instead of just the day, resulting in the overlapped tiles. I'm still looking where this occurs, cannot reproduce immediately.Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/102Extend authentication with external authorization source2021-11-21T16:08:10+01:00Lubomir DoležalExtend authentication with external authorization source- as last (if WMS cascading is descoped) requirement to complete for evolutions?
- more info on the external service endpoint still to be provided by TPZ
- this issue means completely removing shibauth service and configuring traefik for...- as last (if WMS cascading is descoped) requirement to complete for evolutions?
- more info on the external service endpoint still to be provided by TPZ
- this issue means completely removing shibauth service and configuring traefik forward-auth middleware to this external service & check if we provide enough info to the external service to make the decision (might need splitting of download & rendering requests - split renderer to (downloaded + renderer)https://gitlab.eox.at/esa/prism/vs/-/issues/101Document mundi operations2021-04-02T16:14:57+02:00Lubomir DoležalDocument mundi operationshttps://seeed.fr.atos.net/seeed-project/eox-cws-prerender/-/blob/eox_tenant/README.md
https://seeed.fr.atos.net/seeed-project/eox-cws-prerender/-/blob/eox_tenant/operations.md
migrate to operations part of handbookhttps://seeed.fr.atos.net/seeed-project/eox-cws-prerender/-/blob/eox_tenant/README.md
https://seeed.fr.atos.net/seeed-project/eox-cws-prerender/-/blob/eox_tenant/operations.md
migrate to operations part of handbookLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/100Investigate why test buckets are not properly deleted2021-11-09T14:28:35+01:00Lubomir DoležalInvestigate why test buckets are not properly deleted- sometimes only the bucketname_segments is left here, sometimes even the bucket itself
- review the current form of CI pipeline, for example https://gitlab.eox.at/esa/prism/vs/-/jobs/23526#L4653 should have failed? because the preproces...- sometimes only the bucketname_segments is left here, sometimes even the bucket itself
- review the current form of CI pipeline, for example https://gitlab.eox.at/esa/prism/vs/-/jobs/23526#L4653 should have failed? because the preprocessing was not done yet during ingestion test?https://gitlab.eox.at/esa/prism/vs/-/issues/99Improve and properly document current operations2021-11-22T16:21:44+01:00Lubomir DoležalImprove and properly document current operations- properly document operations part of the deployed production and staging instance of prism
- target of the `prism` documentation should be a new entry on https://handbook.eox.at/
- Some parts of current readme / operators guide could ...- properly document operations part of the deployed production and staging instance of prism
- target of the `prism` documentation should be a new entry on https://handbook.eox.at/
- Some parts of current readme / operators guide could be moved there - review
- improve on ssh accessing on ever-growing multinode setup
- add some kibana alerts on production service similar as for mundi: renderer errors, cache errors rate over threshold, ingestion failure (above 0) - emg collections
- to document: proxy usage, deploy stack, tear stack, cleanup leftover preprocessing temp files after errors, pull newest images version ...Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/97Update VHR18 & DEM traefik rules to run SSO2021-02-15T15:58:01+01:00Lubomir DoležalUpdate VHR18 & DEM traefik rules to run SSO- VHR_2018 SSO config should be added.
- For DEM collections SSO config should be added. However we have an additional issue: in fact not all user categories are authorized for the same datasets, so theoretically we should split the coll...- VHR_2018 SSO config should be added.
- For DEM collections SSO config should be added. However we have an additional issue: in fact not all user categories are authorized for the same datasets, so theoretically we should split the collection in 2 sub-collections having different authorizations.
- Nevertheless Danielle would propose to use the same config for all Copernicus DEM (including access from the public).
- We should have most of the things in older commits as deleted content.
- To unblock TTOLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/96EMG SARs ingestion preprocessor configuration2021-06-17T15:58:08+02:00Lubomir DoležalEMG SARs ingestion preprocessor configuration- to unblock the TTO- to unblock the TTOLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/81prism & eoxc need migration to webpack 4 at least2021-11-09T13:15:01+01:00Lubomir Doležalprism & eoxc need migration to webpack 4 at least- unresolvable dependency conflicts are starting to pop up with newest node versions
- currently the `prism.js` needs maximum node 12 installed to do just npm install & npm run build, node 14 breaks already- unresolvable dependency conflicts are starting to pop up with newest node versions
- currently the `prism.js` needs maximum node 12 installed to do just npm install & npm run build, node 14 breaks alreadyhttps://gitlab.eox.at/esa/prism/vs/-/issues/79Ingestor fails to save success/fail file if already exists.2021-02-02T17:14:55+01:00Lubomir DoležalIngestor fails to save success/fail file if already exists.Ingestor:
When it tries to save the "ingestor success" file when it is already present there. (ingested before already), the commands crashes because it already exists. Here are the logs from one such event on reference platform.
```
Feb...Ingestor:
When it tries to save the "ingestor success" file when it is already present there. (ingested before already), the commands crashes because it already exists. Here are the logs from one such event on reference platform.
```
Feb 2, 2021 @ 09:17:52.950 [2021-02-02 08:17:52,947][filedaemon] ERROR: Destination path '/var/ingestor/success/CDD_PDAS_20200925_090956239.xml' already exists
Feb 2, 2021 @ 09:17:52.950 Traceback (most recent call last):
Feb 2, 2021 @ 09:17:52.950 File "/filedaemon.py", line 101, in process_IN_CLOSE_WRITE
Feb 2, 2021 @ 09:17:52.950 save_mount_report(event.pathname, True)
Feb 2, 2021 @ 09:17:52.950 File "/ingestor/util.py", line 49, in save_mount_report
Feb 2, 2021 @ 09:17:52.950 shutil.move(data, save_dir)
Feb 2, 2021 @ 09:17:52.950 File "/usr/lib/python3.6/shutil.py", line 548, in move
Feb 2, 2021 @ 09:17:52.950 raise Error("Destination path '%s' already exists" % real_dst)
Feb 2, 2021 @ 09:17:52.950 shutil.Error: Destination path '/var/ingestor/success/CDD_PDAS_20200925_090956239.xml' already exists
Feb 2, 2021 @ 09:17:52.945 [2021-02-02 08:17:52,944][filedaemon] INFO: Parsing browse file: /mnt/data/CDD_PDAS_20200925_090956239.xml
Feb 2, 2021 @ 09:17:52.945 [2021-02-02 08:17:52,945][filedaemon] DEBUG: data26/0000120743/SP07_NAO_MS4__3_20180922T122315_20180922T122332_TOU_1234_aa94.DIMA.tar
Feb 2, 2021 @ 09:17:52.938 [2021-02-02 08:17:52,938][filedaemon] DEBUG: data26/0000120698/SP07_NAO_MS4__3_20190725T095121_20190725T095133_TOU_1234_d25e.DIMA.tar
Feb 2, 2021 @ 09:17:52.936 [2021-02-02 08:17:52,933][filedaemon] INFO: Parsing browse file: /mnt/data/CDD_PDAS_20200925_090742918.xml
```
It does not crash the whole ingestion, as this step happens after putting item to preprocess queue.
Solution: overwrite the file or fail silently.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/78Emergency/DEM cache below certain zoom (EMG) or with a certain pattern (DEM) ...2021-01-21T10:18:33+01:00Lubomir DoležalEmergency/DEM cache below certain zoom (EMG) or with a certain pattern (DEM) returns 502When zooming below certain zoom on Emergency and on random tiles from DEM, mapcache returns 502 with following errors:
```
| Jan 5, 2021 @ 11:39:00.439 | realloc(): invalid next size |
| Jan 5, 2021 @ 11:39:01.456 | [Tue Jan 05 10:39:01....When zooming below certain zoom on Emergency and on random tiles from DEM, mapcache returns 502 with following errors:
```
| Jan 5, 2021 @ 11:39:00.439 | realloc(): invalid next size |
| Jan 5, 2021 @ 11:39:01.456 | [Tue Jan 05 10:39:01.454923 2021] [core:notice] [pid 132:tid 140477779360704] AH00052: child pid 1428 exit signal Aborted (6) |
```
example request with enough available memory on caches (restarted all cache containers):
TileMatrix 15 error:
https://emg.pass.copernicus.eu/cache/ows/wmts?layer=Emergency__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=15&TileCol=21662&TileRow=13694&time=2018-05-01T00%3A00%3A00Z%2F2018-05-08T06%3A46%3A32Z
accessed from client https://emg.pdas.prism.eox.at/?x=-60.987155&y=14.785008&z=16×tart=2018-05-01T00%3A00%3A00Z&timeend=2018-05-08T06%3A46%3A32Z
saved yes: Emergency__TRUE_COLOR/WGS84/2018-05-04T14:26:05Z/2018-05-04T14:26:08Z/14/10829/9537.xxx
saved not: Emergency__TRUE_COLOR/WGS84/2018-05-04T14:26:05Z/2018-05-04T14:26:08Z/15/13689/9537.xxx
TileMatrix 14 error:
https://emg.pass.copernicus.eu/cache/ows/wmts?layer=Emergency__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=14&TileCol=17996&TileRow=4092&time=2019-03-01T00%3A00%3A00Z%2F2019-04-01T00%3A00%3A00Z
accessed from client:
https://emg.pdas.prism.eox.at/?x=17.718770&y=45.017191&z=15×tart=2019-03-01T00%3A00%3A00Z&timeend=2019-04-01T00%3A00%3A00Z
saved yes: Emergency__TRUE_COLOR/WGS84/2019-03-28T09:49:41Z/2019-03-28T09:49:44Z/13/8998/6145.xxx
saved not: Emergency__TRUE_COLOR/WGS84/2019-03-28T09:49:41Z/2019-03-28T09:49:44Z/14/17994/????.xxx
Problem was not seen on VHR_IMAGE_2018 at all
VHR_IMAGE_2018 works even up to z=17
https://vhr18.pass.copernicus.eu/cache/ows/wmts?layer=VHR_IMAGE_2018_Level_3__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=17&TileCol=119009&TileRow=44932&time=2018-01-30T21%3A04%3A46Z%2F2019-03-08T10%3A02%3A54Z
DEM collection has this issue happening with a different pattern, but not randomly (for some tiles of a single product yes, for some not) even on TileMatrix 10 but if you are viewing individual layers, which compose the DEM layer (DEM_COP-DEM_EEA-10-DGED__EARTH for example), none of those runs into an error.
https://dem.pass.copernicus.eu/cache/ows/wmts?layer=DEM__EARTH&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=8&TileCol=264&TileRow=64&time=2011-03-27T00%3A00%3A00Z%2F2011-04-03T00%3A00%3A00Z -> error
https://dem.pass.copernicus.eu/cache/ows/wmts?layer=DEM_COP-DEM_EEA-10-DGED__EARTH&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=8&TileCol=264&TileRow=65&time=2011-03-27T00%3A00%3A00Z%2F2011-04-03T00%3A00%3A00Z - runs fine and same with all six individual layers
-> seen on client https://dem.pdas.prism.eox.at/?x=4.866364&y=45.224732&z=9×tart=2011-03-27T00%3A00%3A00Z&timeend=2011-04-03T00%3A00%3A00Z
Suspicion that the length of the key stored to the object storage would be an issue was denied.
Size of products should also not be a problem as some DEM products are smaller and some larger than average VHR products, but all of them work in individual DEM layers, but on DEM shared one not.
Problem is happening on all nodes.https://gitlab.eox.at/esa/prism/vs/-/issues/76client download functionality regression2021-11-09T13:04:59+01:00Lubomir Doležalclient download functionality regression- broken for emergency and any newly registered collection
- we used to rewrite `productid` to `productid__coverage` to get `coverageid` but now naming of coverage identifiers changed to be in format `productid__rasteridentifier__coverag...- broken for emergency and any newly registered collection
- we used to rewrite `productid` to `productid__coverage` to get `coverageid` but now naming of coverage identifiers changed to be in format `productid__rasteridentifier__coverage` which is shown correctly in WCS links in summary, but download panel can not use those yet - need to be implemented in `eoxc`. During implementation, take in account, that more coverages can now be available for one product (pan/ms). or multiband 1 band = 1 coverage for S2L2A - for example here https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/opensearch/collections/S2L2A/atom/?bbox=-8.302734375,38.859619140625,32.302734375,59.140380859375&start=2020-08-01T00:00:00.000Z&end=2020-08-31T23:59:59.000Z
```
<owc:operation code="GetCoverage" method="GET" type="image/tiff" href="https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/ows?service=WCS&version=2.0.1&request=GetCoverage&coverageId=S2A_33UWP_20200816_0_L2A_B02"/>
<owc:operation code="DescribeCoverage" method="GET" type="application/xml" href="https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=S2A_33UWP_20200816_0_L2A_B03"/>
```
- also breaks in EOEPCALubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/75Eoxserver gdal could use version 3.2.x2021-11-21T15:46:41+01:00Lubomir DoležalEoxserver gdal could use version 3.2.x- could increase rendering performance due to decoding performance increase of deflate compression
- also should benefit jpeg2000 in case files were only registered (s2 in eoepca?)- could increase rendering performance due to decoding performance increase of deflate compression
- also should benefit jpeg2000 in case files were only registered (s2 in eoepca?)https://gitlab.eox.at/esa/prism/vs/-/issues/74reporting interface filenames, replace colon2020-12-16T10:21:40+01:00Lubomir Doležalreporting interface filenames, replace colonname includes unsupported characters (at least for Windows, where colon `:` can not be part of filename). Replace potentially unsupported characters by `_` for example [item_20201209T122658_urn_eop_EUSI_EW02_10300500ACF8D700_013448908010...name includes unsupported characters (at least for Windows, where colon `:` can not be part of filename). Replace potentially unsupported characters by `_` for example [item_20201209T122658_urn_eop_EUSI_EW02_10300500ACF8D700_013448908010.xml](/uploads/9afba82c72fb52fdae24344166b29d42/item_20201209T122658_urn_eop_EUSI_EW02_10300500ACF8D700_013448908010.xml).https://gitlab.eox.at/esa/prism/vs/-/issues/73add persistent volume to fluentd2021-11-21T15:45:38+01:00Lubomir Doležaladd persistent volume to fluentd- If fluentd stops getting logs from some stack, they are lost when we redeploy stack, although they might have been buffered?
- would persistent volume for fluentd help here?- If fluentd stops getting logs from some stack, they are lost when we redeploy stack, although they might have been buffered?
- would persistent volume for fluentd help here?https://gitlab.eox.at/esa/prism/vs/-/issues/72Preprocessor buckets for tests need to be uniquely named2021-01-28T15:34:49+01:00Lubomir DoležalPreprocessor buckets for tests need to be uniquely named... to prevent one running pipeline deleting content of another pipeline... to prevent one running pipeline deleting content of another pipelinehttps://gitlab.eox.at/esa/prism/vs/-/issues/70full prism functionality test2021-11-09T13:48:05+01:00Mussab Abdallafull prism functionality testA test which would verify the functionality of the entire prism stack should be added.
Currently a product is being ingested and the expected processed result is inspected, this test can be expanded so that the registration / reporting ...A test which would verify the functionality of the entire prism stack should be added.
Currently a product is being ingested and the expected processed result is inspected, this test can be expanded so that the registration / reporting / rendering of the ingested product can be tested.Mussab AbdallaMussab Abdalla