VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2021-09-10T18:36:04+02:00https://gitlab.eox.at/esa/prism/vs/-/issues/146Tagged releases do not create images2021-09-10T18:36:04+02:00Lubomir DoležalTagged releases do not create imagesCurrently because our tagged releases start with word `release` they are not caught by the TAG_REGEX in ci_image_build.sh
https://gitlab.eox.at/esa/prism/vs/-/blob/staging/ci_image_build.sh#L11-15
Solution: there is a CI env variable `C...Currently because our tagged releases start with word `release` they are not caught by the TAG_REGEX in ci_image_build.sh
https://gitlab.eox.at/esa/prism/vs/-/blob/staging/ci_image_build.sh#L11-15
Solution: there is a CI env variable `CI_COMMIT_TAG` which can be used to know if we are doing this for a tag or not.
https://docs.gitlab.com/ee/ci/variables/#list-all-environment-variablesNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/82terrain data processing for visualization in Cesium2021-09-10T14:53:16+02:00Mussab Abdallaterrain data processing for visualization in CesiumAs discussed earlier, terrain data needs to be converted into a cesium acceptable format ( either Height maps, or [quantize mesh ](https://github.com/CesiumGS/quantized-mesh)).
It's strongly recommended to use the quantize mesh format si...As discussed earlier, terrain data needs to be converted into a cesium acceptable format ( either Height maps, or [quantize mesh ](https://github.com/CesiumGS/quantized-mesh)).
It's strongly recommended to use the quantize mesh format since it's provides better performance and results.
Doing a quick scan , I ran into 3 candidate opensource tools to use :
- [Tin-terrain](https://github.com/heremaps/tin-terrain) written in C++ - more maintained fork in [cognitive-earth/tin-terrain](https://github.com/cognitive-earth/tin-terrain)
- [cesium terrain builder (docker)](https://github.com/tum-gis/cesium-terrain-builder-docker) docker image built in c++
- [quantize mesh tile](https://github.com/loicgasser/quantized-mesh-tile) written in python
I would recommend to explore in depth, to clarify performance of these tools ( performance with large size tiffs, high resolution tiffs, large extent..... etc)DEM First ReleaseLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/127Renderer errors2021-09-08T19:02:40+02:00Lubomir DoležalRenderer errorsA set of dem renderer requests for unknown reason throws problems with SWIFT AUTHENTICATION:
<ows:ExceptionText>
Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN>
</ows:ExceptionTe...A set of dem renderer requests for unknown reason throws problems with SWIFT AUTHENTICATION:
<ows:ExceptionText>
Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN>
</ows:ExceptionText>
<!--Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/eoxserver/services/views.py", line 75, in ows result = handler.handle(request) File "/usr/local/lib/python3.8/dist-packages/eoxserver/services/ows/wms/basehandlers.py", line 221, in handle result_bytes, content_type, filename = map_renderer.render_map(map_) File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/map_renderer.py", line 113, in render_map layers_plus_factories_plus_data = [ File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/map_renderer.py", line 114, in <listcomp> (layer, factory, factory.create(map_obj, layer)) File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/factories.py", line 455, in create for _ in generator: File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/factories.py", line 357, in make_browse_layer_generator ms.set_env(map_obj, creation_info.env, True) File "/usr/local/lib/python3.8/dist-packages/eoxserver/contrib/mapserver.py", line 289, in set_env raise Exception( Exception: Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN> -->
These errors happens for different areas and for a certain area on all zooms, but on all renderer nodes (problem is not node specific).
Example failing request to the DEM layer zoomed on a single product:
https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=blackbody&time=2011-02-13T16%3A59%3A03Z%2F2014-09-09T16%3A51%3A50Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=13.359375%2C47.63671875%2C13.53515625%2C47.8125
Example passing request to the product itself:
https://dem.pass.copernicus.eu/ows?service=WMS&version=1.3.0&request=GetMap&layers=urn%3Aeop%3ADLR%3ACDEM10%3ACopernicus_DSM_03_N47_00_E013_00%3AV4683-2020_1&format=image%2Fpng&TRANSPARENT=true&width=100&height=100&CRS=EPSG%3A4326&STYLES=&BBOX=47.000000%2C13.000000%2C48.000000%2C14.000000
Example of passing request from another area on DEM:
https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=blackbody&time=2011-03-19T16%3A42%3A37Z%2F2011-03-19T16%3A42%3A38Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=17.578125%2C45.703125%2C18.28125%2C46.40625
Could this be a regression caused by latest eoxserver updates. Version used: `eoxa/eoxserver:release-1.0.0-rc36` (`vs 1.4.4`)
Also tested on `vs` 1.3.11 and `vs` 1.4.1. and happens as well, could be misconfuguration actuallyLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/108Implement Harvester2021-09-02T19:28:44+02:00Fabian SchindlerImplement HarvesterAccording to https://app.diagrams.net/#G1L8WXp2Ldp1kAqYZMDZWjs0r_g8UzB3hX and https://docs.google.com/document/d/1EWL3C0FrHqoJMMK1_1XisLBOokpPUpbwSaKteQrcYo0/edit#
the harvester should be capable of
- listening on a queue for new harve...According to https://app.diagrams.net/#G1L8WXp2Ldp1kAqYZMDZWjs0r_g8UzB3hX and https://docs.google.com/document/d/1EWL3C0FrHqoJMMK1_1XisLBOokpPUpbwSaKteQrcYo0/edit#
the harvester should be capable of
- listening on a queue for new harvesting jobs
- querying the configured backend with static/dynamic parameters
- resolve items to be pushed to a configurable queue (registration, preprocessing, deletion, ...)
Endpoints include:
- resolving hierarchies of STAC-Catalogues
- OpenSearch
- STAC API
- OGC API RecordsNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/109Implement Harvesting Scheduler2021-09-02T19:28:43+02:00Fabian SchindlerImplement Harvesting SchedulerThe harvesting scheduler shall allow to configure regular intervals where harvesting jobs are to be executedThe harvesting scheduler shall allow to configure regular intervals where harvesting jobs are to be executedNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/142Setup liveness probe for services2021-09-02T17:43:30+02:00Nikola JankovicSetup liveness probe for servicesAdd and update with following
> can you please make the livenessProbe in viewserver helm chart configurable, currently it is not possible to override default values
> - periodSeconds: 10
> - successThreshold: 1
> - timeoutSecond...Add and update with following
> can you please make the livenessProbe in viewserver helm chart configurable, currently it is not possible to override default values
> - periodSeconds: 10
> - successThreshold: 1
> - timeoutSeconds: 1
>
> I propose to increase timeoutSeconds (e.g. to 5 sec) and also to configure failureThreshold (default 3) to 10 - it seems that the viewserver is often busy with requests and therefore the individual livenessProbe requests won't go through and the pod is considered unhealthy (and gets restarted)
>
> if we can't improve situation that we should talk about possibilities to either priorizite livenessProbe requests (don't know if/how this can be done in python with gunicorn or similar) or to use different means to indicate liveness (implementation effort)Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/134Parallelize CI/CD2021-08-25T13:39:59+02:00Nikola JankovicParallelize CI/CDTry by splitting building of services as separate tasks.Try by splitting building of services as separate tasks.Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/135Update to latest dind image and test run2021-08-25T13:39:59+02:00Nikola JankovicUpdate to latest dind image and test runNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/133Errors during preprocessing and registration state on 4.8.20212021-08-13T15:57:04+02:00Lubomir DoležalErrors during preprocessing and registration state on 4.8.2021Some of following errors were tracked in the operational system of PASS. Needs further investigation:
registration of what should be an existing file fails without other errors, needs inspection, happens on multiple collections (everywh...Some of following errors were tracked in the operational system of PASS. Needs further investigation:
registration of what should be an existing file fails without other errors, needs inspection, happens on multiple collections (everywhere where we gdal.Open the file to get number of bands)
```
ERROR registrar.daemon: `/vsiswift/emg-data/data26/0000433307/EW01_WV6_PAN_SO_20210726T144857_20210726T144858_DGI_77083_9ECF.0000.tar/21JUL26144857-P2AS-014234494010_02_P001.tif' not recognized as a supported file format.
```
registration error happening sometimes (not always) on demF collection - needs further investigation
```
INFO registrar.backend: Registering coverage [['dem-data', 'data24/0000448124/DEM1_SAR_DTE_90_20130502T113059_20140612T115113_ADS_000000_0460.DEM.tar/Copernicus_DSM_30_S83_00_E123_00_DEM.tif']] as int16_grayscale
ERROR registrar.daemon: get() returned more than one Grid -- it returned 2!
```
tried to fix this by adding config https://github.com/openshift/origin-aggregated-logging/blob/1a5481f25b91c38cf3a8d2b3522cffecb84bfa66/fluentd/configs.d/openshift/output-es-config.conf#L30 will see if that helps
```
failed to flush the buffer. retry_time=0 next_retry_seconds=2021-08-04 09:29:32 +0000 chunk="5c8b8701a194e5fb71ae5c51c2282ca2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch\", :port=>9200, :scheme=>\"http\"}): read timeout reached"
```Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/132Add pdf version of Operator Guide2021-08-10T12:10:45+02:00Stephan Meißlstephan.meissl@eox.atAdd pdf version of Operator GuideCopy Makefile and build from User Guide and also add a similar link to the pdf.Copy Makefile and build from User Guide and also add a similar link to the pdf.Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/98investigate NO-DATA processing policy2021-08-03T11:06:45+02:00Mussab Abdallainvestigate NO-DATA processing policycurrently the DEM data is pre-processed by assigning Nodata pixels to zero, this could lead to some issues regarding:
- visualizing sea areas, some lakes.
- generating of the terrain mesh data to be visualized in cesium.
A possibility ...currently the DEM data is pre-processed by assigning Nodata pixels to zero, this could lead to some issues regarding:
- visualizing sea areas, some lakes.
- generating of the terrain mesh data to be visualized in cesium.
A possibility of using an interpolation smoothing filter ( using `gdal_fillnodata.py` ) can be considered, however a workaround that avoids re-pre-processing data would be recommended.DEM First ReleaseMussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/96EMG SARs ingestion preprocessor configuration2021-06-17T15:58:08+02:00Lubomir DoležalEMG SARs ingestion preprocessor configuration- to unblock the TTO- to unblock the TTOLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/122Modify nginx config to not cache index.html2021-06-09T22:26:56+02:00Lubomir DoležalModify nginx config to not cache index.html- the problem is that the client html page is taken from browser cache (default browser behavior to my knowledge) and redirects to IDP are not performed on this first client html load attempt but only later on successive requests to the ...- the problem is that the client html page is taken from browser cache (default browser behavior to my knowledge) and redirects to IDP are not performed on this first client html load attempt but only later on successive requests to the opensearch endpoint to fetch the collections. This runs into: CORS error when requesting automatic redirect to the IDP side:
Cross Origin Resource Sharing error: MissingAllowOriginHeader.
- update cache-control headers to never cache index.html fileLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/111Use kubernetes secret to store DJANGO_SECRET_KEY2021-05-19T15:57:27+02:00Fabian SchindlerUse kubernetes secret to store DJANGO_SECRET_KEYNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/85Add rendering of DEM tile number to client - DEM-0222021-05-17T19:37:11+02:00Lubomir DoležalAdd rendering of DEM tile number to client - DEM-022- Simply add a Text label into center of polygon in `eoxc`
- Check if we have needed metadata ingested into db, if not adapt registrar and reingest
Example how this could look like ![image](/uploads/5bcba8e0ffa59c7aee0ea34fe000057e/image...- Simply add a Text label into center of polygon in `eoxc`
- Check if we have needed metadata ingested into db, if not adapt registrar and reingest
Example how this could look like ![image](/uploads/5bcba8e0ffa59c7aee0ea34fe000057e/image.png)DEM First ReleaseLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/115Aspect ratio of thumbnails in search results seems off2021-05-06T17:27:29+02:00Stephan Meißlstephan.meissl@eox.atAspect ratio of thumbnails in search results seems offIt seems that thumbnails in the search results are scaled to full width when not needed:
![Screenshot_from_2021-05-05_12-06-27](/uploads/a09a60b93a5cea5fbcff27fd659cf1f7/Screenshot_from_2021-05-05_12-06-27.png)
To test https://data-acc...It seems that thumbnails in the search results are scaled to full width when not needed:
![Screenshot_from_2021-05-05_12-06-27](/uploads/a09a60b93a5cea5fbcff27fd659cf1f7/Screenshot_from_2021-05-05_12-06-27.png)
To test https://data-access.185.52.193.87.nip.io/?x=21.969425&y=37.895765&z=9×tart=2020-09-05T09%3A20%3A29Z&timeend=2020-09-05T09%3A20%3A29Zhttps://gitlab.eox.at/esa/prism/vs/-/issues/80Move client repository eoxc/prism here2021-05-06T09:57:28+02:00Lubomir DoležalMove client repository eoxc/prism here- To allow development on actual data from renderer without the CORS plugin, the https://github.com/eoxc/prism `update-eoxc` checked out to branch should be moved to `vs` repository
- Build of the client docker image will include buildin...- To allow development on actual data from renderer without the CORS plugin, the https://github.com/eoxc/prism `update-eoxc` checked out to branch should be moved to `vs` repository
- Build of the client docker image will include building the js client
- remove hardcoded docs pdf and html links from https://github.com/eoxc/prism/blob/update-eoxc/src/languages/en.json#L23DEM First ReleaseLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/112Versions in chart and values.yaml2021-04-30T11:15:00+02:00Stephan Meißlstephan.meissl@eox.atVersions in chart and values.yamlIn general it should not be necessary to specify any version like image tags in the values particularly for production deploys. These tasks are identified so far:
- [x] Deployments like for the [cache](https://gitlab.eox.at/esa/prism/vs/...In general it should not be necessary to specify any version like image tags in the values particularly for production deploys. These tasks are identified so far:
- [x] Deployments like for the [cache](https://gitlab.eox.at/esa/prism/vs/-/blob/staging/chart/templates/cache-deployment.yaml#L32) use already `{{ .Values.cache.image.tag | default ( print "release-" .Chart.AppVersion ) }}`. Confirm that the fallback to AppVersion is actually working for all deployment.
- [x] [The tag version in `values.yaml`](https://gitlab.eox.at/esa/prism/vs/-/blob/staging/chart/values.yaml#L359) should not be needed anymore and probably be removed.
- [x] `helm template` without providing any values should run successfully.
- [ ] Remove versions in flux helm releases
- [ ] EOEPCA develop e.g. https://github.com/EOEPCA/eoepca/blob/develop/system/clusters/develop/resource-management/hr-data-access.yaml#L455
- [ ] EOEPCA demo e.g. https://github.com/EOEPCA/eoepca/blob/demo/system/clusters/develop/resource-management/hr-data-access.yaml#L394
- [ ] all three agri deploymentsNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/105Django admin is not accessible when more renderers spawned2021-04-27T11:46:03+02:00Lubomir DoležalDjango admin is not accessible when more renderers spawned- as secret key is different for each instance of renderer- as secret key is different for each instance of rendererDEM First ReleaseNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/103Visualizing in the metadata view with cache loads unnecessary data at differe...2021-04-07T15:21:39+02:00Nikola JankovicVisualizing in the metadata view with cache loads unnecessary data at different zoom levelsWhen visualizing the data in the information view some additional tiles appear that don't seem to belong to the queried product. Best observed with products here:
* https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/?x=15.061055&y=48.704231...When visualizing the data in the information view some additional tiles appear that don't seem to belong to the queried product. Best observed with products here:
* https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/?x=15.061055&y=48.704231&z=9×tart=2020-03-31T10%3A07%3A06Z&timeend=2020-03-31T10%3A07%3A17Z.
The request loads (when zooming out in the information view) the overlapping tile between the two products on the same day but different time:
* https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/cache/ows/wmts?layer=S2L2A__FALSE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=8&TileCol=277&TileRow=58&time=2020-03-31T10%3A07%3A06Z%2F2020-03-31T10%3A07%3A06Z
Not really sure where the fault lies, maybe the cache config?
Also observed with @stephan.meissl that for some requests that the client seems to send requests to the server that have temporal values for the full month instead of just the day, resulting in the overlapped tiles. I'm still looking where this occurs, cannot reproduce immediately.Nikola JankovicNikola Jankovic