VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2021-11-09T12:19:44+01:00https://gitlab.eox.at/esa/prism/vs/-/issues/129Implement file level metadata in registrar2021-11-09T12:19:44+01:00Nikola JankovicImplement file level metadata in registrarThe idea is to add a function to the source that would fetch file level metadata and use it in the scheme to register in the backendThe idea is to add a function to the source that would fetch file level metadata and use it in the scheme to register in the backendNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/127Renderer errors2021-09-08T19:02:40+02:00Lubomir DoležalRenderer errorsA set of dem renderer requests for unknown reason throws problems with SWIFT AUTHENTICATION:
<ows:ExceptionText>
Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN>
</ows:ExceptionTe...A set of dem renderer requests for unknown reason throws problems with SWIFT AUTHENTICATION:
<ows:ExceptionText>
Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN>
</ows:ExceptionText>
<!--Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/eoxserver/services/views.py", line 75, in ows result = handler.handle(request) File "/usr/local/lib/python3.8/dist-packages/eoxserver/services/ows/wms/basehandlers.py", line 221, in handle result_bytes, content_type, filename = map_renderer.render_map(map_) File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/map_renderer.py", line 113, in render_map layers_plus_factories_plus_data = [ File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/map_renderer.py", line 114, in <listcomp> (layer, factory, factory.create(map_obj, layer)) File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/factories.py", line 455, in create for _ in generator: File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/factories.py", line 357, in make_browse_layer_generator ms.set_env(map_obj, creation_info.env, True) File "/usr/local/lib/python3.8/dist-packages/eoxserver/contrib/mapserver.py", line 289, in set_env raise Exception( Exception: Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN> -->
These errors happens for different areas and for a certain area on all zooms, but on all renderer nodes (problem is not node specific).
Example failing request to the DEM layer zoomed on a single product:
https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=blackbody&time=2011-02-13T16%3A59%3A03Z%2F2014-09-09T16%3A51%3A50Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=13.359375%2C47.63671875%2C13.53515625%2C47.8125
Example passing request to the product itself:
https://dem.pass.copernicus.eu/ows?service=WMS&version=1.3.0&request=GetMap&layers=urn%3Aeop%3ADLR%3ACDEM10%3ACopernicus_DSM_03_N47_00_E013_00%3AV4683-2020_1&format=image%2Fpng&TRANSPARENT=true&width=100&height=100&CRS=EPSG%3A4326&STYLES=&BBOX=47.000000%2C13.000000%2C48.000000%2C14.000000
Example of passing request from another area on DEM:
https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=blackbody&time=2011-03-19T16%3A42%3A37Z%2F2011-03-19T16%3A42%3A38Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=17.578125%2C45.703125%2C18.28125%2C46.40625
Could this be a regression caused by latest eoxserver updates. Version used: `eoxa/eoxserver:release-1.0.0-rc36` (`vs 1.4.4`)
Also tested on `vs` 1.3.11 and `vs` 1.4.1. and happens as well, could be misconfuguration actuallyLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/126Pansharpening on the fly - rendering2021-11-21T16:10:27+01:00Lubomir DoležalPansharpening on the fly - rendering**Implementation of Pansharpening function for renderer**
- Based on registering of multispectral images and panchromatic COG images inside one product, renderer should be configurable to apply the pansharpening operation on the fly (as ...**Implementation of Pansharpening function for renderer**
- Based on registering of multispectral images and panchromatic COG images inside one product, renderer should be configurable to apply the pansharpening operation on the fly (as a browse type?) for resolutions higher than resolution of the multispectral image if possibleFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/125Pansharpening on the fly - registration & preprocessing2021-09-30T11:35:02+02:00Lubomir DoležalPansharpening on the fly - registration & preprocessing**Preprocessing & registration chain needs to be configurable to allow ingestion of images to be used for pansharpening**
- currently we allow just mosaicking of tiled products, but not special handling of multiple images with same bbox
...**Preprocessing & registration chain needs to be configurable to allow ingestion of images to be used for pansharpening**
- currently we allow just mosaicking of tiled products, but not special handling of multiple images with same bbox
- some products have a set of PAN and MS images inside 1 product archive (also tiled), now we just throw away the PAN band via path regex during archive extraction
- ingestion needs to be extended to support the preprocessing & registration of multiple files per product, i.e., a COG holding the multispectral as well as a COG holding the panchromatic bands
These multiple products are then also accessible via the download service.
- This is important to apply for VHR_IMAGE_2015 and Urban Atlas new collections
linked issue which will be closed by this https://gitlab.eox.at/esa/prism/vs/-/issues/24Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/124SAR Preprocessing automation2021-11-21T15:26:57+01:00Lubomir DoležalSAR Preprocessing automation**Implementation of automatic procedures for SAR products ingestion**
- Following emg product types CS0x, RS02, TX01 or along the way all from https://gitlab.eox.at/esa/prism/vs/-/issues/56 need to have new configurations of preprocessor...**Implementation of automatic procedures for SAR products ingestion**
- Following emg product types CS0x, RS02, TX01 or along the way all from https://gitlab.eox.at/esa/prism/vs/-/issues/56 need to have new configurations of preprocessor mainly doing following:
- automatically identify the type and number of bands. Based on this
identification the Preprocessor automatically decides which preprocessing steps to apply for product types where this is configured
- unfortunately, inside one product type different preprocessings will need to be applied (1 band, 2 bands, 4 bands in one product type RS02 or TX01)
- implement gdal_calc configured computations for conversion to decibel range for better viewing based probably on https://gitlab.eox.at/esa/prism/vhr_image_2018/-/blob/master/preprocessor/transform_chain.py#L501-588 https://gitlab.eox.at/esa/prism/vhr_image_2018/-/blob/master/preprocessor/preprocessor.py#L289-301
- ingest backlog products
- Release of updated PASS SW and Operator Guide documents (I would instead add a TN and optionally a operators guide)
- Execution of Test Procedures for new requirements documented in
updated PASS Operator Guide (I would create a separate TN, not put this into operator guide, except for configuration changes, if applicable)
- Execution of regression tests following Test Procedures documented in
updated PASS Operator Guide (that means, before it threw an error, now works) (I would create a separate TN, not put this into operator guide, except for configuration changes, if applicable)
- along the way, would be great to solve https://gitlab.eox.at/esa/prism/vs/-/issues/64, https://gitlab.eox.at/esa/prism/vs/-/issues/53Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/123Across track incidence angle and along track incidence angle not respected by...2021-11-22T16:30:41+01:00Lubomir DoležalAcross track incidence angle and along track incidence angle not respected by opensearch querytwo new metadata fields `across_track_incidence_angle` and `along_track_incidence_angle` added in https://github.com/EOxServer/eoxserver/commit/f1d1cd545fd6868bffd2bd28ca37f2235c3930fe are not correctly respected by opensearch query (fil...two new metadata fields `across_track_incidence_angle` and `along_track_incidence_angle` added in https://github.com/EOxServer/eoxserver/commit/f1d1cd545fd6868bffd2bd28ca37f2235c3930fe are not correctly respected by opensearch query (filtering on this parameter does not happen), but they should be correctly registered as they are listed in metadata list on client details page
- query: https://vhr18.pdas.prism.eox.at/opensearch/collections/VHR_IMAGE_2018_Level_3/atom/?count=1&acrossTrackIncidenceAngle=[-10,60]&cloudCover=[0,2]&bbox=14.992165253879458,51.39996854155048,15.386105484165885,51.787848152909426&start=2018-08-01T00:00:00.000Z&end=2018-08-05T22:58:39.377Z shows that filtering on cloudCover works [0,2], but on acrossTrackIncidenceAngle[-10,60] does not, as value is `-16.4885876115`
- client request leading to this opensearch query: https://vhr18.pdas.prism.eox.at/?x=14.689134&y=50.869664&start=2018-08-01T00%3A00%3A00Z&end=2018-08-05T22%3A58%3A39Z&z=8&leftpanel=true&area=POLYGON%28%2814.992165+51.399969%2C15.386105+51.399969%2C15.386105+51.787848%2C14.992165+51.787848%2C14.992165+51.399969%29%29#https://gitlab.eox.at/esa/prism/vs/-/issues/122Modify nginx config to not cache index.html2021-06-09T22:26:56+02:00Lubomir DoležalModify nginx config to not cache index.html- the problem is that the client html page is taken from browser cache (default browser behavior to my knowledge) and redirects to IDP are not performed on this first client html load attempt but only later on successive requests to the ...- the problem is that the client html page is taken from browser cache (default browser behavior to my knowledge) and redirects to IDP are not performed on this first client html load attempt but only later on successive requests to the opensearch endpoint to fetch the collections. This runs into: CORS error when requesting automatic redirect to the IDP side:
Cross Origin Resource Sharing error: MissingAllowOriginHeader.
- update cache-control headers to never cache index.html fileLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/121demF client misbehave2021-11-22T16:23:07+01:00Lubomir DoležaldemF client misbehave![image](/uploads/cda404f2eae6177b2cb769d90e87b96e/image.png)
Some more investigation needed on this, comparing metadata registered in DB and visualized in the client. It seems to be only on demF and not on other newly registered collec...![image](/uploads/cda404f2eae6177b2cb769d90e87b96e/image.png)
Some more investigation needed on this, comparing metadata registered in DB and visualized in the client. It seems to be only on demF and not on other newly registered collections (sace-emg / emg).
products cover a really long duration both in cache created tiles and timeslider entries
Both symptoms could have same origin actually (very large area on timeslider could create mismatch on mapcache fetched products to renderer) but I am just guessing.
- after this is fixed, we need to purge demF cache and dem cacheLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/120fix bump2version config for client package.json increment2021-11-09T14:30:22+01:00Lubomir Doležalfix bump2version config for client package.json increment```
[bumpversion:glob:client/package*.json]
search = "name": "VSClient",
"version": "{current_version}"
replace = "name": "VSClient",
"version": "{new_version}"
```
while running bumpversion - following part of config will have whites...```
[bumpversion:glob:client/package*.json]
search = "name": "VSClient",
"version": "{current_version}"
replace = "name": "VSClient",
"version": "{new_version}"
```
while running bumpversion - following part of config will have whitespace "spaces" changed to "tabs", which breaks the search and creates mangled package.json and package-lock.json
for now, version incremented manually via `npm --prefix client/ version 1.3.11`https://gitlab.eox.at/esa/prism/vs/-/issues/119Add cache-control headers to renderer responses2021-11-09T14:37:15+01:00Stephan Meißlstephan.meissl@eox.atAdd cache-control headers to renderer responsesProper cache-control and expires headers should at least for thumbnails reduce the load quite a bit.Proper cache-control and expires headers should at least for thumbnails reduce the load quite a bit.https://gitlab.eox.at/esa/prism/vs/-/issues/118Fix Mapcache WMTS in QGIS - type unknown error2021-11-09T14:30:56+01:00Lubomir DoležalFix Mapcache WMTS in QGIS - type unknown errorSuggested PR & linked issue https://github.com/MapServer/mapcache/pull/248 https://github.com/MapServer/mapcache/issues/247
Suggested solution https://github.com/MapServer/mapcache/pull/248#pullrequestreview-628959909 to be added to our ...Suggested PR & linked issue https://github.com/MapServer/mapcache/pull/248 https://github.com/MapServer/mapcache/issues/247
Suggested solution https://github.com/MapServer/mapcache/pull/248#pullrequestreview-628959909 to be added to our branch as patch or directly to the PR.
Related issue: https://gitlab.eox.at/esa/prism/vs/-/issues/88Fabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/117gunicorn: configurable worker timeouts2021-11-09T14:32:44+01:00Fabian Schindlergunicorn: configurable worker timeoutsThis setting should be configurable via ENV/Chart
https://docs.gunicorn.org/en/latest/settings.html#timeoutThis setting should be configurable via ENV/Chart
https://docs.gunicorn.org/en/latest/settings.html#timeouthttps://gitlab.eox.at/esa/prism/vs/-/issues/116Ingestor test should not use sleep2021-11-09T14:28:39+01:00Lubomir DoležalIngestor test should not use sleep- `sleep 60` is non-deterministic and sometimes fails the CI pipeline when there is no change in code causing this failure
- it should wait for completion of preprocessing caused by ingestor by for example listening on redis set `prepro...- `sleep 60` is non-deterministic and sometimes fails the CI pipeline when there is no change in code causing this failure
- it should wait for completion of preprocessing caused by ingestor by for example listening on redis set `preprocess-success_set` and `preprocess-failure_set` until product_path appears there (after clearing the set first) and this should have a reasonable timeout (a few minutes)https://gitlab.eox.at/esa/prism/vs/-/issues/115Aspect ratio of thumbnails in search results seems off2021-05-06T17:27:29+02:00Stephan Meißlstephan.meissl@eox.atAspect ratio of thumbnails in search results seems offIt seems that thumbnails in the search results are scaled to full width when not needed:
![Screenshot_from_2021-05-05_12-06-27](/uploads/a09a60b93a5cea5fbcff27fd659cf1f7/Screenshot_from_2021-05-05_12-06-27.png)
To test https://data-acc...It seems that thumbnails in the search results are scaled to full width when not needed:
![Screenshot_from_2021-05-05_12-06-27](/uploads/a09a60b93a5cea5fbcff27fd659cf1f7/Screenshot_from_2021-05-05_12-06-27.png)
To test https://data-access.185.52.193.87.nip.io/?x=21.969425&y=37.895765&z=9×tart=2020-09-05T09%3A20%3A29Z&timeend=2020-09-05T09%3A20%3A29Zhttps://gitlab.eox.at/esa/prism/vs/-/issues/114Database replication in VS2021-11-22T16:24:31+01:00Lubomir DoležalDatabase replication in VSResearch options to provide master-master replication of database as an alternative to buccardo.Research options to provide master-master replication of database as an alternative to buccardo.Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/113Adjust URLs for User and Operator Guide versions2021-09-17T20:52:50+02:00Stephan Meißlstephan.meissl@eox.atAdjust URLs for User and Operator Guide versionsAs we're only building the latest point release for each minor version I propose to remove if from the URL and instead of https://esa.pages.eox.at/prism/vs/user/release-1.3.8/index.html use https://esa.pages.eox.at/prism/vs/user/release-...As we're only building the latest point release for each minor version I propose to remove if from the URL and instead of https://esa.pages.eox.at/prism/vs/user/release-1.3.8/index.html use https://esa.pages.eox.at/prism/vs/user/release-1.3/index.html, etc.Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/112Versions in chart and values.yaml2021-04-30T11:15:00+02:00Stephan Meißlstephan.meissl@eox.atVersions in chart and values.yamlIn general it should not be necessary to specify any version like image tags in the values particularly for production deploys. These tasks are identified so far:
- [x] Deployments like for the [cache](https://gitlab.eox.at/esa/prism/vs/...In general it should not be necessary to specify any version like image tags in the values particularly for production deploys. These tasks are identified so far:
- [x] Deployments like for the [cache](https://gitlab.eox.at/esa/prism/vs/-/blob/staging/chart/templates/cache-deployment.yaml#L32) use already `{{ .Values.cache.image.tag | default ( print "release-" .Chart.AppVersion ) }}`. Confirm that the fallback to AppVersion is actually working for all deployment.
- [x] [The tag version in `values.yaml`](https://gitlab.eox.at/esa/prism/vs/-/blob/staging/chart/values.yaml#L359) should not be needed anymore and probably be removed.
- [x] `helm template` without providing any values should run successfully.
- [ ] Remove versions in flux helm releases
- [ ] EOEPCA develop e.g. https://github.com/EOEPCA/eoepca/blob/develop/system/clusters/develop/resource-management/hr-data-access.yaml#L455
- [ ] EOEPCA demo e.g. https://github.com/EOEPCA/eoepca/blob/demo/system/clusters/develop/resource-management/hr-data-access.yaml#L394
- [ ] all three agri deploymentsNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/111Use kubernetes secret to store DJANGO_SECRET_KEY2021-05-19T15:57:27+02:00Fabian SchindlerUse kubernetes secret to store DJANGO_SECRET_KEYNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/110Database startup is really slow2021-11-22T16:06:34+01:00Fabian SchindlerDatabase startup is really slowIn the bitnami chart postgresql there is quite some delay in the `Generating local authentication configuration` step
```
postgresql 09:16:22.09
postgresql 09:16:22.10 Welcome to the Bitnami postgresql container
postgresql 09:16:22.10 S...In the bitnami chart postgresql there is quite some delay in the `Generating local authentication configuration` step
```
postgresql 09:16:22.09
postgresql 09:16:22.10 Welcome to the Bitnami postgresql container
postgresql 09:16:22.10 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 09:16:22.13 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 09:16:22.13
postgresql 09:16:22.33 INFO ==> ** Starting PostgreSQL setup **
postgresql 09:16:22.38 INFO ==> Validating settings in POSTGRESQL_* env vars..
postgresql 09:16:22.41 INFO ==> Loading custom pre-init scripts...
postgresql 09:16:22.43 INFO ==> Initializing PostgreSQL database...
postgresql 09:16:22.57 INFO ==> pg_hba.conf file not detected. Generating it...
postgresql 09:16:22.58 INFO ==> Generating local authentication configuration
postgresql 09:17:46.90 INFO ==> Starting PostgreSQL in background...
postgresql 09:18:06.32 INFO ==> Changing password of postgres
postgresql 09:18:06.46 INFO ==> Creating user dbuser
postgresql 09:18:06.51 INFO ==> Granting access to "dbuser" to the database "dbname"
postgresql 09:18:06.63 INFO ==> Configuring replication parameters
postgresql 09:18:06.75 INFO ==> Configuring fsync
postgresql 09:18:06.97 INFO ==> Loading custom scripts...
postgresql 09:18:06.98 INFO ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
postgresql 09:18:06.99 INFO ==> Starting PostgreSQL in background...
```https://gitlab.eox.at/esa/prism/vs/-/issues/109Implement Harvesting Scheduler2021-09-02T19:28:43+02:00Fabian SchindlerImplement Harvesting SchedulerThe harvesting scheduler shall allow to configure regular intervals where harvesting jobs are to be executedThe harvesting scheduler shall allow to configure regular intervals where harvesting jobs are to be executedNikola JankovicNikola Jankovic