VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2021-11-09T15:03:19+01:00https://gitlab.eox.at/esa/prism/vs/-/issues/140STAC: Preprocessor2021-11-09T15:03:19+01:00Fabian SchindlerSTAC: PreprocessorFind an approach of how STAC can be adopted as an input/output format and how the configuration must be adopted in order to support the change.
- Preprocessor adds metadata to the actual content of STAC (image url + metadata), reference...Find an approach of how STAC can be adopted as an input/output format and how the configuration must be adopted in order to support the change.
- Preprocessor adds metadata to the actual content of STAC (image url + metadata), reference to the GSC etc.ViewServer 2.0Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/139STAC: Ingestor2021-11-21T16:22:10+01:00Fabian SchindlerSTAC: IngestorFind an approach to adopt STAC as an output format for the Ingestor.
How are the fields from the Browse Report XML mapped to the STAC item properties.
Example Browse Reports:
- Footprint browse: https://github.com/EOX-A/ngeo-b_autotes...Find an approach to adopt STAC as an output format for the Ingestor.
How are the fields from the Browse Report XML mapped to the STAC item properties.
Example Browse Reports:
- Footprint browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/reference_test_data/browseReport_ASA_IM__0P_20100722_213840.xml
- Regular Grid browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/test_data/ASA_WSM_1PNDPA20050331_075939_000000552036_00035_16121_0775.xml
- Rectified browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/test_data/S2.xml
- Model in GeoTIFF browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/test_data/rotated_axes.xmlViewServer 2.0Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/138Investigate usage of Mapchete as a preprocessing tool2021-11-22T16:06:15+01:00Fabian SchindlerInvestigate usage of Mapchete as a preprocessing toolViewServer 2.0https://gitlab.eox.at/esa/prism/vs/-/issues/137Splitting VS into component repositories2021-11-22T16:14:23+01:00Fabian SchindlerSplitting VS into component repositoriesShould help with unit testing, documentation
Tasks:
- ~~Create gitlab group~~ -> `vs` group
- ~~Create repository for each specific component~~
- ~~Move source code~~
- ~~Scan for component specific issues and move them to the repo~...Should help with unit testing, documentation
Tasks:
- ~~Create gitlab group~~ -> `vs` group
- ~~Create repository for each specific component~~
- ~~Move source code~~
- ~~Scan for component specific issues and move them to the repo~~
- move deployment configurations to a separate repository -> https://gitlab.eox.at/esa/prism/configsViewServer 2.0https://gitlab.eox.at/esa/prism/vs/-/issues/136Adopt vsq2021-11-22T16:44:31+01:00Fabian SchindlerAdopt vsq`vsq` (https://gitlab.eox.at/esa/prism/vsq) should be adopted as the main queue interface for all components as it abstracts common tasks like enqueuing/fetching, waiting for results, setting up of a daemon, etc.
- Ensure that a compone...`vsq` (https://gitlab.eox.at/esa/prism/vsq) should be adopted as the main queue interface for all components as it abstracts common tasks like enqueuing/fetching, waiting for results, setting up of a daemon, etc.
- Ensure that a component can listen on and push to multiple configured queues rather than just one.
- copy over ngeo new daemon part of using different intermediate key for avoiding losses of reports (key is at least in one queue at a time), beware that there can be multiple registrars/preprocessors picking the items - to avoid multiple instances working on same item, numeric ID of container needs to be present in the intermediate queueViewServer 2.0https://gitlab.eox.at/esa/prism/vs/-/issues/135Update to latest dind image and test run2021-08-25T13:39:59+02:00Nikola JankovicUpdate to latest dind image and test runNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/134Parallelize CI/CD2021-08-25T13:39:59+02:00Nikola JankovicParallelize CI/CDTry by splitting building of services as separate tasks.Try by splitting building of services as separate tasks.Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/133Errors during preprocessing and registration state on 4.8.20212021-08-13T15:57:04+02:00Lubomir DoležalErrors during preprocessing and registration state on 4.8.2021Some of following errors were tracked in the operational system of PASS. Needs further investigation:
registration of what should be an existing file fails without other errors, needs inspection, happens on multiple collections (everywh...Some of following errors were tracked in the operational system of PASS. Needs further investigation:
registration of what should be an existing file fails without other errors, needs inspection, happens on multiple collections (everywhere where we gdal.Open the file to get number of bands)
```
ERROR registrar.daemon: `/vsiswift/emg-data/data26/0000433307/EW01_WV6_PAN_SO_20210726T144857_20210726T144858_DGI_77083_9ECF.0000.tar/21JUL26144857-P2AS-014234494010_02_P001.tif' not recognized as a supported file format.
```
registration error happening sometimes (not always) on demF collection - needs further investigation
```
INFO registrar.backend: Registering coverage [['dem-data', 'data24/0000448124/DEM1_SAR_DTE_90_20130502T113059_20140612T115113_ADS_000000_0460.DEM.tar/Copernicus_DSM_30_S83_00_E123_00_DEM.tif']] as int16_grayscale
ERROR registrar.daemon: get() returned more than one Grid -- it returned 2!
```
tried to fix this by adding config https://github.com/openshift/origin-aggregated-logging/blob/1a5481f25b91c38cf3a8d2b3522cffecb84bfa66/fluentd/configs.d/openshift/output-es-config.conf#L30 will see if that helps
```
failed to flush the buffer. retry_time=0 next_retry_seconds=2021-08-04 09:29:32 +0000 chunk="5c8b8701a194e5fb71ae5c51c2282ca2" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch\", :port=>9200, :scheme=>\"http\"}): read timeout reached"
```Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/132Add pdf version of Operator Guide2021-08-10T12:10:45+02:00Stephan Meißlstephan.meissl@eox.atAdd pdf version of Operator GuideCopy Makefile and build from User Guide and also add a similar link to the pdf.Copy Makefile and build from User Guide and also add a similar link to the pdf.Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/131GetEOCoverageSet requests2021-09-14T18:37:55+02:00Mussab AbdallaGetEOCoverageSet requestsGetEOCoverageSet requests returns an exception: `Missing required parameter 'format'`, even though `format` is not a mandatory parameter according to [OGC Standerds](https://docs.opengeospatial.org/is/10-140r2/10-140r2.html#geteocoverage...GetEOCoverageSet requests returns an exception: `Missing required parameter 'format'`, even though `format` is not a mandatory parameter according to [OGC Standerds](https://docs.opengeospatial.org/is/10-140r2/10-140r2.html#geteocoverageset_operation).
When provided the response returns that format is not supported (e.g: `'Format 'image/tiff' is not supported.'`) even if the format is supported- listed in element`wcs:formatSupported` in wcs `GetCapabilities` response -https://gitlab.eox.at/esa/prism/vs/-/issues/130contour rendering2021-11-22T16:04:32+01:00Mussab Abdallacontour renderingthe contours visualization does not render properly as seen in the image below:
![contour_error](/uploads/743e3ec883effefd55fd241ac41cc73d/contour_error.png)
I did a brief investigation, and it seems that the EOxServer does render the w...the contours visualization does not render properly as seen in the image below:
![contour_error](/uploads/743e3ec883effefd55fd241ac41cc73d/contour_error.png)
I did a brief investigation, and it seems that the EOxServer does render the wms requests fine. Below is a wms request for the same product above (`http://127.0.0.1:81/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=urn:eop:DLR:CDEM30:Copernicus_DSM_10_N41_00_E047_00:V9577__contours&STYLES=blackbody&WIDTH=800&HEIGHT=800&SRS=EPSG%3A4326&BBOX=47.0%2C41.0%2C48.0%2C42.0`):
![wms_contour](/uploads/9e994584e8ee4e8ca25227064ea9c566/wms_contour.png)
Could it be an issue with cache ??https://gitlab.eox.at/esa/prism/vs/-/issues/129Implement file level metadata in registrar2021-11-09T12:19:44+01:00Nikola JankovicImplement file level metadata in registrarThe idea is to add a function to the source that would fetch file level metadata and use it in the scheme to register in the backendThe idea is to add a function to the source that would fetch file level metadata and use it in the scheme to register in the backendNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/128Using common data exchange format between components2021-11-22T17:45:36+01:00Fabian SchindlerUsing common data exchange format between components# Introduction
This is a collection of ideas and concepts with their respective advantages and drawbacks in their use.
The basic idea is to specify a common data exchange format encoding for most of the communications between the compo...# Introduction
This is a collection of ideas and concepts with their respective advantages and drawbacks in their use.
The basic idea is to specify a common data exchange format encoding for most of the communications between the components. The intention is to better decouple the systems and allow for better composability of the available components and potentially future ones.
It is hereby proposed to use STAC Items as a data exchange format between the components. The STAC Items are transient, in the sense that they are only put into the queues and not stored on volumes/buckets. [`VSQ`](https://gitlab.eox.at/esa/prism/vsq) allows to embed the STAC items into the JSON message structure. used.
General advantages are:
- it is possible to encode the footprint of the product directly in the JSON (but it can be set to `null` if not immediately available)
- there are several Python libraries available to digest, create or transform items (e.g: [PySTAC](https://github.com/stac-utils/pystac) and [stac.py](https://github.com/brazil-data-cube/stac.py)) but they are optional, as it is sometimes easier to simply work with the raw Python objects.
- it combines the data/metadata assets with readily available metadata values.
- referenced assets are not required to be on the same storage, allowing more flexibility
- the transient nature eliminates the requirement to create sidecar files to store metadata from one component to the next (such as GSC files generated in the preprocessor for the registrar)
Disadvantages:
- some concepts are harder to represent with STAC, such as data directories (object storage prefixes)
- it is not automatically clear how to deal with missing metadata. e.g: the `geometry` could be `null`, but how would the components handle that?
- verbosity. As the whole STAC Item is put into the queues, it may not be handy anymore to directly inspect the queues without additional tools.
## Components involved with registration/ingestion
This listing details what each component inputs/outputs and an assessment how the new format could be of use.
### *Ingestor*
- Input: Browse Report XML files
- Outputs: custom JSON format (basically translation of XML -> JSON) which currently only the preprocessor is able to handle properly
- Assessment: The custom JSON format could easily be replaced with the STAC Item format, which would standardize it, and allow for an easier integration with other components.
### *Preprocessor*
- Input: Object storage prefix or custom JSON format
- Output: Object storage prefix
- Assessment: Arguably, this component would benefit the most of a switch to STAC Items. Using the `assets` it is easily distinguishable which assets are of interest. Also, metadata of the input STAC Item could simply be passed through, without the preprocessor being required to understand it. In essence, only the asset links would have to be replaced or enriched with the processed items.
### *Registrar*
- Input: Object storage prefix
- Output: none
- Assessment: The current approach is not very stable. Several "schemes" are tried and checked whether they can be applied to be registered. Unifying this to STAC Items would greatly reduce the number of code paths. Metadata from the STAC Item could easily be handed through and mapped to the internal metadata model. It could be interesting to allow to forward the registered item to the next queue, so that the registrar is not necessarily the "dead-end" of the whole ingestion queue. (e.g: to start seeding the registered product)
### *Seeder*
- Input: ???
- Output: none
- Assessment: This component is currently not implemented in the new VS. In theory, it could retrieve seeding requests in the form of STAC Items to get the region and time of interest to seed.
### *Harvester*
- Input: custom JSON or raw values
- Output: tbd
- Assessment: currently there is no data format defined, STAC Items would be a "natural" fit as STAC API is actually one of the intended backends. Some backends may be more tricky though: e.g: object storage listings are not easily translatable into STAC Items without actually reading metadata files at that location. Some OADS outputs (`.index` files, basically just CSV) could actually map quite nicely into STAC Items.
## Usage example
### Harvester -> Preprocessor -> Registrar -> Seeder
In this example scenario, the Harvester queries an external catalogue and either passes through the STAC Items or transforms them to that format. The items are written to the queue and the harvester is oblivious of which component is the next in the chain.
The preprocessor has an immediate list of files (`assets`) to work with. There is usually no need to retrieve additional metadata, but if necessary a referenced metadata file can be opened to read that. It processes selected files from the assets, and creates a copy of the STAC Item input file and adds the preprocessed files as new assets. All other metadata is kept for other components to digest. This new STAC Item is send to the next queue.
The registrar receives the STAC Item and based on its contents and the configuration starts the registration into its backends. If successful, the STAC Item is passed on the the next component without modification.
The seeder uses the stored spatiotemporal information in the STAC Item to start the seeding process.ViewServer 2.0https://gitlab.eox.at/esa/prism/vs/-/issues/127Renderer errors2021-09-08T19:02:40+02:00Lubomir DoležalRenderer errorsA set of dem renderer requests for unknown reason throws problems with SWIFT AUTHENTICATION:
<ows:ExceptionText>
Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN>
</ows:ExceptionTe...A set of dem renderer requests for unknown reason throws problems with SWIFT AUTHENTICATION:
<ows:ExceptionText>
Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN>
</ows:ExceptionText>
<!--Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/eoxserver/services/views.py", line 75, in ows result = handler.handle(request) File "/usr/local/lib/python3.8/dist-packages/eoxserver/services/ows/wms/basehandlers.py", line 221, in handle result_bytes, content_type, filename = map_renderer.render_map(map_) File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/map_renderer.py", line 113, in render_map layers_plus_factories_plus_data = [ File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/map_renderer.py", line 114, in <listcomp> (layer, factory, factory.create(map_obj, layer)) File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/factories.py", line 455, in create for _ in generator: File "/usr/local/lib/python3.8/dist-packages/eoxserver/render/mapserver/factories.py", line 357, in make_browse_layer_generator ms.set_env(map_obj, creation_info.env, True) File "/usr/local/lib/python3.8/dist-packages/eoxserver/contrib/mapserver.py", line 289, in set_env raise Exception( Exception: Would override previous value of SWIFT_AUTH_TOKEN: <ONE SWITH AUTH TOKEN> with <ANOTHER SWIFT AUTH TOKEN> -->
These errors happens for different areas and for a certain area on all zooms, but on all renderer nodes (problem is not node specific).
Example failing request to the DEM layer zoomed on a single product:
https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=blackbody&time=2011-02-13T16%3A59%3A03Z%2F2014-09-09T16%3A51%3A50Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=13.359375%2C47.63671875%2C13.53515625%2C47.8125
Example passing request to the product itself:
https://dem.pass.copernicus.eu/ows?service=WMS&version=1.3.0&request=GetMap&layers=urn%3Aeop%3ADLR%3ACDEM10%3ACopernicus_DSM_03_N47_00_E013_00%3AV4683-2020_1&format=image%2Fpng&TRANSPARENT=true&width=100&height=100&CRS=EPSG%3A4326&STYLES=&BBOX=47.000000%2C13.000000%2C48.000000%2C14.000000
Example of passing request from another area on DEM:
https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=blackbody&time=2011-03-19T16%3A42%3A37Z%2F2011-03-19T16%3A42%3A38Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=17.578125%2C45.703125%2C18.28125%2C46.40625
Could this be a regression caused by latest eoxserver updates. Version used: `eoxa/eoxserver:release-1.0.0-rc36` (`vs 1.4.4`)
Also tested on `vs` 1.3.11 and `vs` 1.4.1. and happens as well, could be misconfuguration actuallyLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/126Pansharpening on the fly - rendering2021-11-21T16:10:27+01:00Lubomir DoležalPansharpening on the fly - rendering**Implementation of Pansharpening function for renderer**
- Based on registering of multispectral images and panchromatic COG images inside one product, renderer should be configurable to apply the pansharpening operation on the fly (as ...**Implementation of Pansharpening function for renderer**
- Based on registering of multispectral images and panchromatic COG images inside one product, renderer should be configurable to apply the pansharpening operation on the fly (as a browse type?) for resolutions higher than resolution of the multispectral image if possibleFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/125Pansharpening on the fly - registration & preprocessing2021-09-30T11:35:02+02:00Lubomir DoležalPansharpening on the fly - registration & preprocessing**Preprocessing & registration chain needs to be configurable to allow ingestion of images to be used for pansharpening**
- currently we allow just mosaicking of tiled products, but not special handling of multiple images with same bbox
...**Preprocessing & registration chain needs to be configurable to allow ingestion of images to be used for pansharpening**
- currently we allow just mosaicking of tiled products, but not special handling of multiple images with same bbox
- some products have a set of PAN and MS images inside 1 product archive (also tiled), now we just throw away the PAN band via path regex during archive extraction
- ingestion needs to be extended to support the preprocessing & registration of multiple files per product, i.e., a COG holding the multispectral as well as a COG holding the panchromatic bands
These multiple products are then also accessible via the download service.
- This is important to apply for VHR_IMAGE_2015 and Urban Atlas new collections
linked issue which will be closed by this https://gitlab.eox.at/esa/prism/vs/-/issues/24Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/124SAR Preprocessing automation2021-11-21T15:26:57+01:00Lubomir DoležalSAR Preprocessing automation**Implementation of automatic procedures for SAR products ingestion**
- Following emg product types CS0x, RS02, TX01 or along the way all from https://gitlab.eox.at/esa/prism/vs/-/issues/56 need to have new configurations of preprocessor...**Implementation of automatic procedures for SAR products ingestion**
- Following emg product types CS0x, RS02, TX01 or along the way all from https://gitlab.eox.at/esa/prism/vs/-/issues/56 need to have new configurations of preprocessor mainly doing following:
- automatically identify the type and number of bands. Based on this
identification the Preprocessor automatically decides which preprocessing steps to apply for product types where this is configured
- unfortunately, inside one product type different preprocessings will need to be applied (1 band, 2 bands, 4 bands in one product type RS02 or TX01)
- implement gdal_calc configured computations for conversion to decibel range for better viewing based probably on https://gitlab.eox.at/esa/prism/vhr_image_2018/-/blob/master/preprocessor/transform_chain.py#L501-588 https://gitlab.eox.at/esa/prism/vhr_image_2018/-/blob/master/preprocessor/preprocessor.py#L289-301
- ingest backlog products
- Release of updated PASS SW and Operator Guide documents (I would instead add a TN and optionally a operators guide)
- Execution of Test Procedures for new requirements documented in
updated PASS Operator Guide (I would create a separate TN, not put this into operator guide, except for configuration changes, if applicable)
- Execution of regression tests following Test Procedures documented in
updated PASS Operator Guide (that means, before it threw an error, now works) (I would create a separate TN, not put this into operator guide, except for configuration changes, if applicable)
- along the way, would be great to solve https://gitlab.eox.at/esa/prism/vs/-/issues/64, https://gitlab.eox.at/esa/prism/vs/-/issues/53Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/123Across track incidence angle and along track incidence angle not respected by...2021-11-22T16:30:41+01:00Lubomir DoležalAcross track incidence angle and along track incidence angle not respected by opensearch querytwo new metadata fields `across_track_incidence_angle` and `along_track_incidence_angle` added in https://github.com/EOxServer/eoxserver/commit/f1d1cd545fd6868bffd2bd28ca37f2235c3930fe are not correctly respected by opensearch query (fil...two new metadata fields `across_track_incidence_angle` and `along_track_incidence_angle` added in https://github.com/EOxServer/eoxserver/commit/f1d1cd545fd6868bffd2bd28ca37f2235c3930fe are not correctly respected by opensearch query (filtering on this parameter does not happen), but they should be correctly registered as they are listed in metadata list on client details page
- query: https://vhr18.pdas.prism.eox.at/opensearch/collections/VHR_IMAGE_2018_Level_3/atom/?count=1&acrossTrackIncidenceAngle=[-10,60]&cloudCover=[0,2]&bbox=14.992165253879458,51.39996854155048,15.386105484165885,51.787848152909426&start=2018-08-01T00:00:00.000Z&end=2018-08-05T22:58:39.377Z shows that filtering on cloudCover works [0,2], but on acrossTrackIncidenceAngle[-10,60] does not, as value is `-16.4885876115`
- client request leading to this opensearch query: https://vhr18.pdas.prism.eox.at/?x=14.689134&y=50.869664&start=2018-08-01T00%3A00%3A00Z&end=2018-08-05T22%3A58%3A39Z&z=8&leftpanel=true&area=POLYGON%28%2814.992165+51.399969%2C15.386105+51.399969%2C15.386105+51.787848%2C14.992165+51.787848%2C14.992165+51.399969%29%29#https://gitlab.eox.at/esa/prism/vs/-/issues/122Modify nginx config to not cache index.html2021-06-09T22:26:56+02:00Lubomir DoležalModify nginx config to not cache index.html- the problem is that the client html page is taken from browser cache (default browser behavior to my knowledge) and redirects to IDP are not performed on this first client html load attempt but only later on successive requests to the ...- the problem is that the client html page is taken from browser cache (default browser behavior to my knowledge) and redirects to IDP are not performed on this first client html load attempt but only later on successive requests to the opensearch endpoint to fetch the collections. This runs into: CORS error when requesting automatic redirect to the IDP side:
Cross Origin Resource Sharing error: MissingAllowOriginHeader.
- update cache-control headers to never cache index.html fileLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/121demF client misbehave2021-11-22T16:23:07+01:00Lubomir DoležaldemF client misbehave![image](/uploads/cda404f2eae6177b2cb769d90e87b96e/image.png)
Some more investigation needed on this, comparing metadata registered in DB and visualized in the client. It seems to be only on demF and not on other newly registered collec...![image](/uploads/cda404f2eae6177b2cb769d90e87b96e/image.png)
Some more investigation needed on this, comparing metadata registered in DB and visualized in the client. It seems to be only on demF and not on other newly registered collections (sace-emg / emg).
products cover a really long duration both in cache created tiles and timeslider entries
Both symptoms could have same origin actually (very large area on timeslider could create mismatch on mapcache fetched products to renderer) but I am just guessing.
- after this is fixed, we need to purge demF cache and dem cacheLubomir DoležalLubomir Doležal