VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2020-11-27T18:37:41+01:00https://gitlab.eox.at/esa/prism/vs/-/issues/66ingestor listening in daemon mode2020-11-27T18:37:41+01:00Mussab Abdallaingestor listening in daemon modeThe ingestor should be listening to the shared volume with the sftp image `/mnt/data `. Once brows reports are created there, it starts the process of feeding reports into the preprocessor through redis queue.
However, when posting brows...The ingestor should be listening to the shared volume with the sftp image `/mnt/data `. Once brows reports are created there, it starts the process of feeding reports into the preprocessor through redis queue.
However, when posting browse reports, there is no response from the ingestor.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/65sftp permission error2020-11-21T18:10:49+01:00Mussab Abdallasftp permission errorthe current sftp configuration does not allow writing files into `/data/from/fepd`.
The `atmoz/sftp` documentation shows a configuration syntax that is required to grant users permission in /home/data directory.
syntax: `user:pass[:e][...the current sftp configuration does not allow writing files into `/data/from/fepd`.
The `atmoz/sftp` documentation shows a configuration syntax that is required to grant users permission in /home/data directory.
syntax: `user:pass[:e][:uid[:gid[:dir1[,dir2]...]]]`
According to `atmoz/sftp` documentation This configuration is valid if the directory is not already created ( which is not the case in our sftp container)https://gitlab.eox.at/esa/prism/vs/-/issues/64[living] emg preprocessing findings2021-11-21T15:26:57+01:00Lubomir Doležal[living] emg preprocessing findingsLiving document to add findings from emg preprocessing, which will not be fixed immediately
- ~~temp folders do not get cleaned when preprocessing is aborted (by docker stack rm), but data stay there - fills up disk space - WONTFIX~~
- w...Living document to add findings from emg preprocessing, which will not be fixed immediately
- ~~temp folders do not get cleaned when preprocessing is aborted (by docker stack rm), but data stay there - fills up disk space - WONTFIX~~
- we should fix #53 for preprocessor, it is really bothersome, as OOM restarts containers, but whole node is unreachable for a while, so new containers spawn on other node, which now has all of them in 2 node setup
- copy over ngeo new daemon part of using different intermediate key for avoiding losses of reports (key is at least in one queue at a time), beware that there can be multiple registrars/preprocessors picking the items - to avoid multiple instances working on same item, numeric ID of container needs to be present in the intermediate queue
- ~~do not put skips into failed sets on registrar and preprocessor~~
- Registrar: add plausibility check during registration if footprint is completely somewhere else than image bounds with some margin - either just throw a warning or completely fail registration (config based decision)Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/63Post Release 1.0.02021-11-21T14:47:01+01:00Lubomir DoležalPost Release 1.0.0- [x] send 3 metadata files to IDP prod
- [x] configure SFTP + ingestor on production - first attempt on stale https://gitlab.eox.at/esa/prism/vs/-/tree/prod-testing
- [ ] operator guide registrar modularized
- [x] user guide deploy new...- [x] send 3 metadata files to IDP prod
- [x] configure SFTP + ingestor on production - first attempt on stale https://gitlab.eox.at/esa/prism/vs/-/tree/prod-testing
- [ ] operator guide registrar modularized
- [x] user guide deploy new client image
- [x] deploy the whole thingie to prod VMs + check Offnadir angle if filter works (client now works, but core opensearch seems not to be applied) @bucekl -> does not work, moved to #123
- [x] add Authentication flow to TN @bucekl
- [x] add ingestion input flow to TN - either make a test or just document how that part is done (for filedaemon branch https://gitlab.eox.at/esa/prism/vs/-/tree/prod-testing was done, but not finished - see commits there @abdallam
- [x] add reporting output flow to TN - referencing the test done via CI possibly @abdallam
- [x] match requirements from requirements matrix as done in Software release note @fabian.schindler
- [ ] Operators guide preprocessor configuration creation - at least briefly, there is already a config schema in preprocessor, reuse that
- [x] Registry is now open, remove the docker login partsLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/61Rethink pvs-starter vs. helm templates2021-10-14T09:42:06+02:00Stephan Meißlstephan.meissl@eox.atRethink pvs-starter vs. helm templatesNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/58Modularize Registrator2020-11-11T11:46:23+01:00Fabian SchindlerModularize RegistratorFor future deployments beyond PRISM it is necessary to beef up the registrator.
Here is an outline of the planned components
# Registration context
Objects to be passed around to the components. Shall include the following properties:...For future deployments beyond PRISM it is necessary to beef up the registrator.
Here is an outline of the planned components
# Registration context
Objects to be passed around to the components. Shall include the following properties:
- identifier
- metadata
- file paths
# Source abstractions
This should abstract the object storage access to
- List files (with filter?)
- Download files (metadata file)
- Convert to VSI path
The following backends shall be supported:
- S3
- Swift
- Local path
# Registration configuration handler
Configures and ties together the various components
- select files for Registration
- fetch metadata files and extract metadata
- detect product type/level
# Registration backend
Shall create the necessary models, tables, configs, whatever. Only EOxServer planned for now
Rollbacks should be supported as well?
# Pre-handler
Some setup steps not related to actual registration
# Post-handler
Configurable post-registration steps. e.g: reportingFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/56Add emergency preprocessor configuration2021-11-21T15:26:57+01:00Lubomir DoležalAdd emergency preprocessor configurationThis will take a bit more time than vhr18 + dem preprocessor configuration, so making a new issue for it.
- [ ] CS0x - two versions - with and without real numbers preprocessing
- [ ] DM01
- [x] DM02
- [x] EW02
- [x] EW03
- [x] EW01
- ...This will take a bit more time than vhr18 + dem preprocessor configuration, so making a new issue for it.
- [ ] CS0x - two versions - with and without real numbers preprocessing
- [ ] DM01
- [x] DM02
- [x] EW02
- [x] EW03
- [x] EW01
- [x] GE01
- [x] GY01
- [ ] IK02
- [x] KS03
- [ ] RE00
- [ ] RS02
- [ ] SP05
- [x] SP06 + SP07
- [ ] TX01
- [x] PL00
- [x] SW03
- [x] SK00
- [x] PH1A
- [x] PH1BLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/54Add support of KML / KMZ AOI input for spatial filter2020-07-31T11:07:39+02:00Lubomir DoležalAdd support of KML / KMZ AOI input for spatial filterclient feature for ESAclient feature for ESALubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/53Mitigate memory leaks of registrar + (preprocessor)2021-11-21T15:26:57+01:00Lubomir DoležalMitigate memory leaks of registrar + (preprocessor)While listening on register+preprocess redis queues launch actual operation after popping item from queue in a separate process via `subprocess` in order to mitigate memory leaks. Another solution: https://github.com/amoffat/shWhile listening on register+preprocess redis queues launch actual operation after popping item from queue in a separate process via `subprocess` in order to mitigate memory leaks. Another solution: https://github.com/amoffat/shLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/50Render ordering by resolution2021-11-21T15:28:48+01:00Lubomir DoležalRender ordering by resolutionThe idea would be to register image resolution (either from metadata, or from image itself) in a common unit - metres. Then WMS on the layer would be able to sort the products used also based on resolution.
Comments:
We could use this f...The idea would be to register image resolution (either from metadata, or from image itself) in a common unit - metres. Then WMS on the layer would be able to sort the products used also based on resolution.
Comments:
We could use this field: https://github.com/EOxServer/eoxserver/blob/master/eoxserver/resources/coverages/models.py#L266
That can be wired to be used as a parameter for `sortBy`
We would need to set this of course.
Remarks from the geoville report which are somewhat connected: Priority of the images should be individually selectable as the ranking is different within the projects, for example:
Image ranking in Riparian Zones and Natura 2000 (if more images for one location are available)
* Years: 18-19-17
* Months: Sep-Aug-Jul-Jun-May-Oct-Dec-Nov-Apr-Mar-Feb-Jan
* Days: 31-1
HRL Lot 1 – Imperviousness
* Cloud percentage
HRL Lot 4 – Water/Wetness
* Images of the different years separately
If there are images of equal rank, images with the higher quality should be displayed.
Provision of downloadable PSIL layer from WMS, including information on image metadata (e.g. date and name of the scene) and image hierarchyhttps://gitlab.eox.at/esa/prism/vs/-/issues/49Mitigate renderer memory usage2021-11-09T13:50:55+01:00Lubomir DoležalMitigate renderer memory usageattempts:
first try in https://gitlab.eox.at/esa/prism/vhr_image_2018/-/commit/4f6420dde5658c9d54fbafd766cc1f8a99747b79 - later increased to 8GB
then also currently `-max-requests 10 --max-requests-jitter 3` added to gunicorn in https:/...attempts:
first try in https://gitlab.eox.at/esa/prism/vhr_image_2018/-/commit/4f6420dde5658c9d54fbafd766cc1f8a99747b79 - later increased to 8GB
then also currently `-max-requests 10 --max-requests-jitter 3` added to gunicorn in https://gitlab.eox.at/esa/prism/vs/-/commit/25ff31fcff76f8e2a366f54c76d656d66ea9c049
Further investigation can be done.Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/48Extend client detailsDisplay to allow per-product wms endpoint2021-11-09T13:08:27+01:00Lubomir DoležalExtend client detailsDisplay to allow per-product wms endpointon DEM collection -> detailsdisplay, we can not show individual products if they overlap, as it targets the whole layer Will need to select product ID with CQL probably via REPLACE (will need to get the ID of product)
Currently detailsD...on DEM collection -> detailsdisplay, we can not show individual products if they overlap, as it targets the whole layer Will need to select product ID with CQL probably via REPLACE (will need to get the ID of product)
Currently detailsDisplay on DEM collection goes to url https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=earth&time=2011-06-05T17%3A57%3A56Z%2F2011-06-05T17%3A57%3A57Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=0%2C37.96875%2C0.703125%2C38.671875 but on this area&time there are 6 products and always the same one is shown no matter the arrows change in client
Response Fabian:
```
The WMS URL of the product itself is also provided via OpenSearch.
So a switch in the detailsdisplay widget is probably the best:
re-use the Layers of the map view if no specific WMS layer is available, otherwise use the specific one.
I think this needs to be implemented in the client.
```https://gitlab.eox.at/esa/prism/vs/-/issues/47Render ordering by resolution2020-07-23T13:14:04+02:00Stephan Meißlstephan.meissl@eox.atRender ordering by resolutionhttps://gitlab.eox.at/esa/prism/vs/-/issues/46Extract preprocessor steps to allow for better re-use in VS2020-07-23T13:16:03+02:00Fabian SchindlerExtract preprocessor steps to allow for better re-use in VSCurrently the preprocessor in VHR18 is very specific for the use of the products/collections of that project. For the use in VS, the implementation needs to be generalized.
In order to generalize the preprocessor, the necessary steps ca...Currently the preprocessor in VHR18 is very specific for the use of the products/collections of that project. For the use in VS, the implementation needs to be generalized.
In order to generalize the preprocessor, the necessary steps can be extracted:
- Data retrieval: download from the swift object storage (maybe abstract this to allow other sources in the future as well)
- Unpacking: (recursively) unpack downloaded source files
- File selection: from the unpacked files, select the ones that actually have significance in VS (data files and metadata files)
- Data file merging: stacking bands if separated into multiple files, combine tiles into a single file, etc
- Output data file generation: COGs (possibly others)
- Metadata extraction
- Upload to output swift bucket (maybe other object storages in the future)https://gitlab.eox.at/esa/prism/vs/-/issues/45mitigate renderer eating up all memory2020-07-23T12:09:10+02:00Stephan Meißlstephan.meissl@eox.atmitigate renderer eating up all memoryhttps://gitlab.eox.at/esa/prism/vs/-/issues/44DEM collection detailsDisplay issue2020-07-23T12:05:23+02:00Lubomir DoležalDEM collection detailsDisplay issueon DEM collection -> detailsdisplay, we can not show individual products if they overlap, as it targets the whole layer
Will need to select product ID with CQL probably via REPLACE (will need to get the ID of product)
Currently detailsD...on DEM collection -> detailsdisplay, we can not show individual products if they overlap, as it targets the whole layer
Will need to select product ID with CQL probably via REPLACE (will need to get the ID of product)
Currently detailsDisplay on DEM collection goes to url https://dem.pass.copernicus.eu/ows?SERVICE=WMS&VERSION=1.1.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=DEM&STYLES=earth&time=2011-06-05T17%3A57%3A56Z%2F2011-06-05T17%3A57%3A57Z&WIDTH=512&HEIGHT=512&SRS=EPSG%3A4326&BBOX=0%2C37.96875%2C0.703125%2C38.671875
but on this area&time there are 6 products and always the same one is shown no matter the arrows change in client
- [x] also layer change in detailsDisplay via display.extraParameters.LAYERS is now broken with update to wmtsLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/33Support for additional “dimensions” like cloud coverage2021-11-22T16:02:26+01:00Fabian SchindlerSupport for additional “dimensions” like cloud coverageECSYS32/SYSAC-10ECSYS32/SYSAC-10https://gitlab.eox.at/esa/prism/vs/-/issues/30Idea: if gsc does not contain footprint, registrar/preprocessor could attempt...2021-11-09T13:22:18+01:00Lubomir DoležalIdea: if gsc does not contain footprint, registrar/preprocessor could attempt to recreate it at least approximately from image boundsI think preprocessor in ngeo already has some code for itI think preprocessor in ngeo already has some code for ithttps://gitlab.eox.at/esa/prism/vs/-/issues/29GetCapabilities of a single product could show available dim_bands2021-11-09T13:55:24+01:00Lubomir DoležalGetCapabilities of a single product could show available dim_bandsvia cql=identifier=... or via layers= (with and without __coverage)via cql=identifier=... or via layers= (with and without __coverage)https://gitlab.eox.at/esa/prism/vs/-/issues/27Configuration to allow a single or multiple buckets to store preprocessed dat...2020-08-18T10:47:18+02:00Fabian SchindlerConfiguration to allow a single or multiple buckets to store preprocessed data / metadata