VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2021-09-30T11:35:02+02:00https://gitlab.eox.at/esa/prism/vs/-/issues/125Pansharpening on the fly - registration & preprocessing2021-09-30T11:35:02+02:00Lubomir DoležalPansharpening on the fly - registration & preprocessing**Preprocessing & registration chain needs to be configurable to allow ingestion of images to be used for pansharpening**
- currently we allow just mosaicking of tiled products, but not special handling of multiple images with same bbox
...**Preprocessing & registration chain needs to be configurable to allow ingestion of images to be used for pansharpening**
- currently we allow just mosaicking of tiled products, but not special handling of multiple images with same bbox
- some products have a set of PAN and MS images inside 1 product archive (also tiled), now we just throw away the PAN band via path regex during archive extraction
- ingestion needs to be extended to support the preprocessing & registration of multiple files per product, i.e., a COG holding the multispectral as well as a COG holding the panchromatic bands
These multiple products are then also accessible via the download service.
- This is important to apply for VHR_IMAGE_2015 and Urban Atlas new collections
linked issue which will be closed by this https://gitlab.eox.at/esa/prism/vs/-/issues/24Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/124SAR Preprocessing automation2021-11-21T15:26:57+01:00Lubomir DoležalSAR Preprocessing automation**Implementation of automatic procedures for SAR products ingestion**
- Following emg product types CS0x, RS02, TX01 or along the way all from https://gitlab.eox.at/esa/prism/vs/-/issues/56 need to have new configurations of preprocessor...**Implementation of automatic procedures for SAR products ingestion**
- Following emg product types CS0x, RS02, TX01 or along the way all from https://gitlab.eox.at/esa/prism/vs/-/issues/56 need to have new configurations of preprocessor mainly doing following:
- automatically identify the type and number of bands. Based on this
identification the Preprocessor automatically decides which preprocessing steps to apply for product types where this is configured
- unfortunately, inside one product type different preprocessings will need to be applied (1 band, 2 bands, 4 bands in one product type RS02 or TX01)
- implement gdal_calc configured computations for conversion to decibel range for better viewing based probably on https://gitlab.eox.at/esa/prism/vhr_image_2018/-/blob/master/preprocessor/transform_chain.py#L501-588 https://gitlab.eox.at/esa/prism/vhr_image_2018/-/blob/master/preprocessor/preprocessor.py#L289-301
- ingest backlog products
- Release of updated PASS SW and Operator Guide documents (I would instead add a TN and optionally a operators guide)
- Execution of Test Procedures for new requirements documented in
updated PASS Operator Guide (I would create a separate TN, not put this into operator guide, except for configuration changes, if applicable)
- Execution of regression tests following Test Procedures documented in
updated PASS Operator Guide (that means, before it threw an error, now works) (I would create a separate TN, not put this into operator guide, except for configuration changes, if applicable)
- along the way, would be great to solve https://gitlab.eox.at/esa/prism/vs/-/issues/64, https://gitlab.eox.at/esa/prism/vs/-/issues/53Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/96EMG SARs ingestion preprocessor configuration2021-06-17T15:58:08+02:00Lubomir DoležalEMG SARs ingestion preprocessor configuration- to unblock the TTO- to unblock the TTOLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/64[living] emg preprocessing findings2021-11-21T15:26:57+01:00Lubomir Doležal[living] emg preprocessing findingsLiving document to add findings from emg preprocessing, which will not be fixed immediately
- ~~temp folders do not get cleaned when preprocessing is aborted (by docker stack rm), but data stay there - fills up disk space - WONTFIX~~
- w...Living document to add findings from emg preprocessing, which will not be fixed immediately
- ~~temp folders do not get cleaned when preprocessing is aborted (by docker stack rm), but data stay there - fills up disk space - WONTFIX~~
- we should fix #53 for preprocessor, it is really bothersome, as OOM restarts containers, but whole node is unreachable for a while, so new containers spawn on other node, which now has all of them in 2 node setup
- copy over ngeo new daemon part of using different intermediate key for avoiding losses of reports (key is at least in one queue at a time), beware that there can be multiple registrars/preprocessors picking the items - to avoid multiple instances working on same item, numeric ID of container needs to be present in the intermediate queue
- ~~do not put skips into failed sets on registrar and preprocessor~~
- Registrar: add plausibility check during registration if footprint is completely somewhere else than image bounds with some margin - either just throw a warning or completely fail registration (config based decision)Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/53Mitigate memory leaks of registrar + (preprocessor)2021-11-21T15:26:57+01:00Lubomir DoležalMitigate memory leaks of registrar + (preprocessor)While listening on register+preprocess redis queues launch actual operation after popping item from queue in a separate process via `subprocess` in order to mitigate memory leaks. Another solution: https://github.com/amoffat/shWhile listening on register+preprocess redis queues launch actual operation after popping item from queue in a separate process via `subprocess` in order to mitigate memory leaks. Another solution: https://github.com/amoffat/shLubomir DoležalLubomir Doležal