VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2020-11-11T11:59:30+01:00https://gitlab.eox.at/esa/prism/vs/-/issues/11Explore `docker secrets` as a substitute for passwords in db and obs env2020-11-11T11:59:30+01:00Lubomir DoležalExplore `docker secrets` as a substitute for passwords in db and obs envBaseline evolutionMussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/9Integrate ELK stack with Fluentd2020-11-11T11:54:54+01:00Fabian SchindlerIntegrate ELK stack with FluentdBaseline evolutionFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/8Testing using gitlab.ci2021-11-09T13:24:25+01:00Fabian SchindlerTesting using gitlab.ci**How is tested:**
- gitlab.ci: split current build stage in build and deploy. Add testing stage in between
**Preparations:**
- Create a new bucket with
- Input data (copied from ONDA): for each mission a couple files
- Preprocess...**How is tested:**
- gitlab.ci: split current build stage in build and deploy. Add testing stage in between
**Preparations:**
- Create a new bucket with
- Input data (copied from ONDA): for each mission a couple files
- Preprocessed data (copy from our internal buckets)
**Setup:**
- Override root bucket to `testing_<commit-hash>`
- Pass this bucket name to configure.sh
**What is tested:**
- **ingestor:**
- input: browse report put via sftp to pyinotify watched folder
- output: item gets parsed, put to preprocess queue, and is preprocessed
- **preprocessor:**
- input: input data from our own static test data bucket
- output: preprocessed images stored on testing bucket
- tests:
- [x] existence of file
- [x] structure of preprocessed image (format, size, overviews)
- **registrar/renderer/seeder/cache:**
- input: preprocessed image on static test data bucket
- output: database models
- tests:
- [x] existence of DB models
- [ ] Testing renderer services
- [x] OpenSearch
- [x] WMS
- [x] WCS
- [ ] DSEO
- Testing cache services
- [ ] WMS
- **redis flow - ingestor/sftp -> preprocessor -> registrar -> reporting/sftp**
- [ ] tpz accessibility item as input
**Teardown:**
- [x] Delete the bucket from the setup stage
**Useful approach:**
- re-run the gitlab.ci steps locally to avoid multiple pushingBaseline evolutionMussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/7Create service to allow sending browse reports (and images) against2020-11-11T11:57:38+01:00Fabian SchindlerCreate service to allow sending browse reports (and images) againstold BROW way:
- upload images via WebDAV
- send Browse Report to `/ingest/`
VHR_IMAGE_2018 way:
- preprocessor listens on Redis queue and expects paths on Swift
new Ingestion Service that:
- accepts browse reports with references t...old BROW way:
- upload images via WebDAV
- send Browse Report to `/ingest/`
VHR_IMAGE_2018 way:
- preprocessor listens on Redis queue and expects paths on Swift
new Ingestion Service that:
- accepts browse reports with references to images on Swift
- extract the metadata (id, time, footprint, image reference)
- puts the metadata into collection specific preprocessing queuesBaseline evolutionFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/6Create script to bootstrap collections (docker-compose, configs, env)2020-11-11T11:53:37+01:00Fabian SchindlerCreate script to bootstrap collections (docker-compose, configs, env)Baseline evolutionFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/43assess performance of deployed PASS service2020-06-30T17:18:29+02:00Stephan Meißlstephan.meissl@eox.atassess performance of deployed PASS service* [x] performance regarding parallel users
* [x] performance regarding functionality
* [x] compare with requirements
The specific requirements are:
* Both protocols (WMS & WCS) supporting up to 100 requests in parallel.
* WMTS response ...* [x] performance regarding parallel users
* [x] performance regarding functionality
* [x] compare with requirements
The specific requirements are:
* Both protocols (WMS & WCS) supporting up to 100 requests in parallel.
* WMTS response time for a tile: between 500 to 1500 milliseconds.Work order closurehttps://gitlab.eox.at/esa/prism/vs/-/issues/42Training material2020-07-31T11:07:46+02:00Fabian SchindlerTraining materialSome slides
Can use materials from the User/Operator guideSome slides
Can use materials from the User/Operator guideWork order closurehttps://gitlab.eox.at/esa/prism/vs/-/issues/41Test Report update2020-06-30T17:18:52+02:00Fabian SchindlerTest Report updateWork order closurehttps://gitlab.eox.at/esa/prism/vs/-/issues/40Operator Guide2020-06-30T17:18:22+02:00Fabian SchindlerOperator GuideDetailing:
* Installation
* Operation
* Handling of the individual services
* ...Detailing:
* Installation
* Operation
* Handling of the individual services
* ...Work order closurehttps://gitlab.eox.at/esa/prism/vs/-/issues/39User guide update2020-06-30T18:12:38+02:00Fabian SchindlerUser guide updateWork order closureLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/38Software KIT & SRN2020-06-30T18:12:19+02:00Fabian SchindlerSoftware KIT & SRNEntailing all software items
* Source code: `git archive`
* Software itself: `docker export`
SRN is basically a listing of all itemsEntailing all software items
* Source code: `git archive`
* Software itself: `docker export`
SRN is basically a listing of all itemsWork order closureFabian SchindlerFabian Schindlerhttps://gitlab.eox.at/esa/prism/vs/-/issues/152Investigate multi project pipelines2021-11-21T16:29:00+01:00Nikola JankovicInvestigate multi project pipelinesCould be useful for integration testing once repos are separated:
- https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
- https://about.gitlab.com/blog/2018/10/31/use-multiproject-pipelines-with-gitlab-cicd/Could be useful for integration testing once repos are separated:
- https://docs.gitlab.com/ee/ci/pipelines/multi_project_pipelines.html
- https://about.gitlab.com/blog/2018/10/31/use-multiproject-pipelines-with-gitlab-cicd/ViewServer 2.0Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/151Branch v2.0 regressions2021-11-21T16:36:08+01:00Lubomir DoležalBranch v2.0 regressions- [x] Post handler in registrar does not work (reports)
- [ ] Registrar can only register products from the configured swift object storage container, not from arbitrary, which would be part of STAC item href
- [x] preprocessor can not c...- [x] Post handler in registrar does not work (reports)
- [ ] Registrar can only register products from the configured swift object storage container, not from arbitrary, which would be part of STAC item href
- [x] preprocessor can not create STAC item for already preprocessed products anew
- [x] preprocessor needs regex based asset name mapping of outputs, sorting is unrealiableViewServer 2.0https://gitlab.eox.at/esa/prism/vs/-/issues/144Create common software library to map to STAC2021-11-09T12:15:03+01:00Fabian SchindlerCreate common software library to map to STACThe task of the library is to map from whatever input is provided to a (list of) STAC items
Common use cases for major platforms should be covered by this library.
Examples:
- CREODIAS - OpenSearch - Sentinel-2:
- interpret OpenSear...The task of the library is to map from whatever input is provided to a (list of) STAC items
Common use cases for major platforms should be covered by this library.
Examples:
- CREODIAS - OpenSearch - Sentinel-2:
- interpret OpenSearch response (GeoJSON)
- pick `productIdentifier` property
- transform this to a valid S3 URL
- use stactools + sentinel2 to lookup that object storage prefix to create a STAC Item
- PRISM
- read list of object storage prefixes
- transform this to STAC items using the stored metadata thereViewServer 2.0Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/141STAC: Registrar2021-11-09T15:15:44+01:00Fabian SchindlerSTAC: RegistrarFind an approach of how STAC can be adopted as an input format and how the configuration must be adopted in order to support the change.Find an approach of how STAC can be adopted as an input format and how the configuration must be adopted in order to support the change.ViewServer 2.0Nikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/140STAC: Preprocessor2021-11-09T15:03:19+01:00Fabian SchindlerSTAC: PreprocessorFind an approach of how STAC can be adopted as an input/output format and how the configuration must be adopted in order to support the change.
- Preprocessor adds metadata to the actual content of STAC (image url + metadata), reference...Find an approach of how STAC can be adopted as an input/output format and how the configuration must be adopted in order to support the change.
- Preprocessor adds metadata to the actual content of STAC (image url + metadata), reference to the GSC etc.ViewServer 2.0Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/139STAC: Ingestor2021-11-21T16:22:10+01:00Fabian SchindlerSTAC: IngestorFind an approach to adopt STAC as an output format for the Ingestor.
How are the fields from the Browse Report XML mapped to the STAC item properties.
Example Browse Reports:
- Footprint browse: https://github.com/EOX-A/ngeo-b_autotes...Find an approach to adopt STAC as an output format for the Ingestor.
How are the fields from the Browse Report XML mapped to the STAC item properties.
Example Browse Reports:
- Footprint browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/reference_test_data/browseReport_ASA_IM__0P_20100722_213840.xml
- Regular Grid browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/test_data/ASA_WSM_1PNDPA20050331_075939_000000552036_00035_16121_0775.xml
- Rectified browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/test_data/S2.xml
- Model in GeoTIFF browse: https://github.com/EOX-A/ngeo-b_autotest/blob/branch-4-1/data/test_data/rotated_axes.xmlViewServer 2.0Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/138Investigate usage of Mapchete as a preprocessing tool2021-11-22T16:06:15+01:00Fabian SchindlerInvestigate usage of Mapchete as a preprocessing toolViewServer 2.0https://gitlab.eox.at/esa/prism/vs/-/issues/137Splitting VS into component repositories2021-11-22T16:14:23+01:00Fabian SchindlerSplitting VS into component repositoriesShould help with unit testing, documentation
Tasks:
- ~~Create gitlab group~~ -> `vs` group
- ~~Create repository for each specific component~~
- ~~Move source code~~
- ~~Scan for component specific issues and move them to the repo~...Should help with unit testing, documentation
Tasks:
- ~~Create gitlab group~~ -> `vs` group
- ~~Create repository for each specific component~~
- ~~Move source code~~
- ~~Scan for component specific issues and move them to the repo~~
- move deployment configurations to a separate repository -> https://gitlab.eox.at/esa/prism/configsViewServer 2.0https://gitlab.eox.at/esa/prism/vs/-/issues/136Adopt vsq2021-11-22T16:44:31+01:00Fabian SchindlerAdopt vsq`vsq` (https://gitlab.eox.at/esa/prism/vsq) should be adopted as the main queue interface for all components as it abstracts common tasks like enqueuing/fetching, waiting for results, setting up of a daemon, etc.
- Ensure that a compone...`vsq` (https://gitlab.eox.at/esa/prism/vsq) should be adopted as the main queue interface for all components as it abstracts common tasks like enqueuing/fetching, waiting for results, setting up of a daemon, etc.
- Ensure that a component can listen on and push to multiple configured queues rather than just one.
- copy over ngeo new daemon part of using different intermediate key for avoiding losses of reports (key is at least in one queue at a time), beware that there can be multiple registrars/preprocessors picking the items - to avoid multiple instances working on same item, numeric ID of container needs to be present in the intermediate queueViewServer 2.0