VS issueshttps://gitlab.eox.at/esa/prism/vs/-/issues2021-05-06T09:57:28+02:00https://gitlab.eox.at/esa/prism/vs/-/issues/80Move client repository eoxc/prism here2021-05-06T09:57:28+02:00Lubomir DoležalMove client repository eoxc/prism here- To allow development on actual data from renderer without the CORS plugin, the https://github.com/eoxc/prism `update-eoxc` checked out to branch should be moved to `vs` repository
- Build of the client docker image will include buildin...- To allow development on actual data from renderer without the CORS plugin, the https://github.com/eoxc/prism `update-eoxc` checked out to branch should be moved to `vs` repository
- Build of the client docker image will include building the js client
- remove hardcoded docs pdf and html links from https://github.com/eoxc/prism/blob/update-eoxc/src/languages/en.json#L23DEM First ReleaseLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/79Ingestor fails to save success/fail file if already exists.2021-02-02T17:14:55+01:00Lubomir DoležalIngestor fails to save success/fail file if already exists.Ingestor:
When it tries to save the "ingestor success" file when it is already present there. (ingested before already), the commands crashes because it already exists. Here are the logs from one such event on reference platform.
```
Feb...Ingestor:
When it tries to save the "ingestor success" file when it is already present there. (ingested before already), the commands crashes because it already exists. Here are the logs from one such event on reference platform.
```
Feb 2, 2021 @ 09:17:52.950 [2021-02-02 08:17:52,947][filedaemon] ERROR: Destination path '/var/ingestor/success/CDD_PDAS_20200925_090956239.xml' already exists
Feb 2, 2021 @ 09:17:52.950 Traceback (most recent call last):
Feb 2, 2021 @ 09:17:52.950 File "/filedaemon.py", line 101, in process_IN_CLOSE_WRITE
Feb 2, 2021 @ 09:17:52.950 save_mount_report(event.pathname, True)
Feb 2, 2021 @ 09:17:52.950 File "/ingestor/util.py", line 49, in save_mount_report
Feb 2, 2021 @ 09:17:52.950 shutil.move(data, save_dir)
Feb 2, 2021 @ 09:17:52.950 File "/usr/lib/python3.6/shutil.py", line 548, in move
Feb 2, 2021 @ 09:17:52.950 raise Error("Destination path '%s' already exists" % real_dst)
Feb 2, 2021 @ 09:17:52.950 shutil.Error: Destination path '/var/ingestor/success/CDD_PDAS_20200925_090956239.xml' already exists
Feb 2, 2021 @ 09:17:52.945 [2021-02-02 08:17:52,944][filedaemon] INFO: Parsing browse file: /mnt/data/CDD_PDAS_20200925_090956239.xml
Feb 2, 2021 @ 09:17:52.945 [2021-02-02 08:17:52,945][filedaemon] DEBUG: data26/0000120743/SP07_NAO_MS4__3_20180922T122315_20180922T122332_TOU_1234_aa94.DIMA.tar
Feb 2, 2021 @ 09:17:52.938 [2021-02-02 08:17:52,938][filedaemon] DEBUG: data26/0000120698/SP07_NAO_MS4__3_20190725T095121_20190725T095133_TOU_1234_d25e.DIMA.tar
Feb 2, 2021 @ 09:17:52.936 [2021-02-02 08:17:52,933][filedaemon] INFO: Parsing browse file: /mnt/data/CDD_PDAS_20200925_090742918.xml
```
It does not crash the whole ingestion, as this step happens after putting item to preprocess queue.
Solution: overwrite the file or fail silently.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/78Emergency/DEM cache below certain zoom (EMG) or with a certain pattern (DEM) ...2021-01-21T10:18:33+01:00Lubomir DoležalEmergency/DEM cache below certain zoom (EMG) or with a certain pattern (DEM) returns 502When zooming below certain zoom on Emergency and on random tiles from DEM, mapcache returns 502 with following errors:
```
| Jan 5, 2021 @ 11:39:00.439 | realloc(): invalid next size |
| Jan 5, 2021 @ 11:39:01.456 | [Tue Jan 05 10:39:01....When zooming below certain zoom on Emergency and on random tiles from DEM, mapcache returns 502 with following errors:
```
| Jan 5, 2021 @ 11:39:00.439 | realloc(): invalid next size |
| Jan 5, 2021 @ 11:39:01.456 | [Tue Jan 05 10:39:01.454923 2021] [core:notice] [pid 132:tid 140477779360704] AH00052: child pid 1428 exit signal Aborted (6) |
```
example request with enough available memory on caches (restarted all cache containers):
TileMatrix 15 error:
https://emg.pass.copernicus.eu/cache/ows/wmts?layer=Emergency__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=15&TileCol=21662&TileRow=13694&time=2018-05-01T00%3A00%3A00Z%2F2018-05-08T06%3A46%3A32Z
accessed from client https://emg.pdas.prism.eox.at/?x=-60.987155&y=14.785008&z=16×tart=2018-05-01T00%3A00%3A00Z&timeend=2018-05-08T06%3A46%3A32Z
saved yes: Emergency__TRUE_COLOR/WGS84/2018-05-04T14:26:05Z/2018-05-04T14:26:08Z/14/10829/9537.xxx
saved not: Emergency__TRUE_COLOR/WGS84/2018-05-04T14:26:05Z/2018-05-04T14:26:08Z/15/13689/9537.xxx
TileMatrix 14 error:
https://emg.pass.copernicus.eu/cache/ows/wmts?layer=Emergency__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=14&TileCol=17996&TileRow=4092&time=2019-03-01T00%3A00%3A00Z%2F2019-04-01T00%3A00%3A00Z
accessed from client:
https://emg.pdas.prism.eox.at/?x=17.718770&y=45.017191&z=15×tart=2019-03-01T00%3A00%3A00Z&timeend=2019-04-01T00%3A00%3A00Z
saved yes: Emergency__TRUE_COLOR/WGS84/2019-03-28T09:49:41Z/2019-03-28T09:49:44Z/13/8998/6145.xxx
saved not: Emergency__TRUE_COLOR/WGS84/2019-03-28T09:49:41Z/2019-03-28T09:49:44Z/14/17994/????.xxx
Problem was not seen on VHR_IMAGE_2018 at all
VHR_IMAGE_2018 works even up to z=17
https://vhr18.pass.copernicus.eu/cache/ows/wmts?layer=VHR_IMAGE_2018_Level_3__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=17&TileCol=119009&TileRow=44932&time=2018-01-30T21%3A04%3A46Z%2F2019-03-08T10%3A02%3A54Z
DEM collection has this issue happening with a different pattern, but not randomly (for some tiles of a single product yes, for some not) even on TileMatrix 10 but if you are viewing individual layers, which compose the DEM layer (DEM_COP-DEM_EEA-10-DGED__EARTH for example), none of those runs into an error.
https://dem.pass.copernicus.eu/cache/ows/wmts?layer=DEM__EARTH&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=8&TileCol=264&TileRow=64&time=2011-03-27T00%3A00%3A00Z%2F2011-04-03T00%3A00%3A00Z -> error
https://dem.pass.copernicus.eu/cache/ows/wmts?layer=DEM_COP-DEM_EEA-10-DGED__EARTH&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=8&TileCol=264&TileRow=65&time=2011-03-27T00%3A00%3A00Z%2F2011-04-03T00%3A00%3A00Z - runs fine and same with all six individual layers
-> seen on client https://dem.pdas.prism.eox.at/?x=4.866364&y=45.224732&z=9×tart=2011-03-27T00%3A00%3A00Z&timeend=2011-04-03T00%3A00%3A00Z
Suspicion that the length of the key stored to the object storage would be an issue was denied.
Size of products should also not be a problem as some DEM products are smaller and some larger than average VHR products, but all of them work in individual DEM layers, but on DEM shared one not.
Problem is happening on all nodes.https://gitlab.eox.at/esa/prism/vs/-/issues/77Wrong cache tiles returned2021-04-03T15:17:31+02:00Fabian SchindlerWrong cache tiles returnedMaybe due to an issue in the mapcache.xml PostgreSQL query
![Bildschirmfoto_2020-12-23_um_11.11.07](/uploads/67f0db0c674fafa2232725fc90db6fa0/Bildschirmfoto_2020-12-23_um_11.11.07.png)Maybe due to an issue in the mapcache.xml PostgreSQL query
![Bildschirmfoto_2020-12-23_um_11.11.07](/uploads/67f0db0c674fafa2232725fc90db6fa0/Bildschirmfoto_2020-12-23_um_11.11.07.png)DEM First ReleaseLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/76client download functionality regression2021-11-09T13:04:59+01:00Lubomir Doležalclient download functionality regression- broken for emergency and any newly registered collection
- we used to rewrite `productid` to `productid__coverage` to get `coverageid` but now naming of coverage identifiers changed to be in format `productid__rasteridentifier__coverag...- broken for emergency and any newly registered collection
- we used to rewrite `productid` to `productid__coverage` to get `coverageid` but now naming of coverage identifiers changed to be in format `productid__rasteridentifier__coverage` which is shown correctly in WCS links in summary, but download panel can not use those yet - need to be implemented in `eoxc`. During implementation, take in account, that more coverages can now be available for one product (pan/ms). or multiband 1 band = 1 coverage for S2L2A - for example here https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/opensearch/collections/S2L2A/atom/?bbox=-8.302734375,38.859619140625,32.302734375,59.140380859375&start=2020-08-01T00:00:00.000Z&end=2020-08-31T23:59:59.000Z
```
<owc:operation code="GetCoverage" method="GET" type="image/tiff" href="https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/ows?service=WCS&version=2.0.1&request=GetCoverage&coverageId=S2A_33UWP_20200816_0_L2A_B02"/>
<owc:operation code="DescribeCoverage" method="GET" type="application/xml" href="https://vs-s52gfrubv875jlgrfdsa.demo.hub.eox.at/ows?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=S2A_33UWP_20200816_0_L2A_B03"/>
```
- also breaks in EOEPCALubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/75Eoxserver gdal could use version 3.2.x2021-11-21T15:46:41+01:00Lubomir DoležalEoxserver gdal could use version 3.2.x- could increase rendering performance due to decoding performance increase of deflate compression
- also should benefit jpeg2000 in case files were only registered (s2 in eoepca?)- could increase rendering performance due to decoding performance increase of deflate compression
- also should benefit jpeg2000 in case files were only registered (s2 in eoepca?)https://gitlab.eox.at/esa/prism/vs/-/issues/74reporting interface filenames, replace colon2020-12-16T10:21:40+01:00Lubomir Doležalreporting interface filenames, replace colonname includes unsupported characters (at least for Windows, where colon `:` can not be part of filename). Replace potentially unsupported characters by `_` for example [item_20201209T122658_urn_eop_EUSI_EW02_10300500ACF8D700_013448908010...name includes unsupported characters (at least for Windows, where colon `:` can not be part of filename). Replace potentially unsupported characters by `_` for example [item_20201209T122658_urn_eop_EUSI_EW02_10300500ACF8D700_013448908010.xml](/uploads/9afba82c72fb52fdae24344166b29d42/item_20201209T122658_urn_eop_EUSI_EW02_10300500ACF8D700_013448908010.xml).https://gitlab.eox.at/esa/prism/vs/-/issues/73add persistent volume to fluentd2021-11-21T15:45:38+01:00Lubomir Doležaladd persistent volume to fluentd- If fluentd stops getting logs from some stack, they are lost when we redeploy stack, although they might have been buffered?
- would persistent volume for fluentd help here?- If fluentd stops getting logs from some stack, they are lost when we redeploy stack, although they might have been buffered?
- would persistent volume for fluentd help here?https://gitlab.eox.at/esa/prism/vs/-/issues/72Preprocessor buckets for tests need to be uniquely named2021-01-28T15:34:49+01:00Lubomir DoležalPreprocessor buckets for tests need to be uniquely named... to prevent one running pipeline deleting content of another pipeline... to prevent one running pipeline deleting content of another pipelinehttps://gitlab.eox.at/esa/prism/vs/-/issues/71Mapcache rendering issues2020-12-03T09:43:14+01:00Lubomir DoležalMapcache rendering issuesFollowing tile has a row of transparent pixels 1 pixel wide on top, right and bottom of the tile.
https://emg.pass.copernicus.eu/cache/ows/wmts?layer=Emergency__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Ve...Following tile has a row of transparent pixels 1 pixel wide on top, right and bottom of the tile.
https://emg.pass.copernicus.eu/cache/ows/wmts?layer=Emergency__TRUE_COLOR&style=default&tilematrixset=WGS84&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fpng&TileMatrix=13&TileCol=9392&TileRow=2298&time=2020-01-24T09%3A26%3A34Z%2F2020-01-24T09%3A26%3A43Z
This produces unpleasant artifacts while viewing the layer in client.
It was confirmed on other collections as well, but only for new tiles, therefore I suspect mapcache update to 1.10 to be culprit, as there have not been any other cache/renderer related updates on production machines since then?
Not sure how to fix this. @fabian.schindler says, that this problem was already observed before on NDVI rendering.
![image](/uploads/7e0c9b46683f12f3da8f3bfff57c9cc0/image.png)
![image](/uploads/6814878b4f7a81965b11caf9cb166347/image.png)Baseline evolution v2https://gitlab.eox.at/esa/prism/vs/-/issues/70full prism functionality test2021-11-09T13:48:05+01:00Mussab Abdallafull prism functionality testA test which would verify the functionality of the entire prism stack should be added.
Currently a product is being ingested and the expected processed result is inspected, this test can be expanded so that the registration / reporting ...A test which would verify the functionality of the entire prism stack should be added.
Currently a product is being ingested and the expected processed result is inspected, this test can be expanded so that the registration / reporting / rendering of the ingested product can be tested.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/69Registrar updates2020-11-27T18:40:31+01:00Lubomir DoležalRegistrar updates- [x] it would be great to extend registrar to allow more coverage types per product type - extend to list and try more until one passes
- [x] update emg init db config, removing different product types per band count, but adding differe...- [x] it would be great to extend registrar to allow more coverage types per product type - extend to list and try more until one passes
- [x] update emg init db config, removing different product types per band count, but adding different coverage types
- [x] deploy changes on prod (migrate products from _bandcount product types to single product type (EW03_3 -> EW03)
summary:
- create new coverage type: BGRNir, sar_hh_hv_vh_vv_gray
- create new product types
- migrate all products from RGBNir to BGRNir cov type (GE01_4, GY01_4, EQ02_4, EW02_4, EW03_4) and change prod type
- migrate RGB product types to shared one (GE01_3, EQ02, EW02, EW03)
- remove sar_hh_hv_vh_vv_rgb, purge it to oblivion, TX01_7, RS02_7
- make just hh_hv_vh_vv from it
- reprocess sar_hh_hv_vh_vv_rgb -> ideally by downloading IMG, removing last 3 bands and reuploading it to same name
- Migrate TX01_3 and TX01_2 to TX01
- Migrate RS02_3 and RS02_2 to RS02
- Migrate EW02_8 and EW03_8 to EW02, EW03
- clean emg cache afterwardsBaseline evolution v2Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/68Ingestor: move browse report files to SUCCESS/FAILED directories2021-01-15T13:59:27+01:00Fabian SchindlerIngestor: move browse report files to SUCCESS/FAILED directoriesFor both POST and file based ingestion: browse reports need to be placed in either the `SUCCESS` or `FAILED` directories.
Due to architectural constraints, we cannot know in the ingestor whether preprocessing/registration was successful,...For both POST and file based ingestion: browse reports need to be placed in either the `SUCCESS` or `FAILED` directories.
Due to architectural constraints, we cannot know in the ingestor whether preprocessing/registration was successful, so we treat successful parsing and pushing to the redis queue as success.
`SUCCESS` and `FAILURE` directories need to be configurable (env var is sufficient) with a sensible default.Baseline evolutionMussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/66ingestor listening in daemon mode2020-11-27T18:37:41+01:00Mussab Abdallaingestor listening in daemon modeThe ingestor should be listening to the shared volume with the sftp image `/mnt/data `. Once brows reports are created there, it starts the process of feeding reports into the preprocessor through redis queue.
However, when posting brows...The ingestor should be listening to the shared volume with the sftp image `/mnt/data `. Once brows reports are created there, it starts the process of feeding reports into the preprocessor through redis queue.
However, when posting browse reports, there is no response from the ingestor.Mussab AbdallaMussab Abdallahttps://gitlab.eox.at/esa/prism/vs/-/issues/65sftp permission error2020-11-21T18:10:49+01:00Mussab Abdallasftp permission errorthe current sftp configuration does not allow writing files into `/data/from/fepd`.
The `atmoz/sftp` documentation shows a configuration syntax that is required to grant users permission in /home/data directory.
syntax: `user:pass[:e][...the current sftp configuration does not allow writing files into `/data/from/fepd`.
The `atmoz/sftp` documentation shows a configuration syntax that is required to grant users permission in /home/data directory.
syntax: `user:pass[:e][:uid[:gid[:dir1[,dir2]...]]]`
According to `atmoz/sftp` documentation This configuration is valid if the directory is not already created ( which is not the case in our sftp container)https://gitlab.eox.at/esa/prism/vs/-/issues/64[living] emg preprocessing findings2021-11-21T15:26:57+01:00Lubomir Doležal[living] emg preprocessing findingsLiving document to add findings from emg preprocessing, which will not be fixed immediately
- ~~temp folders do not get cleaned when preprocessing is aborted (by docker stack rm), but data stay there - fills up disk space - WONTFIX~~
- w...Living document to add findings from emg preprocessing, which will not be fixed immediately
- ~~temp folders do not get cleaned when preprocessing is aborted (by docker stack rm), but data stay there - fills up disk space - WONTFIX~~
- we should fix #53 for preprocessor, it is really bothersome, as OOM restarts containers, but whole node is unreachable for a while, so new containers spawn on other node, which now has all of them in 2 node setup
- copy over ngeo new daemon part of using different intermediate key for avoiding losses of reports (key is at least in one queue at a time), beware that there can be multiple registrars/preprocessors picking the items - to avoid multiple instances working on same item, numeric ID of container needs to be present in the intermediate queue
- ~~do not put skips into failed sets on registrar and preprocessor~~
- Registrar: add plausibility check during registration if footprint is completely somewhere else than image bounds with some margin - either just throw a warning or completely fail registration (config based decision)Lubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/63Post Release 1.0.02021-11-21T14:47:01+01:00Lubomir DoležalPost Release 1.0.0- [x] send 3 metadata files to IDP prod
- [x] configure SFTP + ingestor on production - first attempt on stale https://gitlab.eox.at/esa/prism/vs/-/tree/prod-testing
- [ ] operator guide registrar modularized
- [x] user guide deploy new...- [x] send 3 metadata files to IDP prod
- [x] configure SFTP + ingestor on production - first attempt on stale https://gitlab.eox.at/esa/prism/vs/-/tree/prod-testing
- [ ] operator guide registrar modularized
- [x] user guide deploy new client image
- [x] deploy the whole thingie to prod VMs + check Offnadir angle if filter works (client now works, but core opensearch seems not to be applied) @bucekl -> does not work, moved to #123
- [x] add Authentication flow to TN @bucekl
- [x] add ingestion input flow to TN - either make a test or just document how that part is done (for filedaemon branch https://gitlab.eox.at/esa/prism/vs/-/tree/prod-testing was done, but not finished - see commits there @abdallam
- [x] add reporting output flow to TN - referencing the test done via CI possibly @abdallam
- [x] match requirements from requirements matrix as done in Software release note @fabian.schindler
- [ ] Operators guide preprocessor configuration creation - at least briefly, there is already a config schema in preprocessor, reuse that
- [x] Registry is now open, remove the docker login partsLubomir DoležalLubomir Doležalhttps://gitlab.eox.at/esa/prism/vs/-/issues/621.0.0 release2020-11-12T21:41:07+01:00Lubomir Doležal1.0.0 release- [x] close #20 - Fabian
- [x] fix registrar test to make build pass and merge registrar-modularization branch - Fabian
- [x] update WO TN v3.0.0 - SM
- [x] prepare SRN - MA
- [x] update training material pres v1.1.0 - LB
- [x] create gi...- [x] close #20 - Fabian
- [x] fix registrar test to make build pass and merge registrar-modularization branch - Fabian
- [x] update WO TN v3.0.0 - SM
- [x] prepare SRN - MA
- [x] update training material pres v1.1.0 - LB
- [x] create git tag (bumpversion) - LB
- [x] git archive at 1.0.0 - LB
- [x] build pdf for operator guide - new version v1.1.0 - LB
- [x] build pdf for user guide - new version v1.1.1 - LBBaseline evolution v2https://gitlab.eox.at/esa/prism/vs/-/issues/61Rethink pvs-starter vs. helm templates2021-10-14T09:42:06+02:00Stephan Meißlstephan.meissl@eox.atRethink pvs-starter vs. helm templatesNikola JankovicNikola Jankovichttps://gitlab.eox.at/esa/prism/vs/-/issues/60Define version release procedure2020-11-11T11:42:11+01:00Lubomir DoležalDefine version release procedure- an official release procedure of vs components should be defined and implemented
- currently each Dockerfile has got a version, which is used by gitlab ci to tag images in the registry
- on production servers :latest tag is pulled
An ...- an official release procedure of vs components should be defined and implemented
- currently each Dockerfile has got a version, which is used by gitlab ci to tag images in the registry
- on production servers :latest tag is pulled
An option would be https://github.com/c4urself/bump2version for manually updating VERSION in Dockerfiles. In order to avoid having different "commits" as one version, gitlab.ci would push a tagged version only if the tag does not exist yet (as eoxserver travis.ci does it?)
Pros:
- we have control over releases manually
Cons:
- someone has to do itBaseline evolution v2Lubomir DoležalLubomir Doležal