EOX GitLab Instance

Optional database synchronization with values and include RasterStyle

Lubomir Doležal requested to merge db-sync into main

Previous way of configuration of VS models was: rendered template of init_db.sh script creating all EOxServer database models based on configuration being run on first start of the system. Next startups of registrar and renderer containers check for a first collection to exist and if yes, stops the script.

New way: Python synchronization script utilizing EOxServer CLI or direct model creation. The dbsync script is run as a one-off Kubernetes Job via a post-install hook via helm. Postupgrade is not configured due to fact that Jobs are immutable in k8s and same name can not be reused. In order to invoke the job again, helm uninstall and helm install need to be done. Renderer and registrar do not initialize the db anymore, but the script does. The job uses core image and the script is run via a static init_db.sh in order to utilize the other parts of core script handling like configure.sh to setup the instance and wait-for-it to wait for the database. The whole db synchronization is done inside a single transaction so if one single operation throws an exception, whole sync script is rolled back and the operation needs to be unblocked by the operator. This is done to not leave the DB in inconsistent state (compared to config or compared to original state) after the operation.

There are 3 new default values in config:

    create: true
    replace: false
    delete: false

There are therefore 3 functionalities of the synchronization script:

  • create: true iterates through the config and create attempt to create new models but will not do any modifications to existing models
  • replace: true iterates through the config and when creating new models also additionally updates existing models to match the config. No deletion of existing but unmatched models against config is done though.
  • delete: true iterates through the existing models and those that do not have a counterpart in config are attempted to be deleted. However if for example at the moment when it would be deleted a Collection has Products, the deletion fails and the whole db transaction reverted.

Please note that there will be a short period (few seconds) when renderer and registrar are already starting but db is not synced yet, but this should not be an issue (except for when registrar Redis queue is not-empty at the moment of helm install).

Depends on a merge of https://github.com/EOxServer/eoxserver/pull/566 and a new version release of EOxServer

Docker swarm integration as a replicated-job (depends on Docker version at least 23.0!) is implemented in https://gitlab.eox.at/vs/vs-starter/-/tree/db_sync

Also includes changes from https://gitlab.eox.at/vs/vs/-/tree/raster-styles

Edited by Lubomir Doležal

Merge request reports