EOX GitLab Instance

Commit 89dc1633 authored by Nikola Jankovic's avatar Nikola Jankovic 💻
Browse files

added chart

added dependencies
added default values
updated readme
parent 07b5bd40
.vscode/
stages:
- chart
# Helm chart publishing
publish-helm-chart:
image: devth/helm:v3.6.3
stage: chart
script:
# push Helm chart
- "upload_filename=$(helm package . | sed 's/^Successfully packaged chart and saved it to: \\(.*\\)/\\1/')"
- curl -u $HELM_CHART_REPOSITORY_CREDENTIALS -v -X POST https://charts-public.hub.eox.at/api/charts --data-binary "@${upload_filename}"
only:
- main
dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 9.7.2
- name: redis
repository: https://charts.bitnami.com/bitnami
version: 10.9.0
- name: vs-cache
repository: https://charts-public.hub.eox.at
version: 0.1.0
- name: vs-client
repository: https://charts-public.hub.eox.at
version: 0.1.0
- name: vs-renderer
repository: https://charts-public.hub.eox.at
version: 0.1.0
- name: vs-registrar
repository: https://charts-public.hub.eox.at
version: 0.1.0
- name: vs-scheduler
repository: https://charts-public.hub.eox.at
version: 0.1.0
- name: vs-harvester
repository: https://charts-public.hub.eox.at
version: 0.1.0
- name: vs-ingestor
repository: https://charts-public.hub.eox.at
version: 0.1.0
- name: vs-preprocessor
repository: https://charts-public.hub.eox.at
version: 0.1.0
digest: sha256:ba0ab468622278805e3b7eef4d99db4703d7aae1b99349abb8c56ea86475af41
generated: "2021-10-27T13:06:12.934724946+02:00"
apiVersion: v2
name: vs
description: A Helm chart for a Kubernetes deployment of the View Server (VS)
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 2.0.0
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: 2.0.0
maintainers:
- name: EOX IT Services GmbH
url: https://eox.at
dependencies:
- name: "postgresql"
version: "9.7.2"
repository: "https://charts.bitnami.com/bitnami"
alias: database
- name: "redis"
version: "10.9.0"
repository: "https://charts.bitnami.com/bitnami"
alias: redis
- name: vs-cache
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: cache
- name: vs-client
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: client
- name: vs-renderer
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: renderer
- name: vs-registrar
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: registrar
- name: vs-scheduler
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: scheduler
- name: vs-harvester
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: harvester
- name: vs-ingestor
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: ingestor
- name: vs-preprocessor
version: "0.1.0"
repository: "https://charts-public.hub.eox.at"
alias: preprocessor
# helm-charts
# Chart for the View Server (VS) bundling all services
Contains helm charts for the view server
\ No newline at end of file
All the commands must be ran from the repo folder. Must have `kubectl` and `helm` utilities and one of `k3s` or `minikube` clusters.
Useful commands:
```bash
helm dependency update # updates the dependencies
helm template testing . --output-dir ../tmp/ -f values.custom.yaml # renders the helm template files. Used in conjunction with vs-starter
```
The template command outputs the rendered yaml files for debugging and checking.
## Installing chart locally
In order to test the services together, here's a way to install the chart locally in your `k3s`/`minicube`. This install is based on the default values in the repo and customizes them.
### Prerequisites
When running k3s and minikube in docker, the paths doesn't refer to your machine, but the docker container where k3s runs. The following display how to setup each solution.
#### Minikube
Minikube needs to be started with the following:
```bash
minikube start --mount-string /home/$user/:/minikube-host/
minikube addons enable ingress
```
Here the full home directory is bound to the cluster at `/minikube-host` in order to bind mounts necessary for development. Also the ingress addon is enabled.
#### k3s
k3s is also started creating with specifying a volume
``` bash
k3s cluster create --volume /home/$user/:/k3s-host/
```
Here the full home directory is bound to the cluster at `/k3s-host` in order to bind mounts necessary for development.
#### values.yaml
The default values.yaml should be enough for a very basic setup, but overriding with a separate one is required for more elaborate setups and deployments.
#### Persistent volume claims
For the services to start, you're going to need the pvcs as below. Create a file `pvc.yaml` and run `kubectl apply -f pvc.yaml`
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-access-db
namespace: <your-namespace>
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path # this should be `standard` for minikube
resources:
requests:
storage: 2Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-access-redis
namespace: <your-namespace>
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path # this should be `standard` for minikube
resources:
requests:
storage: 2Gi
```
### Deploying the stack
Install:
```bash
helm install test . --values values.custom.yaml
```
Upgrade with overrides values:
```bash
helm upgrade test . --values values.custom.yaml --values your-values-override-file.yaml
```
If you specify multiple values files, helm will merge them together. [You can even delete keys by setting them to `null` in the override file a](https://helm.sh/docs/chart_template_guide/values_files/) .
A useful override is to set the ingress host e.g. to nip.io:
```yaml
global:
ingress:
hosts:
- host: data-access.<IP-ADDRESS>.nip.io
tls:
- hosts:
- data-access.<IP-ADDRESS>.nip.io
```
Where `<IP-ADDRESS>` is one of
* 127.0.0.1
* Output of `minikube ip`
You might also want to change the number of replicas:
```yaml
renderer:
replicaCount: 1
```
For development, it's useful to mount the code as volume. This can be done via a values override.
When using k3s & minikube, it's enough to define a volume like this:
```yaml
registrar:
volumes:
- name: eoxserver
hostPath:
path: /<app>-host/path/to/eoxserver
type: DirectoryOrCreate
volumeMounts:
- mountPath: /usr/local/lib/python3.8/dist-packages/eoxserver/
name: eoxserver
```
where `app` is `k3s` or `minikube` and a `volumeMount` for the container:
{{/*
Expand the name of the chart.
*/}}
{{- define "vs.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "vs.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "vs.selectorLabels" -}}
app.kubernetes.io/name: {{ include "vs.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "vs.labels" -}}
helm.sh/chart: {{ include "vs.chart" . }}
{{ include "vs.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
# environment variables which are needed for the services
global:
env:
DEBUG: "false"
COLLECTION: COLLECTION
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: .TIF,.tif,.xml
GDAL_DISABLE_READDIR_ON_OPEN: "TRUE"
COLLECT_STATIC: "false"
UPLOAD_CONTAINER: upload-container
GDAL_PAM_ENABLED: "NO"
#### SHOULD BE CHANGED IN PROD ####
DB_NAME: dbname
DB_PORT: "5432"
DB_PW: dbpw
DB_USER: dbuser
POSTGRES_DB: dbname
POSTGRES_PASSWORD: dbpw
POSTGRES_USER: dbuser
DJANGO_MAIL: office@eox.at
DJANGO_PASSWORD: djangopw
DJANGO_USER: djangouser
REDIS_PORT: "6379"
REDIS_PREPROCESS_QUEUE_KEY: preprocess_queue
REDIS_PREPROCESS_MD_QUEUE_KEY: preprocess-md_queue
REDIS_PREPROCESS_FAILURE_KEY: preprocess-failure_set
REDIS_PREPROCESS_PROGRESS_KEY: preprocessing_set
REDIS_PREPROCESS_SUCCESS_KEY: preprocess-success_set
REDIS_REGISTER_QUEUE_KEY: register_queue
REDIS_REGISTER_FAILURE_KEY: register-failure_set
REDIS_REGISTER_PROGRESS_KEY: registering_set
REDIS_REGISTER_SUCCESS_KEY: register-success_set
REDIS_SEED_QUEUE_KEY: seed_queue
REDIS_QUEUE_KEY: seed_queue
REDIS_REGISTERED_SET_KEY: registered_set
REDIS_SET_KEY: registered_set
REDIS_HARVESTER_QUEUE_KEY: harvester_queue
storage:
# storage parameters
# - data is a mapping for name:config for the registrar and preprocessor services
# - source is the data source of the data for the preprocessor
# - target is a target destination for the preprocessor - if not existing it will be the same as data
# - cache is the target for the cache, by default its local for development
data: {}
# examples:
# swift-data: the key is the storage name
# type: swift
# username: "username"
# password: "password"
# tenant_name: "tenant_name"
# tenant_id: "tenant_id"
# region_name: "region_name"
# auth_url: "auth_url"
# auth_url_short: "auth_url_short"
# auth_version: "auth_version"
# user_domain_name: "user_domain_name"
# s3-data:
# type: S3
# bucket: "bucket"
# endpoint_url: "endpoint_url"
# access_key_id: "access_key_id"
# secret_access_key: "secret_access_key"
# region: "region"
source:
{}
# example:
# type: swift
# username: "username"
# password: "password"
# tenant_name: "tenant_name"
# tenant_id: "tenant_id"
# region_name: "region_name"
# auth_url: "auth_url"
# auth_url_short: "auth_url_short"
# auth_version: "auth_version"
# user_domain_name: "user_domain_name"
target:
{}
# can be ommited if storage.data == data.target
cache:
type: local
# type: S3 or local for testing
# bucket: "bucket"
# endpoint_url: "endpoint_url"
# access_key_id: "access_key_id"
# secret_access_key: "secret_access_key"
# region: "region"
collections:
{}
# example
# VHR_IMAGE_2018:
# product_types:
# - PL00
# coverage_types:
# - RGBNir
productTypes:
[]
# example
# - name: SPOT
# defaultBrowse: TRUE_COLOR
# coverages:
# RGBNir:
# assets:
# - preprocessed_image_1
# Pan:
# assets:
# - preprocessed_image_2
# browses:
# TRUE_COLOR:
# red:
# expression: red
# range:
# - 100
# - 1500
# nodata: 0
# green:
# expression: green
# range:
# - 25
# - 1500
# nodata: 0
# blue:
# expression: blue
# range:
# - 75
# - 1500
# nodata: 0
# TRUE_COLOR_PANSHARPENED:
# red:
# expression: pansharpen(pan, red, green, blue)[0]
# range:
# - 100
# - 1500
# nodata: 0
# green:
# expression: pansharpen(pan, red, green, blue)[1]
# range:
# - 25
# - 1500
# nodata: 0
# blue:
# expression: pansharpen(pan, red, green, blue)[2]
# range:
# - 75
# - 1500
# nodata: 0
# FALSE_COLOR:
# red:
# expression: nir
# range:
# - 500
# - 5000
# nodata: 0
# green:
# expression: blue
# range:
# - 50
# - 1700
# nodata: 0
# blue:
# expression: green
# range:
# - 50
# - 1700
# nodata: 0
# NDVI:
# grey:
# expression: (nir-red)/(nir+red)
# range:
# - -1
# - 1
# nodata: -9999
# collections:
# - DS_SPOT6
# masks:
defaultLayer:
layers:
[]
# example
# - id: DS_SPOT6
# title: DS_SPOT6 True Color
# abstract: DS_SPOT6 True Color
# displayColor: "#eb3700"
# grids:
# - name: GoogleMapsCompatible
# zoom: 13
# - name: WGS84
# zoom: 12
# parentLayer: DS_SPOT6
# - id: DS_SPOT6__TRUE_COLOR
# title: DS_SPOT6 True Color
# abstract: DS_SPOT6 True Color
# grids:
# - name: GoogleMapsCompatible
# zoom: 14
# - name: WGS84
# zoom: 13
# parentLayer: DS_SPOT6
# - id: DS_SPOT6__TRUE_COLOR_PANSHARPENED
# title: DS_SPOT6 True Color Pansharpened
# abstract: DS_SPOT6 True Color Pansharpened
# grids:
# - name: GoogleMapsCompatible
# zoom: 17
# - name: WGS84
# zoom: 16
# parentLayer: DS_SPOT6
# - id: DS_SPOT6__FALSE_COLOR
# title: DS_SPOT6 False Color
# abstract: DS_SPOT6 False Color
# grids:
# - name: GoogleMapsCompatible
# zoom: 14
# - name: WGS84
# zoom: 13
# parentLayer: DS_SPOT6
# - id: DS_SPOT6__NDVI
# title: DS_SPOT6 NDVI
# abstract: DS_SPOT6 NDVI
# grids:
# - name: GoogleMapsCompatible
# zoom: 14
# - name: WGS84
# zoom: 13
# parentLayer: DS_SPOT6
overlayLayers:
[]
# example
# - id: test
# title: test
# displayColor: "#eb3700"
coverageTypes:
[]
# example
# - bands:
# - definition: 'http://www.opengis.net/def/property/OGC/0/Radiance'
# description: B01
# identifier: B01
# name: coastal
# nil_values:
# - reason: 'http://www.opengis.net/def/nil/OGC/0/unknown'
# value: 0
# uom: W/m2/um
# wavelength: 442.7
# data_type: Uint16
# name: S2L1C_B01
metadata:
{}
# any metadata relevant for services
database:
persistence:
enabled: false
existingClaim: data-access-db # rename to persistence volume claim
#### SHOULD BE CHANGED IN PROD ####
postgresqlUsername: dbuser
postgresqlPassword: dbpw
postgresqlDatabase: dbname
postgresqlPostgresPassword: dbpgpw
initdbScripts:
enablePostgis.sh: |
echo "Enabling postgis"
PGPASSWORD="${POSTGRES_POSTGRES_PASSWORD}" psql -U postgres -d "${POSTGRES_DB}" -c "CREATE EXTENSION postgis;"
replicaCount: 1
resources:
limits:
cpu: "1.5"
memory: 6Gi
requests:
cpu: "0.1"
memory: 0.5Gi
affinity: {}
redis:
usePassword: false
persistence:
existingClaim: data-access-redis # rename to persistent volume claim
master:
persistence:
enabled: true
storageClass: managed-nfs-storage
cluster:
enabled: false
client: {}
cache: {}
renderer: {}
registrar: {}
harvester: {}
scheduler: {}
preprocessor: {}
ingestor: {}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment