From 7dd49743e1aea3b4d847aa30b10ee32d6573c61a Mon Sep 17 00:00:00 2001
From: Fabian Schindler <fabian.schindler.strauss@gmail.com>
Date: Fri, 5 Jun 2020 11:01:18 +0200
Subject: [PATCH] Improved on docs Started with initialization and setup
---
documentation/operator-guide/index.rst | 3 +
.../operator-guide/initialization.rst | 277 ++++++++++++++++++
documentation/operator-guide/intro.md | 5 -
documentation/operator-guide/intro.rst | 69 +++++
documentation/operator-guide/setup.rst | 74 +++++
5 files changed, 423 insertions(+), 5 deletions(-)
create mode 100644 documentation/operator-guide/initialization.rst
delete mode 100644 documentation/operator-guide/intro.md
create mode 100644 documentation/operator-guide/intro.rst
create mode 100644 documentation/operator-guide/setup.rst
diff --git a/documentation/operator-guide/index.rst b/documentation/operator-guide/index.rst
index de28ce82..8060b25b 100644
--- a/documentation/operator-guide/index.rst
+++ b/documentation/operator-guide/index.rst
@@ -7,6 +7,9 @@ View Server - Operator Guide
:maxdepth: 3
intro
+ initialization
+ setup
+
operator-guide
diff --git a/documentation/operator-guide/initialization.rst b/documentation/operator-guide/initialization.rst
new file mode 100644
index 00000000..9477411b
--- /dev/null
+++ b/documentation/operator-guide/initialization.rst
@@ -0,0 +1,277 @@
+.. _initialization:
+
+Initialization
+==============
+
+In order to set up an instance of VS, the ``pvs_starter`` utility is
+recommended. It is distributed as a Python package, easily installed via
+``pip``.
+
+.. code-block:: bash
+
+ pip3 install pvs_starter # TODO: git url
+
+
+Now the VS instance can be set up like this:
+
+.. code-block:: bash
+
+ python3 -m pvs_starter.cli config.yaml out/ -f
+
+This takes the initialization configuration ``config.yaml`` to generate
+the structure in the ``out/`` directory.
+
+
+Configuration
+-------------
+
+The important part of the initialization is of course the configuration. The
+format is structured in YAML and will be detailed here. It contains the
+following sections:
+
+``database``
+~~~~~~~~~~~~
+
+Here, there access credentials of the database are stored. It defines the
+internal database name, user and password that will be created when the stack
+is deployed. Note that there is no ``host`` setting, as this will be handled
+automatically.
+
+.. code-block:: yaml
+
+ database:
+ name: vs_db
+ user: vs_user
+ password: Go-J_eOUvj2k
+
+``django_admin``
+~~~~~~~~~~~~~~~~
+
+This section deals with the setup of a Django admin account. This is used to
+later access the admin panel to inspect the registered data.
+
+.. code-block:: yaml
+
+ django_admin:
+ user: admin
+ mail: office@eox.at
+ password: jvLwv_20x-69
+
+``preprocessor``
+~~~~~~~~~~~~~~~~
+
+Here, the preprocessing can be configured in detail.
+
+TODO
+
+
+``products``
+~~~~~~~~~~~~
+
+This section defines product type related information. The two most important
+settings here are the ``type_extractor`` and ``level_extractor`` structures
+which specify how the product type and product level will be extracted from
+the metadata. For this, an XPath (or multiple) can be specified to retrieve
+that information.
+
+The ``types`` section defines the available product types and what browse
+and mask types are to be generated.
+
+.. code-block:: yaml
+
+ products:
+ type_extractor:
+ xpath:
+ namespace_map:
+ level_extractor:
+ xpath:
+ namespace_map:
+ types:
+ PL00:
+ default_browse: TRUE_COLOR
+ browses:
+ TRUE_COLOR:
+ red:
+ expression: red
+ range: [1000, 15000]
+ nodata: 0
+ green:
+ expression: green
+ range: [1000, 15000]
+ nodata: 0
+ blue:
+ expression: blue
+ range: [1000, 15000]
+ nodata: 0
+ FALSE_COLOR:
+ red:
+ expression: nir
+ range: [1000, 15000]
+ nodata: 0
+ green:
+ expression: red
+ range: [1000, 15000]
+ nodata: 0
+ blue:
+ expression: green
+ range: [1000, 15000]
+ nodata: 0
+ NDVI:
+ grey:
+ expression: (nir-red)/(nir+red)
+ range: [-1, 1]
+ masks:
+ validity:
+ validity: true
+
+
+
+``collections``
+~~~~~~~~~~~~~~~
+
+In the ``collections`` section, the collections are set up and it is defined
+which products of what type and level will be inserted into them. The
+``product_types`` must list types defined in the ``products`` section.
+
+.. code-block:: yaml
+
+ collections:
+ COLLECTION:
+ product_types:
+ - PL00
+ product_levels:
+ - Level_1
+ - Level_3
+
+``storages``
+~~~~~~~~~~~~
+
+Here, the three relevant storages can be configured: the ``source``,
+``preprocessed`` and ``cache`` storages.
+
+The source storage defines the location from which the original files will be
+pulled to be preprocessed. Preprocessed images and metadata will then be
+pushed to the ``preprocessed`` storage. The cache service will cache images on
+the ``cache`` storage.
+
+Each storage definition uses the same structure and can target various types
+of storages, such as OpenStack swift.
+
+These storage definitions will be used in the appropriate sections.
+
+
+
+TODO: improve example
+
+.. code-block:: yaml
+
+ storages:
+ source:
+ auth_type: keystone
+ auth_url:
+ version: 3
+ username:
+ password:
+ tenant_name:
+ tenant_id:
+ region_name:
+ container:
+ preprocessed:
+ auth_type: keystone
+ auth_url:
+ version: 3
+ username:
+ password:
+ tenant_name:
+ tenant_id:
+ region_name:
+ container:
+ cache:
+ type: swift
+ auth_type: keystone
+ auth_url: https://auth.cloud.ovh.net/v3/
+ auth_url_short: https://auth.cloud.ovh.net/
+ version: 3
+ username:
+ password:
+ tenant_name:
+ tenant_id:
+ region_name:
+ container:
+
+
+``cache``
+~~~~~~~~~
+
+This section defines the exposed services layers of the cache, and how the
+internal layers shall be cached.
+
+.. code-block:: yaml
+
+ cache:
+ metadata:
+ title: PRISM Data Access Service (PASS) developed by EOX
+ abstract: PRISM Data Access Service (PASS) developed by EOX
+ url: https://vhr18.pvs.prism.eox.at/cache/ows
+ keyword: view service
+ accessconstraints: UNKNOWN
+ fees: UNKNOWN
+ contactname: Stephan Meissl
+ contactphone: Please contact via mail.
+ contactfacsimile: None
+ contactorganization: EOX IT Services GmbH
+ contactcity: Vienna
+ contactstateorprovince: Vienna
+ contactpostcode: 1090
+ contactcountry: Austria
+ contactelectronicmailaddress: office@eox.at
+ contactposition: CTO
+ providername: EOX
+ providerurl: https://eox.at
+ inspire_profile: true
+ inspire_metadataurl: TBD
+ defaultlanguage: eng
+ language: eng
+ services:
+ wms:
+ enabled: true
+ wmts:
+ enabled: true
+ connection_timeout: 10
+ timeout: 120
+ expires: 3600
+ key: /{tileset}/{grid}/{dim}/{z}/{x}/{y}.{ext}
+ tilesets:
+ VHR_IMAGE_2018__TRUE_COLOR:
+ title: VHR Image 2018 True Color
+ abstract: VHR Image 2018 True Color
+ VHR_IMAGE_2018__FALSE_COLOR:
+ title: VHR Image 2018 False Color
+ abstract: VHR Image 2018 False Color
+ VHR_IMAGE_2018__NDVI:
+ title: VHR Image 2018 NDVI
+ abstract: VHR Image 2018 NDVI
+ style: earth
+ VHR_IMAGE_2018_Level_1__TRUE_COLOR:
+ title: VHR Image 2018 Level 1 True Color
+ abstract: VHR Image 2018 Level 1 True Color
+ VHR_IMAGE_2018_Level_1__FALSE_COLOR:
+ title: VHR Image 2018 Level 1 False Color
+ abstract: VHR Image 2018 Level 1 False Color
+ VHR_IMAGE_2018_Level_1__NDVI:
+ title: VHR Image 2018 Level 1 NDVI
+ abstract: VHR Image 2018 Level 1 NDVI
+ style: earth
+ VHR_IMAGE_2018_Level_1__TRUE_COLOR:
+ title: VHR Image 2018 Level 3 True Color
+ abstract: VHR Image 2018 Level 3 True Color
+ VHR_IMAGE_2018_Level_1__FALSE_COLOR:
+ title: VHR Image 2018 Level 3 False Color
+ abstract: VHR Image 2018 Level 3 False Color
+ VHR_IMAGE_2018_Level_1__NDVI:
+ title: VHR Image 2018 Level 3 NDVI
+ abstract: VHR Image 2018 Level 3 NDVI
+ style: earth
+ # grids? cache options?
+
+
diff --git a/documentation/operator-guide/intro.md b/documentation/operator-guide/intro.md
deleted file mode 100644
index 7ff54f10..00000000
--- a/documentation/operator-guide/intro.md
+++ /dev/null
@@ -1,5 +0,0 @@
-# Introduction
-
-operator
-
-TODO
diff --git a/documentation/operator-guide/intro.rst b/documentation/operator-guide/intro.rst
new file mode 100644
index 00000000..41a11281
--- /dev/null
+++ b/documentation/operator-guide/intro.rst
@@ -0,0 +1,69 @@
+.. _introduction:
+
+Introduction
+============
+
+This guide will detail the operation of a View Server and all of its
+components.
+
+Since the View Server is a Docker based software and all of its components are
+distributed and executed in context of Docker images and containers, a
+knowledge of Docker and Docker Swarm is required.
+
+## Components
+
+The View Server consists of the following service components (with their
+respective Docker image in parenthesis):
+
+ - Ingestor (ingestor)
+ - Preprocessor (preprocessor)
+ - Registrator (core)
+ - Renderer (core)
+ - Cache (cache)
+ - Seeder (cache)
+ - Client (client)
+ - Database (postgis)
+ - Redis (redis)
+ - Reverse Proxy (traefik)
+
+These services are bundled and managed together in a Docker Swarm via
+Docker Compose configuration files.
+
+Images
+------
+
+The software is distributed as Docker images, which can be instantiated
+and run in their intended role. Some images are hosted on the docker hub,
+the official and default repository for Docker images. Other images reside
+on the gitlab.eox.at repository. Images from the official repository
+are only identified via their name, whereas images from the eox repository
+conventionally use the full URL, including the domain name.
+
+ - mdillon/postgis:10
+ - redis
+ - traefik:2.1
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_core:latest
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_cache:latest
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor:latest
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_client:latest
+
+Configuration Files
+-------------------
+
+The following configuration files will impact the behavior of the View Server:
+
+ - index.html: This is the main file to configure the client. In this file,
+ the viewing layer, search and download enpoints are configured. Usually
+ this is associated with additional backdrop and overlay layers.
+ - preprocessor.yml: This file configures the preprocessing process.
+ - mapcache.xml: This file defines the input sources of the cache and its
+ published layers.
+ - init-db.sh: This file sets up the registrator and renderer side of the VS,
+
+Setup
+-----
+
+In order to help with the initial setup of a VS, the ``pvs_starter`` package
+allows to quickly establish the required structure of configuration files.
+
+It is run from a configuration file and sets up
diff --git a/documentation/operator-guide/setup.rst b/documentation/operator-guide/setup.rst
new file mode 100644
index 00000000..2517af90
--- /dev/null
+++ b/documentation/operator-guide/setup.rst
@@ -0,0 +1,74 @@
+.. _setup:
+
+Setup
+=====
+
+In this chapter the setup of a new VS stack is detailed. Before this step can
+be done, the configuration and environment files need to be present. These
+files can be added manually, or be created in the initialization step.
+
+
+Docker
+------
+
+In order to deploy the Docker Swarm stack to the target machine, Docker and
+its facilities need to be installed. This step depends on the systems
+architecture. On a Debian based system this may look like this:
+
+.. code-block:: bash
+
+ sudo apt-get install \
+ apt-transport-https \
+ ca-certificates \
+ curl \
+ gnupg-agent \
+ software-properties-common
+
+ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
+
+ # add the apt repository
+ sudo add-apt-repository \
+ "deb [arch=amd64] https://download.docker.com/linux/debian \
+ $(lsb_release -cs) \
+ stable"
+
+ # fetch the package index and install Docker
+ sudo apt-get update
+ sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+## TODO: RHEL?
+
+Docker Swarm
+------------
+
+Now that Docker is installed, the machine can either create a new swarm or join
+an existing one.
+
+To create a new Swarm, the following command can be used:
+
+.. code-block:: bash
+
+ docker swarm init
+
+
+Alternatively, an existing Swarm can be joined. The easiest way to do this, is
+to obtain a ``join-token``. On an existing Swarm manager (where a Swarm was
+initialized or already joined as manager) run this command:
+
+
+.. code-block:: bash
+
+ docker swarm join-token worker
+
+
+This prints out a command that can be run on a machine to join the swarm:
+
+.. code-block:: bash
+
+ docker swarm join --token <obtained token>
+
+
+Additional information for swarm management can be obtained in the official
+documentation of the project:
+https://docs.docker.com/engine/reference/commandline/swarm/
+
--
GitLab