Improved on docs

Started with initialization and setup

documentation/operator-guide/index.rst

@@ -7,6 +7,9 @@ View Server - Operator Guide
    :maxdepth: 3
 
    intro
+   initialization
+   setup
+
 
    operator-guide
 

documentation/operator-guide/initialization.rst

+.. _initialization:
+
+Initialization
+==============
+
+In order to set up an instance of VS, the ``pvs_starter`` utility is
+recommended. It is distributed as a Python package, easily installed via
+``pip``.
+
+.. code-block:: bash
+
+    pip3 install pvs_starter # TODO: git url
+
+
+Now the VS instance can be set up like this:
+
+.. code-block:: bash
+
+    python3 -m pvs_starter.cli config.yaml out/ -f
+
+This takes the initialization configuration ``config.yaml`` to generate
+the structure in the ``out/`` directory.
+
+
+Configuration
+-------------
+
+The important part of the initialization is of course the configuration. The
+format is structured in YAML and will be detailed here. It contains the
+following sections:
+
+``database``
+~~~~~~~~~~~~
+
+Here, there access credentials of the database are stored. It defines the
+internal database name, user and password that will be created when the stack
+is deployed. Note that there is no ``host`` setting, as this will be handled
+automatically.
+
+.. code-block:: yaml
+
+    database:
+      name: vs_db
+      user: vs_user
+      password: Go-J_eOUvj2k
+
+``django_admin``
+~~~~~~~~~~~~~~~~
+
+This section deals with the setup of a Django admin account. This is used to
+later access the admin panel to inspect the registered data.
+
+.. code-block:: yaml
+
+    django_admin:
+      user: admin
+      mail: office@eox.at
+      password: jvLwv_20x-69
+
+``preprocessor``
+~~~~~~~~~~~~~~~~
+
+Here, the preprocessing can be configured in detail.
+
+TODO
+
+
+``products``
+~~~~~~~~~~~~
+
+This section defines product type related information. The two most important
+settings here are the ``type_extractor`` and ``level_extractor`` structures
+which specify how the product type and product level will be extracted from
+the metadata. For this, an XPath (or multiple) can be specified to retrieve
+that information.
+
+The ``types`` section defines the available product types and what browse
+and mask types are to be generated.
+
+.. code-block:: yaml
+
+    products:
+      type_extractor:
+        xpath:
+        namespace_map:
+      level_extractor:
+        xpath:
+        namespace_map:
+      types:
+        PL00:
+          default_browse: TRUE_COLOR
+          browses:
+            TRUE_COLOR:
+              red:
+                expression: red
+                range: [1000, 15000]
+                nodata: 0
+              green:
+                expression: green
+                range: [1000, 15000]
+                nodata: 0
+              blue:
+                expression: blue
+                range: [1000, 15000]
+                nodata: 0
+            FALSE_COLOR:
+              red:
+                expression: nir
+                range: [1000, 15000]
+                nodata: 0
+              green:
+                expression: red
+                range: [1000, 15000]
+                nodata: 0
+              blue:
+                expression: green
+                range: [1000, 15000]
+                nodata: 0
+            NDVI:
+              grey:
+                expression: (nir-red)/(nir+red)
+                range: [-1, 1]
+          masks:
+            validity:
+              validity: true
+
+
+
+``collections``
+~~~~~~~~~~~~~~~
+
+In the ``collections`` section, the collections are set up and it is defined
+which products of what type and level will be inserted into them. The
+``product_types`` must list types defined in the ``products`` section.
+
+.. code-block:: yaml
+
+    collections:
+      COLLECTION:
+        product_types:
+          - PL00
+        product_levels:
+          - Level_1
+          - Level_3
+
+``storages``
+~~~~~~~~~~~~
+
+Here, the three relevant storages can be configured: the ``source``,
+``preprocessed`` and ``cache`` storages.
+
+The source storage defines the location from which the original files will be
+pulled to be preprocessed. Preprocessed images and metadata will then be
+pushed to the ``preprocessed`` storage. The cache service will cache images on
+the ``cache`` storage.
+
+Each storage definition uses the same structure and can target various types
+of storages, such as OpenStack swift.
+
+These storage definitions will be used in the appropriate sections.
+
+
+
+TODO: improve example
+
+.. code-block:: yaml
+
+    storages:
+      source:
+        auth_type: keystone
+        auth_url:
+        version: 3
+        username:
+        password:
+        tenant_name:
+        tenant_id:
+        region_name:
+        container:
+      preprocessed:
+        auth_type: keystone
+        auth_url:
+        version: 3
+        username:
+        password:
+        tenant_name:
+        tenant_id:
+        region_name:
+        container:
+      cache:
+        type: swift
+        auth_type: keystone
+        auth_url: https://auth.cloud.ovh.net/v3/
+        auth_url_short: https://auth.cloud.ovh.net/
+        version: 3
+        username:
+        password:
+        tenant_name:
+        tenant_id:
+        region_name:
+        container:
+
+
+``cache``
+~~~~~~~~~
+
+This section defines the exposed services layers of the cache, and how the
+internal layers shall be cached.
+
+.. code-block:: yaml
+
+    cache:
+      metadata:
+        title: PRISM Data Access Service (PASS) developed by EOX
+        abstract: PRISM Data Access Service (PASS) developed by EOX
+        url: https://vhr18.pvs.prism.eox.at/cache/ows
+        keyword: view service
+        accessconstraints: UNKNOWN
+        fees: UNKNOWN
+        contactname: Stephan Meissl
+        contactphone: Please contact via mail.
+        contactfacsimile: None
+        contactorganization: EOX IT Services GmbH
+        contactcity: Vienna
+        contactstateorprovince: Vienna
+        contactpostcode: 1090
+        contactcountry: Austria
+        contactelectronicmailaddress: office@eox.at
+        contactposition: CTO
+        providername: EOX
+        providerurl: https://eox.at
+        inspire_profile: true
+        inspire_metadataurl: TBD
+        defaultlanguage: eng
+        language: eng
+      services:
+        wms:
+          enabled: true
+        wmts:
+          enabled: true
+      connection_timeout: 10
+      timeout: 120
+      expires: 3600
+      key: /{tileset}/{grid}/{dim}/{z}/{x}/{y}.{ext}
+      tilesets:
+        VHR_IMAGE_2018__TRUE_COLOR:
+          title: VHR Image 2018 True Color
+          abstract: VHR Image 2018 True Color
+        VHR_IMAGE_2018__FALSE_COLOR:
+          title: VHR Image 2018 False Color
+          abstract: VHR Image 2018 False Color
+        VHR_IMAGE_2018__NDVI:
+          title: VHR Image 2018 NDVI
+          abstract: VHR Image 2018 NDVI
+          style: earth
+        VHR_IMAGE_2018_Level_1__TRUE_COLOR:
+          title: VHR Image 2018 Level 1 True Color
+          abstract: VHR Image 2018 Level 1 True Color
+        VHR_IMAGE_2018_Level_1__FALSE_COLOR:
+          title: VHR Image 2018 Level 1 False Color
+          abstract: VHR Image 2018 Level 1 False Color
+        VHR_IMAGE_2018_Level_1__NDVI:
+          title: VHR Image 2018 Level 1 NDVI
+          abstract: VHR Image 2018 Level 1 NDVI
+          style: earth
+        VHR_IMAGE_2018_Level_1__TRUE_COLOR:
+          title: VHR Image 2018 Level 3 True Color
+          abstract: VHR Image 2018 Level 3 True Color
+        VHR_IMAGE_2018_Level_1__FALSE_COLOR:
+          title: VHR Image 2018 Level 3 False Color
+          abstract: VHR Image 2018 Level 3 False Color
+        VHR_IMAGE_2018_Level_1__NDVI:
+          title: VHR Image 2018 Level 3 NDVI
+          abstract: VHR Image 2018 Level 3 NDVI
+          style: earth
+          # grids? cache options?
+
+

documentation/operator-guide/intro.md

-# Introduction
-
-operator
-
-TODO

documentation/operator-guide/intro.rst

+.. _introduction:
+
+Introduction
+============
+
+This guide will detail the operation of a View Server and all of its
+components.
+
+Since the View Server is a Docker based software and all of its components are
+distributed and executed in context of Docker images and containers, a
+knowledge of Docker and Docker Swarm is required.
+
+## Components
+
+The View Server consists of the following service components (with their
+respective Docker image in parenthesis):
+
+ - Ingestor (ingestor)
+ - Preprocessor (preprocessor)
+ - Registrator (core)
+ - Renderer (core)
+ - Cache (cache)
+ - Seeder (cache)
+ - Client (client)
+ - Database (postgis)
+ - Redis (redis)
+ - Reverse Proxy (traefik)
+
+These services are bundled and managed together in a Docker Swarm via
+Docker Compose configuration files.
+
+Images
+------
+
+The software is distributed as Docker images, which can be instantiated
+and run in their intended role. Some images are hosted on the docker hub,
+the official and default repository for Docker images. Other images reside
+on the gitlab.eox.at repository. Images from the official repository
+are only identified via their name, whereas images from the eox repository
+conventionally use the full URL, including the domain name.
+
+ - mdillon/postgis:10
+ - redis
+ - traefik:2.1
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_core:latest
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_cache:latest
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor:latest
+ - registry.gitlab.eox.at/esa/prism/vs/pvs_client:latest
+
+Configuration Files
+-------------------
+
+The following configuration files will impact the behavior of the View Server:
+
+ - index.html: This is the main file to configure the client. In this file,
+    the viewing layer, search and download enpoints are configured. Usually
+    this is associated with additional backdrop and overlay layers.
+ - preprocessor.yml: This file configures the preprocessing process.
+ - mapcache.xml: This file defines the input sources of the cache and its
+    published layers.
+ - init-db.sh: This file sets up the registrator and renderer side of the VS,
+
+Setup
+-----
+
+In order to help with the initial setup of a VS, the ``pvs_starter`` package
+allows to quickly establish the required structure of configuration files.
+
+It is run from a configuration file and sets up

documentation/operator-guide/setup.rst

+.. _setup:
+
+Setup
+=====
+
+In this chapter the setup of a new VS stack is detailed. Before this step can
+be done, the configuration and environment files need to be present. These
+files can be added manually, or be created in the initialization step.
+
+
+Docker
+------
+
+In order to deploy the Docker Swarm stack to the target machine, Docker and
+its facilities need to be installed. This step depends on the systems
+architecture. On a Debian based system this may look like this:
+
+.. code-block:: bash
+
+    sudo apt-get install \
+        apt-transport-https \
+        ca-certificates \
+        curl \
+        gnupg-agent \
+        software-properties-common
+
+    curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
+
+    # add the apt repository
+    sudo add-apt-repository \
+        "deb [arch=amd64] https://download.docker.com/linux/debian \
+        $(lsb_release -cs) \
+        stable"
+
+    # fetch the package index and install Docker
+    sudo apt-get update
+    sudo apt-get install docker-ce docker-ce-cli containerd.io
+
+## TODO: RHEL?
+
+Docker Swarm
+------------
+
+Now that Docker is installed, the machine can either create a new swarm or join
+an existing one.
+
+To create a new Swarm, the following command can be used:
+
+.. code-block:: bash
+
+    docker swarm init
+
+
+Alternatively, an existing Swarm can be joined. The easiest way to do this, is
+to obtain a ``join-token``. On an existing Swarm manager (where a Swarm was
+initialized or already joined as manager) run this command:
+
+
+.. code-block:: bash
+
+    docker swarm join-token worker
+
+
+This prints out a command that can be run on a machine to join the swarm:
+
+.. code-block:: bash
+
+    docker swarm join --token <obtained token>
+
+
+Additional information for swarm management can be obtained in the official
+documentation of the project:
+https://docs.docker.com/engine/reference/commandline/swarm/
+