From c95f23fe42e9fe36287b6ea4ff505bcc48b253d8 Mon Sep 17 00:00:00 2001 From: baloola Date: Tue, 20 Oct 2020 11:20:22 +0200 Subject: [PATCH 1/5] adding docker secret to operator guide --- README.md | 2 ++ documentation/operator-guide/configuration.rst | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/README.md b/README.md index ec20c42b..9333a429 100644 --- a/README.md +++ b/README.md @@ -244,6 +244,8 @@ sftp -P 2222 eox@127.0.0.1 You will log in into`/home/eox/data` directory which contains the 2 logging directories : `to/panda` and `from/fepd` **NOTE:** The mounted directory that you are directed into is *`/home/user`*, where `user` is the username, hence when changing the username in the `.conf` file, the `sftp` mounted volumes path in `docker-compse..yml` must change respectivly. + +Once a product is registered, a xml report that contains `WMS` and `WCS` getCapabilities links is generated and saved in the same volume which `to/panda` is mounted to, once you successfuly sftp into the "sftp image" you can navigate to the generated reports. # Documentation diff --git a/documentation/operator-guide/configuration.rst b/documentation/operator-guide/configuration.rst index 578bd680..ff600788 100644 --- a/documentation/operator-guide/configuration.rst +++ b/documentation/operator-guide/configuration.rst @@ -212,6 +212,22 @@ These are the internal access credentials for the database: * ``DB_PORT`` * ``DB_NAME`` +Sensitive variables +^^^^^^^^^^^^^^^^^^^ + +Since environment variables include credentials that are considered sensitive, +avoiding their exposure inside ``.env`` files would be the right practice. +In order to manage transmiting sensitive data securely into the respective containers, +docker secrets with the values of these variables should be created. currently, three +variables shall be passed as secrets before deploying the swarm: +``OS_PASSWORD``, ``OS_PASSWORD_DOWNLOAD`` and ``DJANGO_PASSWORD``. +An example of creating ``OS_PASSWORD`` as secret using the following comand : + +.. code-block:: bash + + printf "" | docker secret create OS_PASSWORD - + + Configuration Files ------------------- -- GitLab From 69c22eca3352a6f94a8c707b2558528911559845 Mon Sep 17 00:00:00 2001 From: Lubomir Bucek Date: Tue, 20 Oct 2020 16:13:16 +0200 Subject: [PATCH 2/5] update readme --- README.md | 20 +++++++++++++++----- conftest.py | 10 ---------- 2 files changed, 15 insertions(+), 15 deletions(-) delete mode 100644 conftest.py diff --git a/README.md b/README.md index 9333a429..3cce22ff 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ configuration, installation instructions, as well as canonical references. # Architecture The PRISM View Server (PVS) uses various Docker images whereas `core`, -`cache`, `client`, and `preprocessor` are build from this repository and +`cache`, `client`, `ingestor`, `fluentd` and `preprocessor` are build from this repository and the others are pulled from docker hub. ## Prerequisites @@ -29,11 +29,17 @@ In base stack * traefik-data +In logging stack + +* logging_es-data + Per collection * db-data used by database * redis-data used by redis * instance-data used by registrar and renderer +* report-data sftp output of reporting interface +* from-fepd - sftp input to **ingestor** ## Services @@ -182,17 +188,19 @@ docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_cache docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_client docker pull registry.gitlab.eox.at/esa/prism/vs/fluentd +docker pull registry.gitlab.eox.at/esa/prism/vs/ingestor ``` Create external network for stack to run: ``` docker network create -d overlay vhr18-extnet docker network create -d overlay emg-extnet +docker network create -d overlay dem-extnet ``` Add following .env files with credentials to the cloned copy of the repository /env folder: `vhr18_db.env`, `vhr18_obs.env`, `vhr18_django.env`. create docker secrets: -Sensitive environment variables are not included in the .env files, and must be generated as docker secrets. To create docker secrets run : +Sensitive environment variables are not included in the .env files, and must be generated as docker secrets. All stacks currently share these secret names, therefore it must stay the same for all stacks. To create docker secrets run: ```bash # replace the "" with the value of the secret printf "" | docker secret create OS_PASSWORD_DOWNLOAD - @@ -200,7 +208,7 @@ printf "" | docker secret create DJANGO_PASSWORD - printf "" | docker secret create OS_PASSWORD - ``` -Deploy the stack: +Deploy the stack in dev environment: ``` docker stack deploy -c docker-compose.vhr18.yml -c docker-compose.vhr18.dev.yml -c docker-compose.logging.yml -c docker-compose.logging.dev.yml vhr18-pvs # start VHR_IMAGE_2018 stack in dev mode, for example to use local sources docker stack deploy -c docker-compose.emg.yml -c docker-compose.emg.dev.yml emg-pvs -c docker-compose.logging.yml -c docker-compose.logging.dev.yml # start Emergency stack in dev mode, for example to use local sources @@ -234,7 +242,7 @@ Select `@timestamp` as time field ### setup sftp -The `SFTP` image allow remote access into 2 logging folders, you can define (edit/add) users, passwords and (UID/GID) in the respictive configuration file ( e.g *config/vhr_sftp_users.conf* ). +The `SFTP` image allow remote access into 2 logging folders, you can define (edit/add) users, passwords and (UID/GID) in the respective configuration file ( e.g *config/vhr_sftp_users.conf* ). The default username is `eox`, once the stack is deployed you can sftp into the logging folders through port 2222 on -if you are running the dev stack- localhost : @@ -243,7 +251,7 @@ sftp -P 2222 eox@127.0.0.1 ``` You will log in into`/home/eox/data` directory which contains the 2 logging directories : `to/panda` and `from/fepd` - **NOTE:** The mounted directory that you are directed into is *`/home/user`*, where `user` is the username, hence when changing the username in the `.conf` file, the `sftp` mounted volumes path in `docker-compse..yml` must change respectivly. + **NOTE:** The mounted directory that you are directed into is *`/home/user`*, where `user` is the username, hence when changing the username in the `.conf` file, the `sftp` mounted volumes path in `docker-compse..yml` must change respectively. Once a product is registered, a xml report that contains `WMS` and `WCS` getCapabilities links is generated and saved in the same volume which `to/panda` is mounted to, once you successfuly sftp into the "sftp image" you can navigate to the generated reports. @@ -291,4 +299,6 @@ docker save -o pvs_core.tar registry.gitlab.eox.at/esa/prism/vs/pvs_core docker save -o pvs_cache.tar registry.gitlab.eox.at/esa/prism/vs/pvs_cache docker save -o pvs_preprocessor.tar registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor docker save -o pvs_client.tar registry.gitlab.eox.at/esa/prism/vs/pvs_client +docker save -o pvs_ingestor.tar registry.gitlab.eox.at/esa/prism/vs/pvs_ingestor +docker save -o fluentd.tar registry.gitlab.eox.at/esa/prism/vs/fluentd ``` diff --git a/conftest.py b/conftest.py deleted file mode 100644 index 7a1d3249..00000000 --- a/conftest.py +++ /dev/null @@ -1,10 +0,0 @@ -def pytest_addoption(parser): - parser.addoption("--name", action="store", default="default name") - - -def pytest_generate_tests(metafunc): - # This is called for every test. Only get/set command line arguments - # if the argument is specified in the list of test "fixturenames". - option_value = metafunc.config.option.name - if 'name' in metafunc.fixturenames and option_value is not None: - metafunc.parametrize("name", [option_value]) \ No newline at end of file -- GitLab From ea1b630700a576af2114c361fb55925dbf31a6a6 Mon Sep 17 00:00:00 2001 From: Lubomir Bucek Date: Tue, 20 Oct 2020 17:15:13 +0200 Subject: [PATCH 3/5] update readme --- README.md | 50 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 3cce22ff..8276c1be 100644 --- a/README.md +++ b/README.md @@ -104,9 +104,14 @@ The following services are defined via docker compose files. * seed-success_set * seed-failure_set -### TODO: ingestor +### ingestor -see new service in #7 +* based on ingestor image +* by default a flask app listening on `/` endpoint for `POST` requests with reports +* or can be overriden to be used as inotify watcher on a configured folder for new appearance of reports +* accepts browse reports with references to images on Swift +* extracts the browse metadata (id, time, footprint, image reference) +* `lpush` metadata into a `preprocess-md_queue` ### TODO: seeder @@ -156,9 +161,20 @@ see new service in #7 * provides external service for OpenSearch, WMS, & WCS * renders WMS requests received from cache or seeder -### TODO: ELK stack +### logging stack -see #9 +* uses elasticsearch:7.9 & kibana:7.9 external images +* fluentd image is build and published to registry because of additional plugins +* ES data stored in local volume on swarm master +* external access allowed to kibana through traefik +* log parsing enabled for cache and core + +### sftp + +* uses external atmoz/sftp image +* provides sftp access to two volumes for report exchange on registration result xmls and ingest requirement xmls +* accessible on swarm master on port 2222 +* credentials supplied via config # Usage @@ -211,7 +227,11 @@ printf "" | docker secret create OS_PASSWORD - Deploy the stack in dev environment: ``` docker stack deploy -c docker-compose.vhr18.yml -c docker-compose.vhr18.dev.yml -c docker-compose.logging.yml -c docker-compose.logging.dev.yml vhr18-pvs # start VHR_IMAGE_2018 stack in dev mode, for example to use local sources -docker stack deploy -c docker-compose.emg.yml -c docker-compose.emg.dev.yml emg-pvs -c docker-compose.logging.yml -c docker-compose.logging.dev.yml # start Emergency stack in dev mode, for example to use local sources +docker stack deploy -c docker-compose.emg.yml -c docker-compose.emg.dev.yml -c docker-compose.logging.yml -c docker-compose.logging.dev.yml emg-pvs # start Emergency stack in dev mode, for example to use local sources +``` +Deploy base stack in production environment: +``` +docker stack deploy -c docker-compose.base.ops.yml base-pvs ``` First steps: ``` @@ -238,8 +258,20 @@ On first run, you need to define an index pattern to select the data source for Since we only have fluentd, you can just use `*` as index pattern. Select `@timestamp` as time field ([see also](https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html)). - - +Example of a kibana query to discover logs of a single service: +``` +https:///app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(path,size,code,log),filters:!(),index:,interval:auto,query:(language:kuery,query:'%20container_name:%20""'),sort:!()) +``` +Development service stacks keep their logging to stdout/stderr unless `logging` dev stack is used. +On production machine, `fluentd` is set as a logging driver for docker daemon by modifying `/etc/docker/daemon.json` to +``` +{ + "log-driver": "fluentd", + "log-opts": { + "fluentd-sub-second-precision": "true" + } +} +``` ### setup sftp The `SFTP` image allow remote access into 2 logging folders, you can define (edit/add) users, passwords and (UID/GID) in the respective configuration file ( e.g *config/vhr_sftp_users.conf* ). @@ -251,9 +283,7 @@ sftp -P 2222 eox@127.0.0.1 ``` You will log in into`/home/eox/data` directory which contains the 2 logging directories : `to/panda` and `from/fepd` - **NOTE:** The mounted directory that you are directed into is *`/home/user`*, where `user` is the username, hence when changing the username in the `.conf` file, the `sftp` mounted volumes path in `docker-compse..yml` must change respectively. - -Once a product is registered, a xml report that contains `WMS` and `WCS` getCapabilities links is generated and saved in the same volume which `to/panda` is mounted to, once you successfuly sftp into the "sftp image" you can navigate to the generated reports. + **NOTE:** The mounted directory that you are directed into is *`/home/user`*, where `user` is the username, hence when changing the username in the `.conf` file, the `sftp` mounted volumes path in `docker-compose..yml` must change respectively. # Documentation -- GitLab From cd24c1d10a746fdf7772d7ab971327f2ae2eddde Mon Sep 17 00:00:00 2001 From: Lubomir Bucek Date: Wed, 21 Oct 2020 15:35:13 +0200 Subject: [PATCH 4/5] add info about traefik secrets to readme --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 8276c1be..c901770d 100644 --- a/README.md +++ b/README.md @@ -222,6 +222,12 @@ Sensitive environment variables are not included in the .env files, and must be printf "" | docker secret create OS_PASSWORD_DOWNLOAD - printf "" | docker secret create DJANGO_PASSWORD - printf "" | docker secret create OS_PASSWORD - +# for production base stack deployment, additonal basic authentication credentials list need to be created +# format of such a list used by traefik are username:hashedpassword (MD5, SHA1, BCrypt) +sudo apt-get install apache2-utils +htpasswd -n >> auth_list.txt +docker secret create BASIC_AUTH_USERS_AUTH auth_list.txt +docker secret create BASIC_AUTH_USERS_APIAUTH auth_list_api.txt ``` Deploy the stack in dev environment: -- GitLab From 91a8c9f9933f8fa0cc26c4cd61658ddef4e26b97 Mon Sep 17 00:00:00 2001 From: Lubomir Bucek Date: Wed, 21 Oct 2020 15:44:18 +0200 Subject: [PATCH 5/5] add traefik secrets to operators guide --- README.md | 4 ++-- .../operator-guide/configuration.rst | 19 +++++++++++++++---- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index c901770d..1a03d587 100644 --- a/README.md +++ b/README.md @@ -222,10 +222,10 @@ Sensitive environment variables are not included in the .env files, and must be printf "" | docker secret create OS_PASSWORD_DOWNLOAD - printf "" | docker secret create DJANGO_PASSWORD - printf "" | docker secret create OS_PASSWORD - -# for production base stack deployment, additonal basic authentication credentials list need to be created +# for production base stack deployment, additional basic authentication credentials list need to be created # format of such a list used by traefik are username:hashedpassword (MD5, SHA1, BCrypt) sudo apt-get install apache2-utils -htpasswd -n >> auth_list.txt +htpasswd -nb >> auth_list.txt docker secret create BASIC_AUTH_USERS_AUTH auth_list.txt docker secret create BASIC_AUTH_USERS_APIAUTH auth_list_api.txt ``` diff --git a/documentation/operator-guide/configuration.rst b/documentation/operator-guide/configuration.rst index ff600788..b93bc1ba 100644 --- a/documentation/operator-guide/configuration.rst +++ b/documentation/operator-guide/configuration.rst @@ -217,16 +217,27 @@ Sensitive variables Since environment variables include credentials that are considered sensitive, avoiding their exposure inside ``.env`` files would be the right practice. -In order to manage transmiting sensitive data securely into the respective containers, -docker secrets with the values of these variables should be created. currently, three -variables shall be passed as secrets before deploying the swarm: +In order to manage transmitting sensitive data securely into the respective containers, +docker secrets with the values of these variables should be created. Currently, three +variables have to be saved as docker secrets before deploying the swarm: ``OS_PASSWORD``, ``OS_PASSWORD_DOWNLOAD`` and ``DJANGO_PASSWORD``. -An example of creating ``OS_PASSWORD`` as secret using the following comand : + +Two other docker secrets need to be created for traefik basic authentication: +``BASIC_AUTH_USERS_AUTH`` - used for access to services, ``BASIC_AUTH_USERS_APIAUTH`` - used for admin access to kibana and traefik. +These secrets should be text files containing a list of username:hashedpassword (MD5, SHA1, BCrypt) pairs. + +An example of creating ``OS_PASSWORD`` as secret using the following command : .. code-block:: bash printf "" | docker secret create OS_PASSWORD - +An example of creating ``BASIC_AUTH_USERS_AUTH`` secret: + +.. code-block:: bash + htpasswd -nb user1 3vYxfRqUx4H2ar3fsEOR95M30eNJne >> auth_list.txt + htpasswd -nb user2 YyuN9bYRvBUUU6COx7itWw5qyyARus >> auth_list.txt + docker secret create BASIC_AUTH_USERS_AUTH auth_list.txt Configuration Files ------------------- -- GitLab