diff --git a/README.md b/README.md index ec20c42b13004998c58f67981bf5e5b08610c9a6..1a03d587fa0612ab48b2668e06a5c9524d89dd91 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ configuration, installation instructions, as well as canonical references. # Architecture The PRISM View Server (PVS) uses various Docker images whereas `core`, -`cache`, `client`, and `preprocessor` are build from this repository and +`cache`, `client`, `ingestor`, `fluentd` and `preprocessor` are build from this repository and the others are pulled from docker hub. ## Prerequisites @@ -29,11 +29,17 @@ In base stack * traefik-data +In logging stack + +* logging_es-data + Per collection * db-data used by database * redis-data used by redis * instance-data used by registrar and renderer +* report-data sftp output of reporting interface +* from-fepd - sftp input to **ingestor** ## Services @@ -98,9 +104,14 @@ The following services are defined via docker compose files. * seed-success_set * seed-failure_set -### TODO: ingestor +### ingestor -see new service in #7 +* based on ingestor image +* by default a flask app listening on `/` endpoint for `POST` requests with reports +* or can be overriden to be used as inotify watcher on a configured folder for new appearance of reports +* accepts browse reports with references to images on Swift +* extracts the browse metadata (id, time, footprint, image reference) +* `lpush` metadata into a `preprocess-md_queue` ### TODO: seeder @@ -150,9 +161,20 @@ see new service in #7 * provides external service for OpenSearch, WMS, & WCS * renders WMS requests received from cache or seeder -### TODO: ELK stack +### logging stack + +* uses elasticsearch:7.9 & kibana:7.9 external images +* fluentd image is build and published to registry because of additional plugins +* ES data stored in local volume on swarm master +* external access allowed to kibana through traefik +* log parsing enabled for cache and core -see #9 +### sftp + +* uses external atmoz/sftp image +* provides sftp access to two volumes for report exchange on registration result xmls and ingest requirement xmls +* accessible on swarm master on port 2222 +* credentials supplied via config # Usage @@ -182,28 +204,40 @@ docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_cache docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor docker pull registry.gitlab.eox.at/esa/prism/vs/pvs_client docker pull registry.gitlab.eox.at/esa/prism/vs/fluentd +docker pull registry.gitlab.eox.at/esa/prism/vs/ingestor ``` Create external network for stack to run: ``` docker network create -d overlay vhr18-extnet docker network create -d overlay emg-extnet +docker network create -d overlay dem-extnet ``` Add following .env files with credentials to the cloned copy of the repository /env folder: `vhr18_db.env`, `vhr18_obs.env`, `vhr18_django.env`. create docker secrets: -Sensitive environment variables are not included in the .env files, and must be generated as docker secrets. To create docker secrets run : +Sensitive environment variables are not included in the .env files, and must be generated as docker secrets. All stacks currently share these secret names, therefore it must stay the same for all stacks. To create docker secrets run: ```bash # replace the "" with the value of the secret printf "" | docker secret create OS_PASSWORD_DOWNLOAD - printf "" | docker secret create DJANGO_PASSWORD - printf "" | docker secret create OS_PASSWORD - +# for production base stack deployment, additional basic authentication credentials list need to be created +# format of such a list used by traefik are username:hashedpassword (MD5, SHA1, BCrypt) +sudo apt-get install apache2-utils +htpasswd -nb >> auth_list.txt +docker secret create BASIC_AUTH_USERS_AUTH auth_list.txt +docker secret create BASIC_AUTH_USERS_APIAUTH auth_list_api.txt ``` -Deploy the stack: +Deploy the stack in dev environment: ``` docker stack deploy -c docker-compose.vhr18.yml -c docker-compose.vhr18.dev.yml -c docker-compose.logging.yml -c docker-compose.logging.dev.yml vhr18-pvs # start VHR_IMAGE_2018 stack in dev mode, for example to use local sources -docker stack deploy -c docker-compose.emg.yml -c docker-compose.emg.dev.yml emg-pvs -c docker-compose.logging.yml -c docker-compose.logging.dev.yml # start Emergency stack in dev mode, for example to use local sources +docker stack deploy -c docker-compose.emg.yml -c docker-compose.emg.dev.yml -c docker-compose.logging.yml -c docker-compose.logging.dev.yml emg-pvs # start Emergency stack in dev mode, for example to use local sources +``` +Deploy base stack in production environment: +``` +docker stack deploy -c docker-compose.base.ops.yml base-pvs ``` First steps: ``` @@ -230,11 +264,23 @@ On first run, you need to define an index pattern to select the data source for Since we only have fluentd, you can just use `*` as index pattern. Select `@timestamp` as time field ([see also](https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html)). - - +Example of a kibana query to discover logs of a single service: +``` +https:///app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(path,size,code,log),filters:!(),index:,interval:auto,query:(language:kuery,query:'%20container_name:%20""'),sort:!()) +``` +Development service stacks keep their logging to stdout/stderr unless `logging` dev stack is used. +On production machine, `fluentd` is set as a logging driver for docker daemon by modifying `/etc/docker/daemon.json` to +``` +{ + "log-driver": "fluentd", + "log-opts": { + "fluentd-sub-second-precision": "true" + } +} +``` ### setup sftp -The `SFTP` image allow remote access into 2 logging folders, you can define (edit/add) users, passwords and (UID/GID) in the respictive configuration file ( e.g *config/vhr_sftp_users.conf* ). +The `SFTP` image allow remote access into 2 logging folders, you can define (edit/add) users, passwords and (UID/GID) in the respective configuration file ( e.g *config/vhr_sftp_users.conf* ). The default username is `eox`, once the stack is deployed you can sftp into the logging folders through port 2222 on -if you are running the dev stack- localhost : @@ -243,7 +289,7 @@ sftp -P 2222 eox@127.0.0.1 ``` You will log in into`/home/eox/data` directory which contains the 2 logging directories : `to/panda` and `from/fepd` - **NOTE:** The mounted directory that you are directed into is *`/home/user`*, where `user` is the username, hence when changing the username in the `.conf` file, the `sftp` mounted volumes path in `docker-compse..yml` must change respectivly. + **NOTE:** The mounted directory that you are directed into is *`/home/user`*, where `user` is the username, hence when changing the username in the `.conf` file, the `sftp` mounted volumes path in `docker-compose..yml` must change respectively. # Documentation @@ -289,4 +335,6 @@ docker save -o pvs_core.tar registry.gitlab.eox.at/esa/prism/vs/pvs_core docker save -o pvs_cache.tar registry.gitlab.eox.at/esa/prism/vs/pvs_cache docker save -o pvs_preprocessor.tar registry.gitlab.eox.at/esa/prism/vs/pvs_preprocessor docker save -o pvs_client.tar registry.gitlab.eox.at/esa/prism/vs/pvs_client +docker save -o pvs_ingestor.tar registry.gitlab.eox.at/esa/prism/vs/pvs_ingestor +docker save -o fluentd.tar registry.gitlab.eox.at/esa/prism/vs/fluentd ``` diff --git a/conftest.py b/conftest.py deleted file mode 100644 index 7a1d3249386440525e74a35d4574a36f9781c633..0000000000000000000000000000000000000000 --- a/conftest.py +++ /dev/null @@ -1,10 +0,0 @@ -def pytest_addoption(parser): - parser.addoption("--name", action="store", default="default name") - - -def pytest_generate_tests(metafunc): - # This is called for every test. Only get/set command line arguments - # if the argument is specified in the list of test "fixturenames". - option_value = metafunc.config.option.name - if 'name' in metafunc.fixturenames and option_value is not None: - metafunc.parametrize("name", [option_value]) \ No newline at end of file diff --git a/documentation/operator-guide/configuration.rst b/documentation/operator-guide/configuration.rst index 578bd680b47cbe2e61596c6357ca9edbf89c2866..b93bc1ba258db58ee729166894d883fa2b5b24fd 100644 --- a/documentation/operator-guide/configuration.rst +++ b/documentation/operator-guide/configuration.rst @@ -212,6 +212,33 @@ These are the internal access credentials for the database: * ``DB_PORT`` * ``DB_NAME`` +Sensitive variables +^^^^^^^^^^^^^^^^^^^ + +Since environment variables include credentials that are considered sensitive, +avoiding their exposure inside ``.env`` files would be the right practice. +In order to manage transmitting sensitive data securely into the respective containers, +docker secrets with the values of these variables should be created. Currently, three +variables have to be saved as docker secrets before deploying the swarm: +``OS_PASSWORD``, ``OS_PASSWORD_DOWNLOAD`` and ``DJANGO_PASSWORD``. + +Two other docker secrets need to be created for traefik basic authentication: +``BASIC_AUTH_USERS_AUTH`` - used for access to services, ``BASIC_AUTH_USERS_APIAUTH`` - used for admin access to kibana and traefik. +These secrets should be text files containing a list of username:hashedpassword (MD5, SHA1, BCrypt) pairs. + +An example of creating ``OS_PASSWORD`` as secret using the following command : + +.. code-block:: bash + + printf "" | docker secret create OS_PASSWORD - + +An example of creating ``BASIC_AUTH_USERS_AUTH`` secret: + +.. code-block:: bash + htpasswd -nb user1 3vYxfRqUx4H2ar3fsEOR95M30eNJne >> auth_list.txt + htpasswd -nb user2 YyuN9bYRvBUUU6COx7itWw5qyyARus >> auth_list.txt + docker secret create BASIC_AUTH_USERS_AUTH auth_list.txt + Configuration Files -------------------