From 3d65d48bd2013cc4fb2895ed4fd221781e47146f Mon Sep 17 00:00:00 2001
From: Lubomir Bucek <lubomir.bucek@eox.at>
Date: Mon, 13 Jul 2020 17:45:56 +0200
Subject: [PATCH] add section about logs to management chapter of
operators-guide
---
documentation/operator-guide/management.rst | 48 +++++++++++++++++++++
1 file changed, 48 insertions(+)
diff --git a/documentation/operator-guide/management.rst b/documentation/operator-guide/management.rst
index 3125157c..4f8abcb2 100644
--- a/documentation/operator-guide/management.rst
+++ b/documentation/operator-guide/management.rst
@@ -59,6 +59,54 @@ A new deployment of the stack will use the updated configuration. The above
mentioned process necessarily involves a certain service downtime between
shutting down of the stack and new deployment.
+Inspecting logs
+---------------
+
+All service components are running inside docker containers and it is therefore possible to inspect the logs for anomalies via standard docker logs calls redirected for example to less command to allow paging through them.
+
+.. code-block:: bash
+
+ docker logs <container-name> 2>&1 | less
+
+In case that only one instance of a service is running on one node, the <container-name> can be returned by fetching the available containers of a service on that node with a command
+
+.. code-block:: bash
+
+ docker logs $(docker ps -qf "name=<stack-name>_<service-name>") 2>&1 | less
+
+It is possible to show logs of all containers belonging to a service from a master node, utilizing `docker service logs` command, but the resulting listing does not enforce sorting by time. Although logs of each task appear in the order they were inserted, logs of all tasks are outputted interleaved. To quickly check latest time-sorted logs from the service, sorting the entries by timestamp column, do:
+
+.. code-block:: bash
+
+ docker service logs <stack-name>_<service-name> -t 2>&1 | sort -k 1 2>&1 | tail -n <number-of-last-lines> 2>&1 | less
+
+The docker service logs is intended as a quick way to view the latest log entries of all tasks of a service, but should not be used as a main way to collect these logs. For that it would be appropriate to use a logging driver, and extract the logs from the nodes to a central log aggregator.
+The docker service logs is intended as a quick way to view the latest log entries of all tasks of a service, but should not be used as a main way to collect these logs. For that it would be appropriate to use a logging driver, and extract the logs from the nodes to a central log aggregator.
+
+Increasing logging level
+------------------------
+In default state, all components are configured to behave in production logging setup, where the amount of information contained in the logs is reduced. Different components contain different ways to increase the reported logging level for debugging purposes.
+
+In order to increase logging level of **EOXServer** component, and therefore of each service, which depends on it, a `DEBUG` configuration option contained in file **$INSTALL_DIR/pvs_instance/settings.py** needs to be set to **True**. This setting needs to be applied on each node, where there is a running a service for which the `DEBUG` logging should be enabled, as it is stored in the respective docker volume <stack-name>_instance-data, which is created per node.
+
+A restart of respective service for the change to be applied is also necessary. In order to change the DEBUG settings on an example of a renderer, do
+
+.. code-block:: bash
+
+ docker exec -it $(docker ps -qf "name=<stack-name>_renderer") bash
+ cd ${INSTALL_DIR}/pvs_instance
+ sed -i 's/DEBUG = False/DEBUG = True/g' settings.py
+
+In order to increase logging level of registrar and preprocessor services to `debug`, the respective Python scripts need to be run with an optional parameter **-v 4**.
+
+The cache services internally uses a Mapcache software, which usually incorporates an Apache 2 HTTP Server. Due to that, logging level is shared throughout the whole service and is based on Apache `.conf` file, which is stored in $APACHE_CONF environment variable. To change the logging level, edit this file, by setting a **LogLevel debug** and then gracefully restart the Apache component (this way, the cache service itself will not restart and renew default configuration).
+
+.. code-block:: bash
+
+ docker exec -it $(docker ps -qf "name=<stack-name>_cache") bash
+ sed -i 's/<\/VirtualHost>/ LogLevel debug\n<\/VirtualHost>/g' $APACHE_CONF
+ apachectl -k graceful
+
Cleaning up
-----------
Current configuration of the services does not have any log rotation set up, which means that service logs can grow significantly over time, if left not maintained and set on verbose logging levels. In order to delete logs older than 7 days from a single node, a following command can be run
--
GitLab