management.rst

.. _management:

Service Management
==================

This section shows how a deployed VS stack can and should be interacted with.


Scaling
-------

Scaling is a handy tool to ensure stable performance, even when dealing with
higher usage on any service. For example, the preprocessor and registrar can
be scaled to a higher replica count to enable a better throughput when
ingesting data into the VS.

The following command scales the ``renderer`` service to 5 replicas:

.. code-block:: bash

    docker service scale <stack-name>_renderer=5

A service can also be scaled to zero replicas, effectively disabling the
service.

.. warning::

    The ``redis`` and ``database`` should never be scaled (their replica count
    should remain 1) as this can lead to service disruptions and corrupted data.


Updating Images
---------------

Updating the service software is done using previously established tools. To
update the service in question, it needs to be scaled to zero replicas. Then
the new image can be pulled, and the service can be scaled back to its original
value. This forces the start of the service from the newly fetched image.
Another option to keep the service running during the upgrade procedure is to
sequentially restart the individual instances of the services after pulling a
newer image using a command:

.. code-block:: bash

    docker service update --force <stack-name>_<service-name>

Updating configurations or environment files
--------------------------------------------

Updating the service configurations or environment files used can not be done
just by rescaling the impacted services to 0 and rerunning. The whole stack
needs to be shut down using the command:

.. code-block:: bash

    docker stack rm <stack-name>

A new deployment of the stack will use the updated configuration. The above
mentioned process necessarily involves a certain service downtime between
shutting down of the stack and new deployment.

Cleaning up
-----------
Current configuration of the services does not have any log rotation set up, which means that service logs can grow significantly over time, if left not maintained and set on verbose logging levels. In order to delete logs older than 7 days from a single node, a following command can be run

.. code-block:: bash

    journalctl --vacuum-time=7d

Additionally in order to delete older logs from docker containers present on a node, keeping only a certain number of newest rows, a following command can be run.

.. code-block:: bash

    truncate -s <number rows to keep> $(docker inspect -f '{{.LogPath}}' $container 2> /dev/null)


The final section :ref:`ingestion` explains how to get data into the VS.