Updating operator docs: description of preprocessor config

documentation/operator-guide/configuration.rst

@@ -240,18 +240,252 @@ such a configuration file is defined and the used in a service:
 
 The following configuration files are used throughout the VS:
 
-* ``<stack-name>_init-db.sh``: This shell script file's purpose is to set up
-  the EOxServer instance used by both the renderer and registrar.
-* ``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``: The
-  clients main HTML page, containing various client settings. The ``dev`` one
-  is used for development only, whereas the ``ops`` one is used for operational
-  deployment.
-* ``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``: The
-  configuration file for MapCache, the software powering the cache service.
-  Similarly to the client configuration files, the ``dev`` and ``ops`` files
-  used for development and operational usage respectively. Further
-  documentation can be found at `the official site
-  <https://mapserver.org/mapcache/config.html>`_.
+``<stack-name>_init-db.sh``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This shell script file's purpose is to set up the EOxServer instance used by
+both the renderer and registrar.
+
+``<stack-name>_index-dev.html``/``<stack-name>_index-ops.html``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The clients main HTML page, containing various client settings. The ``dev`` one
+is used for development only, whereas the ``ops`` one is used for operational
+deployment.
+
+``<stack-name>_mapcache-dev.xml``/``<stack-name>_mapcache-ops.xml``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The configuration file for MapCache, the software powering the cache service.
+Similarly to the client configuration files, the ``dev`` and ``ops`` files
+used for development and operational usage respectively. Further
+documentation can be found at `the official site
+<https://mapserver.org/mapcache/config.html>`_.
+
+``<stack-name>_preprocessor-config.yaml``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The configuration for the proprocessing service to use to process to be
+ingested files.
+
+The files are using YAML as a format and are structured in the following
+fashion:
+
+source/target
+
+  Here, the source file storage and the target file storage are configured.
+  This can either be a local directory or an OpenStack Swift object storage.
+
+workdir
+
+  The workdir can be configured, to determine where the intermediate files are
+  placed. This can be convenient for debugging and development.
+
+keep_temp
+
+  This boolean decides if the temporary directory for the preprocessing will be
+  cleaned up after being finished. Also, convenient for development.
+
+metadata_glob
+
+  This file glob is used to determine the main metadata file to extract the
+  product type from. This file will be searched in the downloaded package.
+
+type_extractor
+
+  This setting configures how the product type is extracted from the previously
+  extracted metadata. In the ``xpath`` setting one or more XPath expressions
+  can supplied to fetch the product type. Each XPath will be tried until one is
+  found that produces a result. These results can then be mapped using the
+  ``map`` dictionary.
+
+level_extractor
+
+  This section works very similar to the ``type_extractor`` but only for the
+  product level. The product level is currently not used.
+
+preprocessing
+
+  This is the actual preprocessing configuration setting. It is split in
+  defaults and product type specific settings. The defaults are applied
+  where there is no setting supplied for that specific type. The product type
+  is the one extracted earlier.
+
+  defaults
+
+    This section allows to configure any one of the available steps. Each step
+    configuration can be overridden in a specific product type configuration.
+
+    The available steps are as follows:
+
+    custom_preprocessor
+
+      A custom python function to be called.
+
+      path
+
+        The Python module path to the function to call.
+
+      args
+
+        A list of arguments to pass to the function.
+
+      kwargs
+
+        A dictionary of keyword arguments to pass to the function.
+
+    subdatasets
+
+      What subdatasets to extract and how to name them.
+
+      data_file_glob
+
+        A file glob pattern to select files to extract from.
+
+      subdataset_types
+
+        Mapping of subdataset identifier to output filename postfix for
+        subdatasets to be extracted for each data file.
+
+    georeference
+
+      How the extracted files shall be georeferenced.
+
+      type
+
+        The type of georeferencing to apply. One of ``gcp``, ``rpc``,
+        ``corner``, ``world``.
+
+      options
+
+        Additional options for the georeferencing. Depends on the type of
+        georeferencing.
+
+        order
+
+          The polynomial order to use for GCP related georeferencing.
+
+        projection
+
+          The projection to use for ungeoreferenced images.
+
+        rpc_file_template
+
+          The file glob template to use to find the RPC file. Template
+          parameters are {filename}, {fileroot}, and {extension}.
+
+        warp_options
+
+          Warp options. See
+          https://gdal.org/python/osgeo.gdal-module.html#WarpOptions for
+          details
+
+        corner_names
+
+          The metadata field name including the corner names. Tuple of four:
+          bottom-left, bottom-right, top-left and top-right
+
+        orbit_direction_name
+
+          The metadata field name containing the orbit direction
+
+        force_north_up
+
+          TODO
+
+        tps
+
+          Whether to use TPS transformation instead of GCP polynomials.
+
+    calc
+
+      Calculate derived data using formulas.
+
+      formulas
+
+        A list of formulas to use to calculate derived data. Each has the
+        following fields
+
+        inputs
+
+          A map of characters in the range of A-Z to respective inputs. Each
+          has the following properties
+
+          glob
+
+            The input file glob
+
+          band
+
+            The input file band index (1-based)
+
+        data_type
+
+          The GDAL data type name for the output
+
+        formula
+
+          The formula to apply. See
+          https://gdal.org/programs/gdal_calc.html#cmdoption-calc for details.
+
+        output_postfix
+
+          The postfix to apply for the filename of the created file.
+
+        nodata_value
+
+          The nodata value to be used.
+
+    stack_bands
+
+      Concatenate bands and arrange them in a single file.
+
+      group_by
+
+        A regex to group the input datasets, if consisting of multiple file.
+        The first regex group is used for the grouping.
+
+      sort_by
+
+        A regex to select a portion of the filename to be used for sorting. The
+        first regex group is used.
+
+      order
+
+        The order of the extracted item used in 'sort_by'. When the value
+        extracted by ``sort_by`` is missing, then that file will be dropped.
+
+    output
+
+      Final adjustments to generate an output file. Add overviews, reproject to
+      a common projection, etc.
+
+      options
+
+        Options to be passed to `gdal.Warp`. See
+        https://gdal.org/python/osgeo.gdal-module.html#WarpOptions for details.
+
+    custom_preprocessor
+
+      A custom python function to be called.
+
+      path
+
+        The Python module path to the function to call.
+
+      args
+
+        A list of arguments to pass to the function.
+
+      kwargs
+
+        A dictionary of keyword arguments to pass to the function.
+
+  types
+
+    This mapping of product type identifier to step configuration allows to
+    define specific step settings, even overriding the values from the
+    defaults.
 
 The next section :ref:`management` describes how an operator interacts with a
 deployed VS stack.