EOX GitLab Instance
Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
View Server 2
harvester
Commits
e65be197
Commit
e65be197
authored
Dec 09, 2021
by
Fabian Schindler
Browse files
Adding postprocessing capabilities
parent
26b364eb
Pipeline
#19505
failed with stage
in 45 seconds
Changes
2
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
harvester/harvester.py
View file @
e65be197
...
...
@@ -10,7 +10,7 @@ from .endpoint import get_endpoint
from
.source
import
get_source
from
.exceptions
import
HarvestError
from
.utils
import
cql_filter
from
.postprocess
import
get_postprocessor
logger
=
logging
.
getLogger
(
__name__
)
...
...
@@ -52,6 +52,13 @@ def main(config: dict, value: str, client: Redis):
# Perform harvest
result
=
resource
.
harvest
()
if
"postprocess"
in
harvest_config
:
postprocessor
=
get_postprocessor
(
harvest_config
[
"postprocess"
])
result
=
(
postprocessor
.
postprocess
(
item
)
for
item
in
result
)
# Filter data
result
=
cql_filter
(
harvest_config
[
"filter"
],
result
)
...
...
harvester/postprocess.py
0 → 100644
View file @
e65be197
from
abc
import
ABC
,
abstractmethod
class
Postprocessor
(
ABC
):
@
abstractmethod
def
postprocess
(
self
,
item
:
dict
)
->
dict
:
pass
POSTPROCESSORS
=
{
}
def
get_postprocessor
(
config
:
dict
)
->
Postprocessor
:
cls
=
POSTPROCESSORS
[
config
.
pop
(
"type"
)]
return
cls
(
**
config
)
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment