EOX GitLab Instance
Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
View Server 2
harvester
Commits
0e73be9b
Commit
0e73be9b
authored
Dec 09, 2021
by
Fabian Schindler
Browse files
Fixing test config
Fixing filescheme
parent
74f9e83c
Pipeline
#19526
passed with stage
in 39 seconds
Changes
2
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
harvester/filescheme/__init__.py
View file @
0e73be9b
...
@@ -17,6 +17,4 @@ def get_filescheme(filescheme_cfg: dict) -> Optional[FileScheme]:
...
@@ -17,6 +17,4 @@ def get_filescheme(filescheme_cfg: dict) -> Optional[FileScheme]:
if
not
cls
:
if
not
cls
:
return
None
return
None
filescheme
=
cls
(
get_source
(
filescheme_cfg
[
"source"
]),
**
filescheme_cfg
)
return
cls
(
get_source
(
filescheme_cfg
.
pop
(
"source"
)),
**
filescheme_cfg
)
return
filescheme
tests/data/config.yml
View file @
0e73be9b
...
@@ -15,16 +15,19 @@ harvesters:
...
@@ -15,16 +15,19 @@ harvesters:
extract_property
:
null
extract_property
:
null
-
name
:
Fusion-data
-
name
:
Fusion-data
resource
:
resource
:
type
:
S3
type
:
FileMatcher
parameters
:
path
:
vrt/SR/
url
:
https://eox-fusion.s3.amazonaws.com/
# file_regex: \d{4}-\d{2}-\d{2}.vrt
time_regex
:
\d{4}-\d{2}-\d{2}
# time_format: "%Y-%m-%d"
url
:
https://eox-fusion.s3.amazonaws.com/
source
:
type
:
S3
bucket
:
eox-fusion
endpoint_url
:
https://eox-fusion.s3.amazonaws.com/
access_key_id
:
AKIAXVU5ZHGDAZ3XXOM2
access_key_id
:
AKIAXVU5ZHGDAZ3XXOM2
secret_access_key
:
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
secret_access_key
:
Su2hUc6clQDZ7QVVX/Q3sdvgPTaWB5A7UZZSMvVk
region
:
eu-central-1
region_name
:
eu-central-1
prefix
:
vrt/SR/
file_regex
:
\d{4}-\d{2}-\d{2}.vrt
time_regex
:
\d{4}-\d{2}-\d{2}
time_format
:
"
%Y-%m-%d"
filter
:
null
filter
:
null
mode
:
item
mode
:
item
queue
:
register
queue
:
register
...
@@ -54,7 +57,6 @@ harvesters:
...
@@ -54,7 +57,6 @@ harvesters:
property_mapping
:
property_mapping
:
start_datetime
:
startDate
start_datetime
:
startDate
end_datetime
:
completionDate
end_datetime
:
completionDate
provider
:
testmodule.provider.SampleProvider
postprocessor
:
testmodule.provider.SampleProvider
filter
:
{}
queue
:
register
queue
:
register
mode
:
item
mode
:
item
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment