EOX GitLab Instance

Skip to content
Snippets Groups Projects

Ingestor status

Merged Mussab Abdalla requested to merge ingestor_status into staging

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Mussab Abdalla added 1 commit

    added 1 commit

    Compare with previous version

  • 94 94 try:
    95 95 client.lpush(queue_name, queue_content)
    96 96
    97 save_Json(filename, 'SUCSSES')
    97 save_json(filename, queue_content, True)
    • The original browse report (not the JSON encoded one sent to the queue) shall be written to the SUCCESS/FAILURE directories. Also, the save_json is not really descriptive (apart from the fact that we want to save XML). What we do, semantically, is to report on the ingestion of a Browse report. Maybe a better name can be found here.

      Also, in case of SFTP upload, the file should actually be removed from the input directory and placed into the SUCCESS/FAILURE dirs.

    • changed this line in version 3 of the diff

    • Please register or sign in to reply
  • Still missing is the documentation in the Operator Guide. About the behavior and the configuration options.

  • Mussab Abdalla added 1 commit

    added 1 commit

    Compare with previous version

  • Mussab Abdalla added 17 commits

    added 17 commits

    Compare with previous version

  • 38 38 ADD install.sh requirements.txt \
    39 39 /
    40 40 RUN /install.sh
    41 RUN mkdir /ingestor
    41 RUN mkdir /ingestor
  • 37 40 def converter(o):
    38 41 if isinstance(o, datetime):
    39 42 return o.isoformat()
    43
    44 def save_report(filename:str, data, success: bool):
  • 37 40 def converter(o):
    38 41 if isinstance(o, datetime):
    39 42 return o.isoformat()
    43
    44 def save_report(filename:str, data, success: bool):
    45
    46 if success:
    47 save_dir = os.environ['INGESTOR_SUCCESS_DIR']
    48 else:
    49 save_dir = os.environ['INGESTOR_FAIL_DIR']
    50
    51 if isinstance(data, str):
    52 shutil.move(data, save_dir)
    53 else :
  • 37 40 def converter(o):
    38 41 if isinstance(o, datetime):
    39 42 return o.isoformat()
    43
    44 def save_report(filename:str, data, success: bool):
    45
    46 if success:
    47 save_dir = os.environ['INGESTOR_SUCCESS_DIR']
    48 else:
    49 save_dir = os.environ['INGESTOR_FAIL_DIR']
    50
    51 if isinstance(data, str):
    52 shutil.move(data, save_dir)
    53 else :
    54
    55 with open(os.path.join(save_dir, '%s.xml' % filename), "w") as outfile:
  • 95 )
    86
    87 with open(event.pathname) as f:
    88 browse_report = parse_browse_report(f)
    89 logger.debug(browse_report)
    90 if isinstance(browse_report, str):
    91 queue_content = browse_report
    92
    93 else:
    94 queue_content = json.dumps(
    95 browse_report, default=converter
    96 )
    97 try:
    96 98 client.lpush(queue_name, queue_content)
    99
    100 save_report('', event.pathname, True)
  • Mussab Abdalla added 20 commits

    added 20 commits

    Compare with previous version

  • Mussab Abdalla added 39 commits

    added 39 commits

    Compare with previous version

  • mentioned in commit 72f9ba19

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading