8000 Retry the event-store read in the cancel check during a short database outage · Issue #119 · NL-BioImaging/biomero · GitHub
[go: up one dir, main page]

Skip to content

Retry the event-store read in the cancel check during a short database outage #119

Open

Description

@TorecLuik

Problem

When the event store is unreachable for a moment, the cancel check fails the whole workflow. raise_if_cancel_requested (biomero/eventsourcing.py) reads the workflow from the event store and raises on the first database error. The NL-BIOMERO 1.8.0-beta.15 local acceptance hit this (plan row 12, fault (a)): stopping the database during a run failed the workflow in this check, although the workflow could resume once the database returned.

Proposal

Retry the reads in the cancel and claim checks (raise_if_cancel_requested, is_workflow_finished) with one connection-aware retry: roll back the session before each attempt, back off with the existing biomero.database helpers, log each attempt at WARNING, and raise only when the retry budget is spent. Do not swallow the error: that would defeat cancel.

A test stubs repository.get to raise OperationalError twice and then succeed, and asserts that the check returns after logged retries. Today it raises on the first error.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0