Skip to content

[checkpoint-sync] Network failure during checkpoint download may lead to a clean start for the pipeline #5613

@abhizer

Description

@abhizer

The pipeline behavior on network failure is determined by the config fail_if_no_checkpoint, which does more than it is supposed to. If true, the pipeline crashes on both i) if no remote checkpoint exists, ii) if there is a network failure during download.
If false however, the pipeline starts fresh on both the following cases:

  • If no remote checkpoint exists. intended behavior
  • If there is a failure when downloading the checkpoint.

Potential solutions

  1. Always fail if there is a network failure when downloading a checkpoint.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinghigh priorityTask should be tackled first, added in the current sprint if necessary

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions