The pipeline behavior on network failure is determined by the config fail_if_no_checkpoint, which does more than it is supposed to. If true, the pipeline crashes on both i) if no remote checkpoint exists, ii) if there is a network failure during download.
If false however, the pipeline starts fresh on both the following cases:
- If no remote checkpoint exists. intended behavior
- If there is a failure when downloading the checkpoint.
Potential solutions
- Always fail if there is a network failure when downloading a checkpoint.