The workflow works when you build it. The question is what happens on the Tuesday six weeks later when an API changes a field name and nobody is watching.
Two kinds of failure
Transient — resolves by itself. Network timeout, 503, 429, brief outage. Retry these.
Permanent — will fail identically forever. 400 malformed, 401 bad credentials, 404 missing, validation errors, a field that no longer exists. Do not retry. Fail, log, and alert.
Retrying a permanent failure wastes quota, delays discovery, and in the worst case looks like an attack.
Retry properly
- Exponential backoff — 1s, 2s, 4s, 8s, with a ceiling.
- Jitter — randomise slightly, so parallel runs do not retry in lockstep.
- A limit — three to five attempts, then stop.
- Respect
Retry-Afterwhen the API sends it.
Beware repeating side effects
The dangerous case: a request times out, you retry, and the original had actually succeeded. Now the refund is issued twice.
Make retried operations safe:
- Use the API's idempotency key if it offers one
- Check whether the effect already happened before repeating it
- Keep a record of completed steps so a resumed run skips them
Retries on a read are free. Retries on a write need thought.
Monitor the absence of things
Because of that, the most valuable alert is not "an error occurred". It is "this has not run when it should have".
Track:
- Did it run? A heartbeat. If the daily job has not reported in 25 hours, alert.
- Did it do anything? Zero items processed for three days running is either a real change or a broken trigger. Both are worth knowing.
- Error rate. A rise from 1% to 15% matters even while it still mostly works.
- Duration. A job that took 30 seconds and now takes 8 minutes is telling you something before it fails.
Log enough to reconstruct
For every run: when it started, what triggered it, the input, each decision and why, each action attempted, the outcome, and the total duration.
The test is simple — when someone asks "why did this customer get that email in March?", can you answer from the logs alone? If not, add fields until you can. You will need it, and the moment you need it is the moment it is too late to start collecting.
Fail loudly, degrade gracefully
When something is genuinely wrong: stop, alert a person, and leave the system in a state someone can inspect. Do not guess past the failure, and do not swallow the error to keep the run alive. A workflow that halts honestly is a small problem. One that continues on bad data is a large one discovered late.