When Automation Breaks (It Will)

Questions this answersQuestions this answersfaq

how do I know when my automation has failed?how do I know when my automation has failed?how should failures surface?

Only if you built it to tell you. Give every step that writes something a second path out, one that turns the error into a sentence with the record in it and posts it where a person actually reads. SO-1041: order has an unfulfilled line item is a sentence; a red dot in a list nobody opens is not. Then cover the run that never happened at all, because silence is not success.Only if you built it to tell you. Give every step that writes something a second path out, one that turns the error into a sentence with the record in it and posts it where a person actually reads. SO-1041: order has an unfulfilled line item is a sentence; a red dot in a list nobody opens is not. Then cover the run that never happened at all, because silence is not success.Wire an error output on every write node into a step that renders a human-readable message carrying the record id, and point the instance-level error workflow at the same channel to catch runs that never fired: dead schedule, expired credential, host down. A stack trace or a red dot in an executions list is not monitoring. A failure your customer reports first cost more than the automation ever saved.

is it safe to run a failed automation again?is it safe to run a failed automation again?how do I make a partial run safely re-runnable?

Only if it looks before it creates. A run that did 30 of 50 writes and then died is the normal case, not the exotic one, and it is only dangerous if running it again redoes the first 30. Key every write on something stable, check whether the record already exists, and cap how much one run can do. Then anyone can rerun it, which on a bad Tuesday is the point.Only if it looks before it creates. A run that did 30 of 50 writes and then died is the normal case, not the exotic one, and it is only dangerous if running it again redoes the first 30. Key every write on something stable, check whether the record already exists, and cap how much one run can do. Then anyone can rerun it, which on a bad Tuesday is the point.Make writes idempotent: find-before-create on a stable key plus a per-run cap, so a partial run recovers by simply running again. Retry only what is worth retrying, 429 and 5xx with exponential backoff and jitter, honouring Retry-After, and never a 400, which is a data problem. The dangerous case is the one with no status code at all: a timeout may have committed the write and lost only the response.