Alerts and on call
When something your organization runs on Ronja breaks, Ronja raises one alert for it, however many times it fails. The alert is open exactly while the thing is broken, and it ends by itself when the thing works again, is turned off or is deleted, or, for a workflow or an automation no schedule runs, once its last failure is 7 days old with no run since. Ronja checks everything every few minutes, so an alert opens even if a failure was missed at the moment it happened.
Alerts in the sidebar lists them, with a count of the open ones. Its Alerts tab shows Open, Acknowledged and Ended alerts, the newest 500 of each; its Settings tab holds the policies that decide who is emailed.
What raises an alert
Section titled “What raises an alert”Six kinds of things raise alerts. Each is broken by the same rule as its own status on screen, so an alert and a badge never disagree.
| Kind | Broken while | The alert ends when |
|---|---|---|
| Workflows | The latest counted run of a live workflow failed, and that failure was in the last 7 days | A run succeeds, whoever started it, or its last failure is 7 days old |
| Automations | The latest finished run of an automation that is switched on failed. For an automation that runs on an email, webhook, table, event or mailbox trigger rather than a schedule, only while that failure was in the last 7 days | A run succeeds, or, for one no schedule runs, its last failure is 7 days old |
| Table health checks | A check that is switched on failed at its latest background run, at either severity | The check passes again |
| Table updates | A table’s updates have failed for 30 minutes while it still shows earlier data | The table updates again |
| Connections | The connection shows Re-auth needed or Stalled | It runs again, within five minutes of being fixed |
| Secrets | Its login has failed three times in a row, its service has refused three new sessions each within an hour of the last, or its sign-in needs re-authorizing, a connected mailbox’s included | It logs in or is re-authorized |
What counts for a workflow. Only runs since the workflow went live count: a draft’s test runs never raise an alert. These runs are looked past, so they neither raise an alert nor end one:
- a run someone cancelled;
- a secret’s login workflow run, which the secret’s own alert covers;
- a run an automation started that the daily AI safety limit paused;
- a failed run an automation started, which the automation’s alert covers. If someone resumes it after the automation’s run has closed and it fails again, it counts for the workflow.
What counts for an automation. A run still going, Cancelled, Skipped, Sender refused and the two Paused entries are looked past. A timeout counts as a failure, and so does a run Ronja lost track of before it recorded an outcome, as when Ronja restarted under it: its alert says so, and that its work may not have finished.
What counts for a health check. A background run is a table rebuild or a freshness check’s re-check, and so is the first run of a check written from the CLI or the API. A check Ronja writes or runs in a chat opens no alert: she tells you its result there.
The 7-day window keeps a workflow, or an automation no schedule runs, that failed weeks ago and has not run since from raising an alert, and ends its alert once the last failure is 7 days old. A scheduled automation has no window: it is in use by definition, so its alert stays open until a run succeeds.
Archived, trashed, turned off or deleted. An archived or trashed workflow, an automation switched off or deleted, a check silenced or removed, an archived or trashed table, a paused or deleted connection, and a deleted or archived secret raise no alert, and an open alert for one ends. So does every alert for something in a feature in the trash. Nobody is emailed that it works again: whoever switched it off is the one who acted. Restoring it opens its alert again if it is still broken.
Who is told
Section titled “Who is told”Every alert follows a policy: the one that holds the alert’s feature, or Default for every feature no policy holds. With nothing set up, Default has one step: the people responsible and your organization’s Admins, at once.
| Kind | The people responsible |
|---|---|
| Workflow | The maintainers of its feature and the person who started the failed run; its author too when nobody maintains the feature |
| Automation | Its creator |
| Table health check, table updates | The maintainers of the table’s feature |
| Connection | Nobody beyond the Admins, since only an Admin can re-authorize or reset a connection; the alert says so |
| Secret | The person who made it. A connected mailbox’s secret is the Admins’ alone |
- A person who has left your organization is not told, and an App User is never emailed.
- A private feature’s alerts stay with its owner. The Admins and access groups in a step narrow to the feature’s owner, or to every Admin once the owner has left your organization. A person or email address a step names is still emailed. For a table, connection or secret in it, the owner is told even if they are not an Admin, and can see and acknowledge the alert.
- An alert about something in a private feature names nothing outside Ronja. Its email, its bell notification and its Acknowledge page say only what kind of thing is failing, such as A workflow in a private feature is failing, with no name and no reason, for everyone it reaches, the owner included. Open it on the Alerts tab to see what happened.
- Somebody is always told. If no step reaches anyone, the Admins are emailed instead, and the email says why. If even that reaches nobody, the alert says so on the Alerts tab.
- Everything due to you within the same minute arrives in one email, with each alert listed under its own Acknowledge button. The subject names your organization, as in [Ronja · Midas] Alert: Workflow “Monthly churn” is failing.
- Ronja support gets a copy of each alert’s first email (not of a test alert), without the button.
Each email ends by naming the policy and step that sent it, with a Change who gets these alerts link to that policy. An email that could not be sent is retried three times, then shown on the alert as Could not email with the address.
Policies and steps
Section titled “Policies and steps”An Admin sets up policies under Alerts › Settings. Every other member except an App User can open the same pages to read them.
The Settings tab lists every policy as one line with its whole flow, Default last. Click New policy to add one, or Edit on a policy to change it:
- Give it a Policy name.
- Under the features section, click Add feature for each feature whose alerts should follow it. A feature is in at most one policy.
- In each step, add who is emailed: a person, an access group, Responsible people, Admins, or any email address, which is labelled email.
- Set Then wait on every step but the last: 5 to 1440 minutes before the next step, if nobody has acknowledged the alert by then. A policy’s waits add up to less than 24 hours.
- Click Save changes.
Ronja emails each step in order until someone acknowledges the alert, and skips a step that reaches nobody. Who a step reaches is settled when the step falls due, so an edit changes only the steps not yet sent. A policy covers every kind of alert from its features alike: it can’t send automations to one person and workflows to another.
What raises an alert lists the six kinds with a switch each. Turned off, an alert of that kind is still recorded on the Alerts tab, but nobody is emailed about it.
Send a test alert sends one through the saved policy, with you as its responsible person, so you can see the email arrive and the button work. It ends when someone acknowledges it, or after an hour. A policy needs at least one feature before you can test it, and your organization can send 5 test alerts an hour.
Delete policy moves its features back to Default. Default itself can’t be deleted.
An agent can keep the same setup in code; see Keep the alert setup in code.
Acknowledge
Section titled “Acknowledge”Acknowledge means “I’m on it”. The first person to acknowledge an alert stops its steps for everyone, and from then on only they get it in the morning summary. A second acknowledgement changes nothing.
- From an email: click Acknowledge under the alert. The page that opens shows the alert’s title (for something in a private feature, only what kind of thing it is) and its state now, and changes nothing until you press Acknowledge on it, so a mail scanner opening the link acknowledges nothing. It needs no sign-in and acknowledges as the address the email went to. The link works once, for 7 days after the email. If a newer email about the same alert reached you, its link is the one that works.
- From Ronja: open the alert on the Alerts tab and click Acknowledge. Anyone who can see an alert can acknowledge it.
- From the bell: each person told about an alert gets one notification for it under Notifications, at the bottom of the sidebar, however long it lasts. It opens the alert on the Alerts tab.
There is no Close
Section titled “There is no Close”An alert can’t be closed or dismissed. It ends when the thing it names works again, is turned off or is deleted, so the Alerts tab never says nothing is wrong while something is broken. To quiet an alert for something that will stay broken, fix it, turn it off or delete it, or switch its kind off under What raises an alert in its policy.
The morning summary
Section titled “The morning summary”At 08:00 in your organization’s reporting timezone, or UTC when none is set, each person gets one email of the alerts still open that they were told about, headed 3 alerts are still broken:
- an alert nobody has acknowledged is in the summary of everyone its steps reached;
- an acknowledged one is only in the summary of the person who acknowledged it;
- an alert you were emailed about in the 12 hours before 08:00 is left out of that day’s summary;
- each alert says whether it failed again since the last email about it, or broke again after working for a while.
Nothing open, no email. An alert stays in the summary until it ends.
When it works again
Section titled “When it works again”An ended alert moves to Ended on the Alerts tab, and its people get no more email about it.
When an alert ends before its first email has gone out, as when a run fails and the next one succeeds at once, the first email still goes and says It broke at 06:00 and has worked again since 06:01. Nothing to do., so a failure is never silently swallowed. When no step reached anyone, that first email goes to the Admins, as for any alert. An alert that ended because its thing was turned off, archived, trashed or deleted sends no such email.
When something breaks again within 24 hours of its alert ending, the same alert opens again and shows Broke again. If nobody had acknowledged it, the steps already emailed are not sent a second time. If someone had, it opens unacknowledged and its steps start again from the first, saying it broke again, so whoever was handling it hears about it.
The failing badge on workflows
Section titled “The failing badge on workflows”The Workflows list marks a failing workflow Failing · since the time it started failing, by the same rule as its alert; hover it to see how many runs in a row failed. The Automations list shows its own failing status, by the automation alert’s rule.
Keep the alert setup in code
Section titled “Keep the alert setup in code”The whole setup is one JSON document, read with GET /api/v2/alerts/config and replaced whole with PUT /api/v2/alerts/config; add ?dryRun=true to see what would change and who each step reaches, without saving. A save carries the version it read, so it is refused if someone changed the setup in the meantime. See Alerts in the API reference.