Incidents
How UptimeHunt detects failures, opens incidents, delivers alerts, and resolves incidents when services recover.
Incidents
An incident represents a period during which a service is considered to be in a degraded or down state. Incidents are the primary unit of accountability for outages — they have a start time, an optional resolution time, and a history of associated check results.
How an incident is opened
UptimeHunt uses an alert engine that continuously evaluates incoming check results against a service's expectations. Rather than counting raw failures, the engine scores each probe by its own recent behaviour, down-weights probes that look broken across many unrelated checks, and clusters the failing probes by geography and network to classify the scope of the problem — a single location, a region, an ISP, or a global outage.
A single confidence value then gates everything. An incident opens only when that confidence clears a threshold and the failure persists across enough consecutive confirmation cycles for the detected scope (a broad, global failure is confirmed faster than a narrow one). Failures below the threshold — an isolated probe blip or a single transient error — are recorded in the probe logs but do not open an incident or fire an alert.
This is why a single probe failing from one location, while every other probe reports success, is treated very differently from all probes failing at once: the former rarely clears the confidence gate, the latter does so quickly.
Incident severity
Each incident records the service state that triggered it, derived from which expectations failed and how the failing probes were clustered:
- Down — a global-scope failure: the service is unreachable or returned a hard error from across the probe fleet.
- Degraded — a narrower-scope problem (a single region or ISP), or a failure of only soft assertions (for example, response time exceeded its threshold while the service still responded).
Whether an incident is still open or has closed is tracked separately by its resolution time — it is not a third state.
Alerts and notifications
When an incident opens, UptimeHunt fires an alert through every enabled integration (Slack, PagerDuty, email, webhook, etc.). The alert message includes the service name, incident type (Down/Degraded), affected probe locations, and a link to the service details page.
When the incident resolves, a recovery notification is sent through the same integrations.
Acknowledging an incident
On interactive channels (Slack, Discord, Telegram, Mattermost), the alert message includes Acknowledge and Resolve buttons. Tapping Acknowledge records that someone is aware of the incident without closing it. Tapping Resolve closes the incident and sends a recovery notification.
Resolving an incident
An incident closes when:
- The alert engine receives a passing check result that crosses the recovery threshold (automatic resolution).
- A team member manually resolves it via the UI or an interactive notification button.
After resolution, if the service fails again, a new incident is opened.
Viewing incidents
The Incidents list (accessible from the main navigation) shows all current and past incidents for your organization's services, with start/end times, duration, and the triggering service. Individual service details pages also show per-service incident history.
Related
- Expectations — the assertions that define failure
- Alerting & Integrations — where incident notifications go
- Service Details — per-service check history and incidents
- API: Services — programmatic access to incidents