Webhooks

Delivery, Retries & Ordering

What counts as a successful webhook delivery, the request time limits, the ten-attempt retry schedule over about 24 hours, per-endpoint ordering, at-least-once delivery, failing and recovered alerts, and how long the delivery log keeps each delivery.

This page explains exactly what DevStride does with an event after it happens, so you can build a receiver that copes with slow periods, outages and duplicates.

The life of a delivery

When a change happens, DevStride queues one delivery of its event for every enabled endpoint subscribed to that event type. Each delivery moves through these statuses, all visible in the delivery log:

StatusMeansWhat happens next
pendingQueued, or waiting for its next retry.DevStride sends it when its turn comes.
succeededYour endpoint answered 2xx.Nothing: it is done.
failedAll 10 attempts failed.Nothing, until you replay it.
deadParked because you disabled the endpoint.Nothing, until you enable the endpoint and replay it.

Endpoints that are disabled when a change happens get no delivery for it at all.

What counts as success

A delivery succeeds when your endpoint answers with any 2xx status (200, 201, 202, 204 …) within the time limits below. The response body does not matter, but DevStride reads at most 64 KB of it: a response larger than 64 KB counts as a failure, even with a 2xx status. Answer with an empty body or a short one.

Everything else is a failed attempt, and is retried:

OutcomeExample
Any status that is not 2xx400, 401, 404, 410, 413, 429, 500, 503
A redirect DevStride does not follow301, 302 or 303 (following them would turn the POST into a GET)
No answer in timeNothing within 10 seconds of connecting, or the whole request (DNS lookup, connection, any redirects and the response) taking more than 15 seconds
A connection or TLS failureConnection refused or reset; a certificate that is expired, self-signed or for another name
A refused addressThe host now resolves to a private, loopback, link-local or cloud-metadata address

DevStride does not treat any status specially: a 400 or 410 is retried exactly like a 500. So never answer with an error for an event you simply choose not to process; answer 2xx and ignore it.

Redirects

DevStride follows 307 and 308 redirects, which keep the request a POST with the same body, for up to 3 hops. Every hop must be https:// and is checked against the same address rules as your endpoint URL; the 15-second limit covers all of them. Prefer registering the final URL.

The retry schedule

After a failed attempt, DevStride waits before trying again. Each wait is a random time between zero and the limit for that attempt (the "jitter" stops thousands of retries hitting a recovering server at the same moment):

After failed attemptNext attempt withinLatest time since the first attempt
11 minute1 minute
25 minutes6 minutes
315 minutes21 minutes
430 minutes51 minutes
51 hour1 hour 51 minutes
62 hours3 hours 51 minutes
74 hours7 hours 51 minutes
88 hours15 hours 51 minutes
98 hours23 hours 51 minutes
10(none)The delivery becomes failed

So a delivery gets 10 attempts over at most about 24 hours, and on average about half that. Because the waits are random, a retry can also come much sooner than its limit. After the tenth failure the delivery is failed, and it stays in the delivery log, ready to replay, until at least 30 days after it was queued.

The Next attempt column of the delivery log shows when a pending delivery is due.

Ordering

Each endpoint has its own queue, and deliveries to one endpoint go out one at a time, in the order they were queued. DevStride does not send the next one until the current one has succeeded or used up its attempts.

That has two consequences worth designing for:

  • A failing delivery holds back everything behind it. While one delivery waits for its retry, newer deliveries to the same endpoint wait too, for up to about a day if the endpoint keeps failing. Once it answers 2xx again, the backlog drains in order. If one specific event makes your handler fail (a bug, a payload you cannot parse), that one event blocks the queue: answer 2xx, store the event and deal with it separately.
  • A slow endpoint is a slow queue. Because only one delivery is in flight at a time, a receiver that takes 5 seconds per event can take at most 12 events a minute. Answer first, process afterwards.

What ordering does not promise:

  • Nothing across endpoints. Two endpoints receive the same event independently; one may be hours behind the other.
  • Not exactly the order things happened. The queue follows the order DevStride queued the events, which almost always matches the order of the changes, but two changes milliseconds apart can be queued the other way round. When the exact order matters, compare occurredAt, or read the current state from the API rather than trusting the last event you received.

At-least-once delivery

DevStride delivers every event at least once, which means occasionally more than once: for example, when your endpoint processed a request but the connection dropped before DevStride saw the answer, or when a replay resends a delivery your endpoint actually handled. Duplicates carry the same event id, in the body and in X-DevStride-Event-Id.

Make your handler idempotent: record each event id you process (a unique key in a table works well) and skip one you have seen. Keep ids for at least 30 days, the period over which a delivery can still be replayed. See Deduplicate by event id.

Failing and recovered alerts

DevStride watches each endpoint's health for you:

  • An endpoint becomes failing when one delivery has failed 3 attempts in a row. One bad response is noise; three is an outage.
  • It becomes healthy again with its next successful delivery.

On each of those two changes, and only then, DevStride emails every active member who holds the Manage Webhooks permission (Owners included):

WhenSubjectWhat it says
It starts failingWebhook endpoint failing: hooks.example.comThe full URL, since when (in UTC), and the last HTTP status or the reason no response arrived. Deliveries are still queued and retried for about a day; nothing is lost yet.
It recoversWebhook endpoint recovered: hooks.example.comIt is accepting webhooks again, since when it had been failing, and that the queued deliveries are going out in order.

Each email has an Open Webhooks settings button. It names the host and status only: never a payload or a secret. You get one email per change, not one per failed attempt.

While an enabled endpoint is failing, the Webhooks settings page shows a red banner, "hooks.example.com has been failing since 2 hours ago; last response 503.", with a Replay failed button, and the endpoint's row shows Failing since.

How long the delivery log keeps deliveries

Every delivery, with the exact payload that was sent and the last failed response, stays in the delivery log for at least 30 days after it was queued. After that, finished deliveries (succeeded, failed and dead) are removed. A pending delivery is never removed. A delivery can be replayed only while it is still in the log.

Deleting an endpoint deletes its delivery log with it. Erasing a requester or redacting a comment also removes the stored deliveries that carried their words; see Privacy.

What is not delivered

  • Events that happened while an endpoint was disabled. They are never queued for it, so there is nothing to replay. Use the API to catch up on that period.
  • Changes that load or correct history, such as items imported with their original creation dates.
  • Anything to an endpoint that was deleted. Its queued deliveries are dropped with it.