Webhook endpoints: separate signature verification, retries and idempotency

EDITORIAL PRACTICE

Why separate them: verification, retries, and idempotency are different concerns

Problems at a webhook endpoint often originate in different layers. Signature verification is a question of authenticity: did this request really come from a legitimate sender? Retries are a question of reachability: how do we avoid losing a notification that failed to arrive because of a network issue or a temporary outage? Idempotency is a question of consistency: what happens to the processing result when the same notification arrives several times? Because the questions differ, the place where each countermeasure belongs — and how a failure shows up — should also differ.

In real implementations, however, these three concerns often melt into a single endpoint handler. It becomes hard to tell, even from the logs, whether verification failed, whether a heavy process missed a timeout, or whether a redelivered notification was processed twice. Incident response then spends its time just isolating where and what happened, and ad-hoc fixes tend to create the next problem. Giving each concern its own name and boundary is, in itself, the first step of the design.

Separation also keeps the unit of change small. If the signature scheme changes, only the verification layer needs review. If assumptions about retry intervals change, only the response behavior needs adjustment. Extending the retention period for idempotency records can be decided without disturbing the other two. The following sections walk through what to hold onto in each layer, following the official documentation.

This separation does not mean introducing some new mechanism. Even with an existing endpoint, much of it can be achieved by fixing the order of processing — verification, a minimal record, the response, then downstream processing — and by keeping the unit of recording separate per layer. What matters is making visible, through code placement and log field names, which process belongs to which concern. When the boundaries are visible, later reviews and handovers become much less confusing.

Sources: [1][2]

EDITORIAL PRACTICE

Principles of signature verification: HMAC, constant-time comparison, and secret storage

The role of signature verification is to confirm that an incoming request comes from a legitimate sender who knows the shared secret. According to GitHub's official documentation, GitHub uses an HMAC hex digest to compute the hash, and the signature is sent in the X-Hub-Signature-256 header, starting with sha256=. The receiver performs the same computation with the shared secret on its own side and checks whether the result matches the signature that was sent. What matters here is that sender and receiver compute over the same raw body with the same secret.

How the comparison is done also deserves care. The same documentation says never to use a plain == operator, and instead to consider a method such as secure_compare or crypto.timingSafeEqual, which performs a constant-time string comparison. Ordinary string comparison can take slightly longer when more of the prefix matches, which in theory leaves room for the signature to be inferred from timing differences. Constant-time comparison is the standard way to close that gap, and the expectation is to use the dedicated functions provided by the language or framework.

The other principles concern how the secret is stored and where verification runs. The shared secret should not be written directly into source code; it belongs in environment variables or a secrets management mechanism, in a state where it can be rotated if a leak is suspected. Verification itself should run at the entrance of the endpoint, before any other processing. Structuring the endpoint so that a request which fails verification never proceeds to business processing keeps the blast radius closed at the door.

Practical pitfalls include mixing up the shared secrets prepared for different environments, and reusing the same secret across multiple senders. Keeping a separate secret per sender, and making explicit which secret each endpoint refers to for verification, avoids incidents where rotating one secret affects the others. During transitions of signature schemes, the header format may also differ from period to period. Deciding on the endpoint side which format is treated as canonical, and how anything else is handled, and writing that down as an operational rule, keeps confusion down.

Sources: [1]

EDITORIAL PRACTICE

Typical causes of verification failure: body modification and conversion pitfalls

A failed signature verification does not necessarily mean an attack or abuse. A common practical cause is that the body being verified changes somewhere inside your own processing. GitHub's official documentation says to make sure that the payload and headers are not modified before verification. Stripe's official documentation likewise states that the request body used for verification must be the exact UTF-8 string Stripe sent, without any modifications. In other words, verification applies to the string exactly as received; data that has been reformatted or transformed will no longer match.

A typical pitfall is the default behavior of web frameworks. If the request body is parsed once as JSON and then reconstructed from the object before verification, whitespace, key order, and number formatting may differ from the original string, and the signature will not match. The same applies when a proxy or gateway rewrites headers or appends something to the body. A workable countermeasure is to keep a dedicated path that preserves the raw body before parsing for verification purposes, and to proceed to business parsing only after verification has passed.

For diagnosis, how verification failures are recorded matters as well. Recording not only the fact of failure, but also clues such as which header was missing and what the body length and endpoint path were — without storing the entire body — makes causes easier to identify. If failure reasons are counted at a granularity such as signature mismatch, missing signature header, or malformed input, rather than being bundled into a single alert, misconfiguration becomes easier to distinguish from external probing.

When verification failures persist, it also helps to have a small self-contained means of checking on your own side. For example, recording only the length and hash of the raw body received — without retaining the body itself — later provides material for narrowing down whether the cause was a modification in transit or a mix-up of secrets. Being able to quickly split the cause between your own conversion and the sender's configuration also shortens the back-and-forth with the other party. Here again, keeping the record self-contained within the verification layer is what speeds up diagnosis.

How failures are recorded should itself be designed as part of verification. For example, if requests that failed verification can be traced in time order — at which endpoint, when, and under which failure category — they can be cross-checked against events such as a change in the sender's configuration or your own deployment. Conversely, carelessly retaining the bodies or signatures of failed requests creates a separate problem in the handling of secrets. Drawing the line that verification records are 'the minimum sufficient to identify the reason for failure' is itself a practical design question within verification.

Sources: [1][2]

EDITORIAL PRACTICE

Designing for retries: assume automatic redelivery, and return success quickly

For retries, the starting point is to estimate correctly what the receiver can and cannot control. Stripe's official documentation states that Stripe automatically retries undelivered events for up to three days. That means a notification is not lost immediately just because the receiver could not respond for a while; the sender's own mechanism provides opportunities to recover. Understanding this premise helps avoid over-engineered receiver designs that try to shoulder the sender's retries.

The receiver's responsibility is rather to tell the sender, quickly and accurately, whether delivery succeeded. The same documentation notes that redirect (3xx) responses are treated as failures, and presents the idea of returning a successful response quickly while deferring heavy processing. The basic shape of an endpoint is to perform signature verification and a minimal record synchronously, hand time-consuming business processing to a queue or a job, and then respond. If processing drags on into a timeout, the sender sees a failure and the event becomes a candidate for redelivery.

Once retries are assumed, receiving the same notification more than once is normal behavior, not an anomaly. This is exactly where the next concern, idempotency, comes in. Retry design and idempotency design are a pair: idempotency is needed because retries exist, and retries can be accepted without fear because idempotency is in place. Writing down the endpoint's response criteria — what counts as success, and what is left to redelivery as a failure — is the foundation for not hesitating in operations.

As for response speed, the sender's timeouts and retry intervals are often not published in detail, so the receiver needs a design that does not rely on guessing numbers. With the principle that synchronous processing is limited to verification and recording, the response time stays largely unchanged even as downstream processing grows. Control over retry intervals is left to the sender's policy; the receiver does not try to tune the number or spacing of retries, and instead absorbs delivered notifications on the idempotency side. Making this division of roles explicit is the essence of retry design.

Sources: [2]

EDITORIAL PRACTICE

Designing idempotency: record what is processed, and stop duplicate processing

The aim of idempotency is that no matter how many times the same notification is received, the business outcome is the same as processing it once. Stripe's official documentation asks implementers to guard against processing the same event twice. A common approach is to record processed status keyed by the unique ID included in the event. The endpoint first checks whether that ID has already been processed; if not, it processes the event and leaves a record, and if so, it skips the business processing.

How the response is returned at that point matters. The same documentation says that when your endpoint receives an event you have already processed, you should skip it and return a 200 response to stop further retries. Returning an error just because the event is a duplicate would make the sender treat it as a failure and keep redelivering. Treating duplication as a normal state on the receiver side, and closing the redelivery with a success response, is the behavior that avoids waste on both sides.

The retention period for these records also needs a guideline. Given that Stripe's automatic retries run for up to three days, a design that keeps processed records for at least longer than that window is a reasonable candidate. The order of recording and business processing is another decision point: if the process stops midway, states such as 'recorded but not processed' or 'processed but not recorded' can arise, so where exactly an event becomes 'processed' needs to be decided in advance. The more external side effects a process has, the more carefully this line should be drawn.

One more thing to decide in advance is doubling up: the processed-status record at the endpoint, plus constraints on the business-data side. Rather than relying only on the endpoint's processed management, giving the business data its own final line of defense — such as a uniqueness constraint — localizes the damage of double processing even if records become inconsistent or an unexpected path re-executes something. In particular, for processing with hard-to-undo side effects such as external notifications or updates, linking the idempotency key to a business-side key, instead of containing it within the endpoint alone, works on the safer side. Idempotency is a matter of endpoint practice, and at the same time a matter of data design.

Nor is it simply better for the retention period to be as long as possible. The longer the period, the more the cost of lookups and the volume of stored information grow. A practical approach is to set the floor based on the sender's retry window, and then add margin according to your own needs for inquiry handling and auditing. What matters is not deciding the period by feel, but being able to tie it to an external basis — the upper bound of retries — and explain it. Whether the reasoning 'why this period' can be put into words changes how consistent operational decisions about idempotency will be.

Sources: [2]

EDITORIAL PRACTICE

Monitoring and records: observe failure reasons layer by layer

A design that separates the three concerns carries directly into monitoring design. If signature verification failures, the number of redeliveries received, and the number of duplicates skipped can be counted separately, each kind of anomaly gets noticed in its own way. A sudden rise in verification failures points at secret rotation or a change in the delivery path; a rise in skipped duplicates points at the sender's retry frequency or your own response latency, and so on.

What to record can also be organized per layer. The verification layer records the failure category and the endpoint or path involved. The response layer records the response code and the time taken to respond. The retry and idempotency layer records the event ID and whether the decision was to process or to skip. There is no need to accumulate large volumes of raw bodies; keeping the minimal clues needed for later cross-checking is a manageable approach, also from the standpoint of storage and personal data.

The same applies to alerting. Notifying everything at the same severity buries the anomalies that genuinely need action. It is more realistic to grade the response: security-related signals such as a surge in verification failures go to a person quickly, while other trend changes are picked up in a daily review. Separation is not only about code structure; it is about aligning the units of observation and response as well.

It also matters not to treat observation as something only looked at during incidents. If the counts of verification failures and skipped duplicates are reviewed regularly, a state of 'always zero' itself becomes a clue that the observation mechanism may not be working. Checking, with a test delivery, that each metric moves as expected is itself a periodic inspection of the endpoint. A mechanism that counts things separately only takes root when it serves not just incident response but also day-to-day health checks.

Sources: [1][2]

EDITORIAL PRACTICE

A checklist for operations where AI agents connect downstream

Configurations are also increasing in which an AI agent or an automation system takes over downstream processing when a webhook arrives. Unlike a setup where a human simply reviews the notification, when actions such as lookups, updates, and messages chain automatically beyond the endpoint, a gap in idempotency can translate directly into the same action being executed twice. The wider the scope of automation, the more valuable it becomes to solidify the endpoint's three concerns on the safe side before optimizing for convenience.

Candidate checkpoints for a review include the following. Is signature verification performed against the raw body, using a constant-time comparison? Is the shared secret kept apart from the code, with a rotation procedure decided? Does the endpoint avoid holding heavy processing and return success quickly? Is there any path that returns a response treated as failure, such as a 3xx? Is processed status managed by event ID, with a retention period that exceeds the sender's retry window? Are already-processed duplicates answered with a 200 to stop redelivery? And can verification failures, redeliveries, and skipped duplicates be observed separately? None of these items is glamorous, but closing them one by one is what ultimately determines the reliability of the endpoint.

A realistic order of adoption is to start by inventorying where verification, response, and idempotency are mixed in the current endpoint, and by separating the metrics. After that, work through the verification path, the response split, and processed-status management in order. At each stage, checking the implementation against the descriptions in the official documentation also helps avoid over-building.

Finally, this design is not finished once a single endpoint is built; its value lies in serving as a template for adding endpoints with the same viewpoints as the number of connected services grows. Building the checklist into operations as the 'checklist for creating a new endpoint' makes it harder for the trio — verification, response, idempotency — to be dropped even when people or timing change. The details in the official documentation differ per service, but the skeleton of thinking separately is common to all of them. Fixing the skeleton first and fitting the details per service is a manageable path to standardization.

Sources: [1][2]