Webhooks, rate limits, timeouts, retries, and idempotency

Connecting two software systems is only the beginning. The systems must also communicate reliably when responses are delayed, requests are repeated, traffic becomes excessive, or one side temporarily fails.

This lesson introduces several concepts that govern how integrations behave under real-world conditions:

• Webhooks
• Rate limits
• Timeouts
• Retries
• Idempotency

Together, these ideas help systems communicate without waiting forever, overwhelming one another, or accidentally performing the same action twice.

The problem with repeatedly asking

Imagine that the coffee app asks a payment provider to charge a customer’s card.

Sometimes the provider can return the final result immediately, but in other cases the payment may require additional processing before its final status is known.

The coffee backend could repeatedly ask: “Is the payment finished yet? Is it finished now? How about now?”

This repeated checking is called polling.

Polling can be appropriate in some situations, but asking too frequently creates unnecessary traffic and work for both systems.

Another option is for the payment provider to contact the coffee company when something changes.

That is the purpose of a webhook.

What is a webhook?

A webhookConcept · lights on your mapwebhookA way for one system to automatically notify another when an event occurs: you provide a URL, and the provider sends an HTTP request to it when the event happens. Polling asks repeatedly; a webhook means the other system calls you. Webhooks must be verified and may arrive more than once. is a way for one system to automatically notify another system that an event has occurred.

The coffee company provides the payment provider with a URL such as: https://api.coffeeapp.com/webhooks/payments

When the payment succeeds or fails, the provider sends an HTTP request to that URL.

A simplified webhook request might look like:

The coffee backend receives the message and updates the order.

A useful analogy is: polling means repeatedly calling to ask whether something happened. A webhook means asking the other system to call you when it happens.

The client and server roles reverse

When the coffee backend initially requests the payment, it is the client and the payment provider is the server.

When the provider later sends a webhook, their roles reverse:

• The payment provider becomes the client.
• The coffee backend becomes the server.
• The coffee company exposes an endpoint that receives the notification.

This reinforces the principle that client and server describe roles within a particular interaction, and a server in one conversation can become a client in the next.

Webhooks must be verified

A webhook endpoint is designed to receive requests from another system. But the backend should not trust every request merely because it arrives at the correct URL.

An attacker could attempt to send a fake message claiming: { "event": "payment.succeeded" }

The coffee backend therefore needs a way to verify that the webhook came from the payment provider.

Providers commonly attach a cryptographic signature to webhook requests, and the receiving backend verifies that signature using a secret shared with the provider.

The exact cryptography will be covered in the security module, so for now, remember: a webhook should be verified before the system trusts or acts on it.

Webhooks can arrive more than once

The payment provider needs to know whether the coffee backend successfully received the webhook, so if the receiving server is unavailable or returns an error, the provider may send the same webhook again later.

Even when the first delivery succeeded, network uncertainty can sometimes lead to duplicate deliveries.

The coffee backend must therefore be prepared to receive the same event more than once without issuing multiple refunds, creating duplicate orders, or sending repeated notifications.

This leads directly to the concept of idempotency, which we will return to shortly.

What is a rate limit?

A rate limitConcept · lights on your maprate limitA restriction on how much activity a client may send to a service during a period: 100 requests per minute, 1,000 calls per day. Protects providers from request loops, abuse, traffic spikes, and one customer consuming all capacity. Exceed it and the server returns 429 Too Many Requests. restricts how much activity a client is allowed to send to a service during a particular period or under a particular allowance.

A provider might allow:

• 100 requests per minute
• 10 payment attempts per second
• 1,000 API calls per day
• A certain amount of processing per account
• Different limits for different subscription plans

Rate limits protect systems from:

• Accidental request loops
• Overloaded clients
• Automated abuse
• Unexpected traffic spikes
• One customer consuming all available capacity
• Excessive infrastructure costs

When a client exceeds a rate limit, the server commonly returns: 429 Too Many Requests

This tells the client that it is making too many requests and should slow down.

Responding to a rate limit

A well-behaved client should not immediately repeat the same request as quickly as possible because that would increase the very traffic causing the problem.

The server may include information indicating when the client should try again, sometimes through a header such as: Retry-After: 30

This might mean: wait 30 seconds before retrying.

The client may also reduce how frequently it sends requests, combine several operations, or request higher limits from the provider.

Rate limits are part of an API’s contract and should be explained in its documentation.

What is a timeout?

A timeoutConcept · lights on your maptimeoutA limit on how long a system will wait for an operation to complete. Without one, a client can wait indefinitely on a slow or unresponsive dependency. Crucially: a timeout means no answer arrived in time; it does not prove the other side did nothing. is a limit on how long a system will wait for an operation to complete.

Suppose the coffee backend sends a request to the payment provider, and normally the response arrives within a second or two.

But what if no answer arrives?

Without a timeout, the backend might wait indefinitely while the customer sees a spinner that never ends.

Instead, the application sets a deadline. If the response does not arrive before that deadline, the client stops waiting and treats the operation as timed out.

A timeout protects the client from being trapped indefinitely by a slow or unresponsive dependency.

A timeout does not reveal what happened

This is one of the most important ideas in the lesson: a timeout means the client did not receive an answer in time. It does not prove that the server did nothing.

The payment provider may have successfully charged the card, but its response may have been delayed or lost, so from the coffee backend’s perspective the result is uncertain.

It knows that it did not receive confirmation, but it does not necessarily know whether the payment succeeded or failed.

This uncertainty makes retries dangerous.

What is a retry?

A retryConcept · lights on your mapretryAnother attempt at an operation after the previous attempt failed or produced no usable response. Useful because many failures are temporary. But retries must be limited, delayed (backoff), and safe to repeat, which is where idempotency comes in. is another attempt to perform an operation after the previous attempt failed or produced no usable response.

Retries are useful because many technical failures are temporary.

A request may fail because:

• The network briefly disconnected
• A server restarted
• A dependency was temporarily overloaded
• A request was rate-limited
• A response was lost
• A short-lived infrastructure problem occurred

Trying again later may succeed.

But not every failure should be retried. An invalid password will not become correct merely because it is submitted repeatedly, and a malformed request must be fixed rather than retried unchanged.

Systems should decide which failures are likely to be temporary.

Retry carefully, not endlessly

Retries can make a system more reliable, but uncontrolled retries can make a failure worse.

Imagine that a payment service becomes overloaded and thousands of clients immediately retry every failed request, and the additional traffic creates even more load, making recovery harder.

Systems therefore commonly use a strategy called backoff, which means waiting before trying again, with the delay often increasing after each failure.

A simplified pattern might be:

• First retry after one second
• Second retry after two seconds
• Third retry after four seconds
• Stop after a limited number of attempts

Systems may also add small random variations to those delays so that every client does not retry at exactly the same moment. This is commonly called jitter.

You only need to recognize these terms for now. The main principle is: retries should be limited, delayed, and used only when another attempt is reasonable.

The danger of repeating an action

Suppose the coffee backend sends this payment request: POST /charges — { "order_id": 1042, "amount": 1450 }

The provider charges the card successfully, but the response never reaches the coffee backend.

The request times out.

If the backend sends the same request again without protection, the provider might interpret it as a second payment and charge the customer twice.

The retry solved the missing-response problem but created a duplicate-action problem.

This is why payment systems need idempotency.

What is idempotency?

IdempotencyConcept · lights on your mapidempotencyDesigning an operation so that repeating it has the same intended effect as performing it once. The client attaches a unique idempotency key to the attempt; if the same key arrives again, the server recognizes a retry and returns the original result instead of acting twice. What makes retries safe. means that repeating an operation has the same intended effect as performing it once.

For a payment request, that means several copies of the same logical request should create only one charge.

The client can attach a unique idempotency key to the payment attempt:

The provider records that it already processed the key: order-1042-payment-1

If the same request arrives again with the same key, the provider recognizes it as a retry of the original attempt.

Instead of creating another charge, it can return the result of the first operation.

The simplified idea is: same logical request, same identifier, one effect.

Idempotency depends on identity

The idempotency key must identify one logical operation.

If the customer deliberately places a second separate order, the new payment attempt should use a different key, or the provider may mistake the new order for a duplicate of the first.

For example, order-1042-payment-1 and order-1043-payment-1 represent separate payment operations.

The client should reuse a key only when retrying the same intended action.

Some HTTP methods are naturally idempotent

Certain HTTP methods are intended to be idempotent.

A GET request should retrieve information without creating additional effects, so repeating it should not create new orders or charges.

A DELETE request to remove the same resource several times should leave the resource removed.

A PUT request commonly replaces a resource with a specified state, so repeating the same replacement should lead to the same result.

A POST request often creates something new and is not naturally idempotent, and repeating it may create duplicates unless the API adds protection such as an idempotency key.

These are design expectations, and systems still need to implement them correctly.

Webhooks also need idempotency

Imagine that the payment provider delivers the following webhook twice: { "event_id": "evt_5519", "event": "payment.succeeded", "order_id": 1042 }

The coffee backend can record the unique event ID: evt_5519

If the same event arrives again, the backend sees that it has already processed it and does not repeat the action.

Without this protection, a duplicate webhook could send two confirmation messages or update the same order repeatedly.

Reliable integrations assume that messages may be delayed, lost, reordered, or duplicated.

What is build versus buy?

These integration concerns lead to a recurring business and technical decision: build versus buy.

BuildConcept · lights on your mapbuildCreating and operating a capability internally. You gain control and pay for it in responsibility; every specialized concern becomes yours. The alternative is to buy. means creating and operating the capability internally.

BuyConcept · lights on your mapbuyUsing another company’s product or service instead of building one. You gain speed and expertise and pay in fees, dependency, and reduced control. Buying shifts responsibilities; it does not remove them. means using a product or service provided by another company.

If the coffee company builds payment processing itself, it gains greater control but assumes responsibility for:

• Bank and card-network integrations
• Security, compliance, and fraud detection
• Rate limiting, timeouts, retries, and idempotency
• Monitoring, outages, and on-call support

If it uses a payment provider instead, that machinery becomes the provider’s job. In exchange the company accepts usage fees, a third-party dependency, integration work, and the provider’s contract and pricing.

Buying does not remove responsibility

Using a provider shifts the work; it does not erase it. What remains is the client’s side of the same concerns: the coffee company still sets timeouts on its own requests and retries them safely, deduplicates webhook deliveries, watches the provider’s health from the outside, and faces its own customers when the provider fails.

Build versus buy is therefore not a choice between responsibility and no responsibility. It is a decision about which responsibilities the company owns directly and which it depends on a provider to own.

Seeing the complete integration flow

Imagine that Erik places a coffee order.

The coffee backend sends a payment request with an idempotency key.

If the provider responds quickly, the backend continues processing the order.

If the request times out, the backend may retry later using the same idempotency key, and the provider recognizes that the request belongs to the same payment attempt and avoids creating a duplicate charge.

If the payment’s final status is delayed, the provider later sends a signed webhook.

The coffee backend verifies the webhook, checks whether the event has already been processed, updates the order, and sends Erik a confirmation.

Meanwhile, both systems respect rate limits and use timeouts so that neither waits forever or overwhelms the other.

This is what reliable machine-to-machine communication looks like in practice.

The mental model to remember

A webhook allows one system to notify another system when an event occurs.

Polling means repeatedly asking whether something has changed.

A rate limit restricts how much activity a client may send to a service.

A timeout limits how long a system waits for an operation.

A retry makes another attempt after a temporary failure or uncertain result.

Backoff increases the delay between retries.

Idempotency ensures that repeating the same logical operation does not create repeated effects.

An idempotency key identifies one logical attempt so the server can recognize retries.

Build versus buy is the decision between creating a capability internally and relying on an outside provider.

You should now understand how integrations protect themselves from delays, duplicate requests, excess traffic, and temporary failures.