When Zapier or Make Hit the Reliability Ceiling: Build a Durable Webhook Inbox

Why did one webhook create forty-three CRM updates?

At 2:07 AM, an alert source sends a burst. Your Zap catches every request and starts updating the same GoHighLevel contact. The first run adds a note. The second changes the opportunity stage. The third sends a text. The next forty runs arrive before the CRM reflects the first write, so every run decides the contact still needs the same action.

By sunrise, the client has duplicate notes, repeated notifications, and one very angry lead who received the same SMS more than once. Zap history shows green runs. The trigger worked. The action worked. The automation still failed.

That is the reliability ceiling people describe as “Zapier cannot handle volume” or “Make dropped my webhook.” Sometimes the platform is throttling. Sometimes the connected API is throttling. Sometimes nothing was dropped at all. The real failure is that the workflow had no durable record of what arrived, what had already been accepted, what action was authorized, or what could be replayed safely.

The fix is not always replacing Zapier or Make. I often keep them. They are good at visible routing, account-owner handoff, and integrations that operations staff need to understand. I move the parts that require transaction semantics into code and Postgres.

The open loop is what happens to the forty-fourth event after you add a queue. A queue slows work down. It does not, by itself, make the work correct.

[REAL NUMBER] Insert the count of duplicate downstream actions from one real incident, derived from CRM, email, SMS, or billing records rather than workflow-run count alone.

What does a webhook incident actually cost?

A webhook incident is rarely priced as “one failed task.”

The cost depends on the side effect. A duplicate internal Slack message is noise. A duplicate invoice, SMS, account creation, refund, inventory decrement, or lead assignment is an operational incident. The same architectural flaw can sit unnoticed for months because its normal side effect is cheap, then become expensive when someone adds one more action to the workflow.

The first cost is customer damage. Duplicate customer-facing actions feel careless because they expose the machinery. A lead may tolerate one delayed message. They do not tolerate five identical messages from a company claiming to automate its operations.

The second cost is cleanup. Your team has to determine which actions were duplicated and which events were legitimately distinct. That is difficult when the only evidence is a visual run history with transformed payloads and expired retention.

The third cost is lost work. A provider may retry because your endpoint responded slowly. Zapier may hold or throttle runs. Make may store incomplete executions depending on scenario settings. If nobody can say which events remain unprocessed, the safe response is usually to replay too much or to replay nothing.

The fourth cost is hidden state corruption. Consider an order workflow:

order webhook
  -> create CRM opportunity
  -> decrement a custom inventory table
  -> send fulfillment request
  -> notify customer

If the CRM step is idempotent but the inventory step is not, a duplicate event may look harmless in the CRM while stock is decremented twice. If the fulfillment request is accepted but the response times out, replay may create a second shipment.

A green run does not prove the business operation happened once. A red run does not prove it happened zero times.

You need an incident record that follows the business event across tools.

create table automation_incidents (
  id uuid primary key default gen_random_uuid(),
  tenant_id uuid not null,
  incident_type text not null,
  first_detected_at timestamptz not null default now(),
  resolved_at timestamptz,
  source_provider text,
  source_event_ids text[] not null default '{}',
  affected_object_keys text[] not null default '{}',
  duplicate_side_effect_count integer,
  missing_side_effect_count integer,
  customer_impact text,
  root_cause text,
  repair_notes text
);

Do not fill the duplicate counts from assumptions. Reconcile against the downstream source of truth.

A useful operational metric is not “workflow success rate.” It is accepted business events that reached their required terminal state without human correction.

That exposes a hard truth. A workflow that returned 200 and sent an email may still be incomplete if the required CRM write was throttled. A workflow that failed after creating the record may be commercially complete but operationally unacknowledged.

[REAL NUMBER] Insert the total staff time required to identify, reverse, and explain one webhook incident. Include support, engineering, account management, and any customer credit.

Why does the normal Zapier or Make pattern break?

The normal pattern is one long request path.

provider
  -> Zapier Catch Hook or Make Custom Webhook
  -> filters
  -> search record
  -> create or update
  -> send message
  -> return or finish

It breaks for five separate reasons.

The first is at-least-once delivery. Most serious webhook providers retry when they do not receive an acceptable response. They may also retry after internal delivery uncertainty. Your endpoint should assume the same business event can arrive more than once.

The second is concurrency. Two webhook runs can read the same old state before either writes the new state. Both decide to create. Search-before-create is not an atomic operation.

The third is partial success. Step four can succeed while step five fails. Replaying the whole run repeats step four unless that action is independently protected.

The fourth is ordering. A customer update can arrive before the customer-created event, or a cancellation can be processed before the activation run finishes. Arrival order is not business order.

The fifth is retention and replay. Visual platforms provide useful run histories, replay features, and error stores, but those are execution tools. They are not automatically a permanent domain event ledger with your own dedupe keys and repair rules.

A concrete example is a SaaS trial conversion.

trial.converted
  -> find company by email domain
  -> create paid account if absent
  -> add owner
  -> send onboarding

Two identical conversion events arrive within milliseconds. Both runs search before either account exists. Both create a paid account. The connected SaaS API accepted both because it has no idempotency key on that endpoint.

Adding a delay can reduce the race. It does not establish uniqueness. A later replay can still repeat it.

The correct mechanism is a unique constraint in a database that every run must cross before the irreversible action.

create table business_operations (
  id uuid primary key default gen_random_uuid(),
  tenant_id uuid not null,
  operation_type text not null,
  business_key text not null,
  status text not null default 'pending',
  result jsonb,
  created_at timestamptz not null default now(),
  completed_at timestamptz,
  unique (tenant_id, operation_type, business_key)
);

For the trial conversion, the business key could be the authoritative subscription ID. Inserting the operation is atomic. One concurrent worker wins. The other reads the existing operation.

Do not use an email address when a stable subscription ID exists. Do not use a timestamp that changes on every retry. The business key must name the effect you intend to happen once.

[DIAGRAM] Draw the race between two visual workflow runs. Both search the CRM and see no record. Both call create. Then draw the corrected flow where both first attempt the same unique database insert and only one receives permission to create.

Is Zapier throttling the same problem as data loss?

No. Throttling, delay, task blockage, and data loss are different failure classes.

Zapier's current Webhooks by Zapier rate-limit documentation says bursts can be throttled to reduce their frequency. Its Zap limits documentation currently states that instant-trigger workflows encounter 429 errors above 20,000 requests every five minutes per user. That exact limit can change, so check the current page during implementation rather than treating this article as a contract.

Zapier also documents errors such as:

Throttled by Zapier! Max ... runs per ... seconds.
Zapier has blocked this task.
Too much data received ... Zapier has blocked this task.

Its troubleshooting guide recommends identifying whether Zapier or the connected app imposed the limit, then using replay, Delay After Queue, or paid flood-protection controls where appropriate.

Those controls are useful. They answer “how do I reduce execution rate?” They do not answer:

Did I receive every provider event?
Did two runs represent the same event?
Did the downstream action already succeed before a timeout?
Can I replay only the missing business effect?
Does a newer event supersede this one?

A throttled event can be delayed and eventually succeed. A lost event may never appear in the platform. A duplicate event may produce two successful runs. A partial run may be marked failed after its irreversible action succeeded.

The first diagnostic step is to compare three counts over the same window:

provider delivery attempts
accepted ingress events
completed business operations

The provider's count may include retries. The accepted ingress count should include duplicates but classify them. The completed operation count should reflect unique intended effects.

For example:

select
  date_trunc('hour', received_at) as hour,
  count(*) as accepted_attempts,
  count(*) filter (where duplicate_of_id is not null) as duplicates,
  count(*) filter (where processing_status = 'dead_letter') as dead_letters
from webhook_inbox
where tenant_id = $1
  and received_at >= $2
  and received_at < $3
group by 1
order by 1;

Then compare that with operation completions.

If Zapier shows fewer trigger runs than your ingress accepted unique events, the dispatch path is behind or rejecting. If both match but downstream operations are lower, the problem is after dispatch. If the provider reports events your ingress never recorded, investigate network, DNS, signature, or endpoint availability.

This is why I put a thin receiver in front of Zapier for workflows that matter. It gives me an independent acceptance record before platform throttling, task limits, or workflow changes enter the path.

Does Make process webhook events in order?

Not by default.

Make's current scenario settings documentation says webhook executions are processed in parallel by default. Enabling Process data in order makes each execution wait for the previous one. The same documentation explains how incomplete executions interact with that setting, and it warns that new runs can pause until incomplete executions are resolved when ordered processing is enabled.

That setting is useful when order matters and throughput is low enough for serial execution. It is not a complete ordering model.

First, Make can preserve arrival order at its webhook queue. It cannot make the provider deliver business events in correct order. If customer.updated reaches Make before customer.created, serial processing faithfully preserves the wrong arrival order.

Second, global serial processing can create head-of-line blocking. One slow or incomplete execution delays every later event for that scenario, including events for unrelated customers.

Third, order at the scenario level may be too broad. You may need order per order ID, contact ID, or account, while allowing different objects to process concurrently.

Fourth, replay changes timing. Make's current scenario run replay runs the current scenario version with trigger data from a previous run. That is valuable for testing and backfill, but a replayed old event can reach downstream systems after newer state unless your workflow rejects stale changes.

A concrete example is a ticket lifecycle:

ticket.created at source version 1
ticket.assigned at source version 2
ticket.closed at source version 3

If the closed event arrives first, a pure arrival-order queue may close a missing ticket, then create it, then assign it. The final state becomes assigned rather than closed.

You need source versioning or a current-state fetch.

create table object_projections (
  tenant_id uuid not null,
  object_type text not null,
  object_id text not null,
  source_version bigint,
  source_updated_at timestamptz,
  projected_state jsonb not null,
  updated_at timestamptz not null default now(),
  primary key (tenant_id, object_type, object_id)
);

Before applying event version 2, compare it with the stored version 3. If 2 is stale, record it as superseded. If the source has no monotonic version, fetch current state for transitions where stale writes would be harmful.

Make's incomplete-execution documentation is also important. Incomplete executions are disabled by default, and storage has allowance limits. Depending on settings, a full incomplete-execution store can pause work or discard failed data. Those are platform recovery controls, not a substitute for your own source event record when losing the event is unacceptable.

I use Process data in order for small workflows where the business key matches the whole scenario and one blocked run should block everything. That is less common than people think.

When is Delay After Queue enough?

Delay After Queue is enough when the problem is rate shaping, the side effects are already idempotent, and global serial processing matches the business requirement.

Zapier's current Delay documentation says Delay After Queue processes queued runs one at a time in the order they were added, with a configured delay between them. That can protect an API that allows only a small number of calls per interval.

A good use case is sending non-critical records to a vendor endpoint with a known rate limit. Each record has a stable external ID. The vendor's update operation is idempotent. Reordering does not change the final state. A delayed run can be replayed without repeating a financial or customer-facing action.

A bad use case is protecting account creation from duplicates.

Delay After Queue reduces overlap. It does not prevent the same event from entering twice. It does not know that two different event IDs request the same business effect. It does not prove whether an earlier create call succeeded after a timeout.

A bad use case is per-customer ordering inside a busy shared queue. One customer's slow action blocks every other customer.

A bad use case is cancellation control. Zapier's common Delay problems explains that delayed tasks can be stopped by turning the Zap off before the delay expires or deleting the run. That is operationally different from a domain command that your application can cancel when a customer reactivates.

Use this decision test:

I do not replace native controls. I put them in the right layer. Zapier Delay can still shape calls after the durable dispatcher. Make Process data in order can still protect a narrow scenario. The external inbox gives you the acceptance and replay contract those settings do not provide.

What should sit in front of the visual automation tool?

A thin ingress service and a durable inbox.

The service can be a Next.js route handler, a small Express or Fastify service, a Supabase Edge Function, a Cloudflare Worker with a database behind it, or another environment that gives you raw request access and predictable response behavior.

The receiver has a short job:

read raw request body
identify provider and tenant
verify signature
extract or derive event identity
persist the original event and headers
return an acceptable response quickly

It should not create CRM records, call AI models, or wait for a ten-step automation.

The architecture looks like this:

Provider webhook
  -> signed ingress endpoint
  -> Postgres webhook_inbox
  -> dispatcher
  -> Zapier or Make internal hook
  -> guarded business operation
  -> downstream APIs
  -> result callback or status update
  -> reconciliation

The inbox schema needs enough information for dedupe, replay, and audit.

create type webhook_status as enum (
  'received',
  'duplicate',
  'ready',
  'dispatched',
  'processing',
  'completed',
  'retryable',
  'dead_letter',
  'superseded',
  'ignored'
);

create table webhook_inbox (
  id uuid primary key default gen_random_uuid(),
  tenant_id uuid not null,
  provider text not null,
  endpoint_key text not null,
  external_event_id text,
  event_type text not null,
  object_type text,
  object_id text,
  source_version bigint,
  source_occurred_at timestamptz,
  received_at timestamptz not null default now(),
  signature_valid boolean not null,
  headers jsonb not null,
  payload jsonb not null,
  payload_sha256 text not null,
  dedupe_key text not null,
  duplicate_of_id uuid references webhook_inbox(id),
  status webhook_status not null default 'received',
  dispatch_attempts integer not null default 0,
  next_attempt_at timestamptz,
  locked_at timestamptz,
  completed_at timestamptz,
  last_error jsonb,
  unique (tenant_id, provider, dedupe_key)
);

There is an important tension in that unique constraint. If you want to retain every delivery attempt, you cannot reject the duplicate row entirely. One pattern is a canonical event table plus a delivery-attempt table.

create table webhook_events (
  id uuid primary key default gen_random_uuid(),
  tenant_id uuid not null,
  provider text not null,
  dedupe_key text not null,
  event_type text not null,
  payload jsonb not null,
  first_received_at timestamptz not null default now(),
  status webhook_status not null default 'received',
  unique (tenant_id, provider, dedupe_key)
);

create table webhook_deliveries (
  id uuid primary key default gen_random_uuid(),
  event_id uuid not null references webhook_events(id),
  received_at timestamptz not null default now(),
  headers jsonb not null,
  payload_sha256 text not null,
  signature_valid boolean not null,
  source_ip inet,
  response_status integer
);

That design lets you say the provider delivered one logical event three times.

I usually retain the raw body too when signature re-verification or exact payload audit matters. Store it encrypted if the data is sensitive. Set a retention policy. Do not keep full webhook bodies forever because it is convenient.

[DIAGRAM] Draw the durable inbox architecture with a clear boundary between acceptance, dispatch, business operation, and downstream side effect. Mark which steps are allowed inside the provider request and which happen asynchronously.

How do you accept a webhook without doing the work inside the request?

Persist first, respond second, process later.

The implementation must preserve the raw body before JSON parsing when the provider's signature scheme signs exact bytes. A common Next.js mistake is parsing or mutating the body before verification.

Here is a simplified route handler shape:

import { createHash, timingSafeEqual } from 'node:crypto';
import { NextRequest, NextResponse } from 'next/server';

export const runtime = 'nodejs';

export async function POST(request: NextRequest): Promise<NextResponse> {
  const rawBody = Buffer.from(await request.arrayBuffer());
  const signature = request.headers.get('x-provider-signature');
  const tenantKey = request.headers.get('x-tenant-key');

  if (!signature || !tenantKey) {
    return NextResponse.json({ error: 'missing_signature_or_tenant' }, { status: 400 });
  }

  const tenant = await resolveTenant(tenantKey);
  const valid = verifyProviderSignature(rawBody, signature, tenant.webhookSecret);

  if (!valid) {
    await recordRejectedDelivery({ tenantId: tenant.id, rawBody, signature });
    return NextResponse.json({ error: 'invalid_signature' }, { status: 401 });
  }

  const payload = JSON.parse(rawBody.toString('utf8')) as ProviderEvent;
  const normalized = normalizeProviderEvent(payload);
  const payloadSha256 = createHash('sha256').update(rawBody).digest('hex');

  const accepted = await acceptWebhookEvent({
    tenantId: tenant.id,
    provider: 'provider_name',
    normalized,
    payload,
    payloadSha256,
    headers: Object.fromEntries(request.headers.entries()),
  });

  return NextResponse.json(
    { received: true, event_id: accepted.eventId, duplicate: accepted.duplicate },
    { status: 200 }
  );
}

The actual signature function must follow the provider's current documentation. Do not invent an HMAC format.

The database write should be atomic. An upsert can record the canonical event and a separate delivery attempt.

begin;

insert into webhook_events (
  tenant_id,
  provider,
  dedupe_key,
  event_type,
  payload,
  status
)
values ($1, $2, $3, $4, $5, 'ready')
on conflict (tenant_id, provider, dedupe_key)
do update set
  payload = case
    when webhook_events.status in ('received', 'ready', 'retryable')
    then excluded.payload
    else webhook_events.payload
  end
returning id, (xmax <> 0) as existed;

insert into webhook_deliveries (
  event_id,
  headers,
  payload_sha256,
  signature_valid,
  response_status
)
values ($6, $7, $8, true, 200);

commit;

Do not blindly replace the canonical payload on duplicate. Some providers include delivery-specific fields. Others may retry with corrected data. Define provider-specific rules.

Respond only after the durable write commits. If the database is unavailable, returning a failure allows a well-behaved provider to retry. Returning 200 before persistence trades provider retry for silent loss.

This receiver should have a strict execution budget. Signature verification and one database transaction are predictable. Calling ten external APIs is not.

For providers that require a custom response body or challenge, handle that protocol explicitly. For synchronous request-response integrations where the caller needs a computed answer, the durable inbox pattern may need a fast path plus asynchronous completion. Do not pretend every webhook can be made asynchronous.

How do you deduplicate events without dropping real changes?

Use the provider's event ID when it names one logical event. Use a business-effect key when multiple distinct events request the same one-time operation. Those are different layers.

Event deduplication answers:

Have I already accepted this source event?

Operation idempotency answers:

Have I already performed this business effect?

You often need both.

Suppose a billing provider sends invoice.paid event evt_123 three times. The event key is evt_123. Accept it once and record three delivery attempts.

Now suppose the provider emits two different events, checkout.completed and subscription.activated, both of which can trigger account creation. Their event IDs differ, but the intended operation is the same. The operation key is something like:

create_paid_account:{tenant_id}:{subscription_id}

The first event creates or claims that operation. The second observes the existing result.

Do not dedupe by full payload hash as your primary method when a provider ID exists. Payloads can contain timestamps, request IDs, reordered arrays, or delivery metadata. Two retries can hash differently.

Do not dedupe by a broad key such as customer email plus day. That can drop legitimate changes from the same customer.

For providers without event IDs, build a provider-specific canonical fingerprint from immutable fields.

function deriveDedupeKey(event: LegacyEvent): string {
  const canonical = [
    event.type,
    event.accountId,
    event.objectId,
    event.sourceVersion ?? event.occurredAt,
    event.operationId ?? '',
  ].join('|');

  return createHash('sha256').update(canonical).digest('hex');
}

This is safe only if those fields uniquely identify the logical event in that provider. Document the assumption and monitor collisions.

Some updates are snapshots rather than events. If a provider says “object X now looks like this” without a version, receiving the same snapshot twice may be harmless. Use upsert semantics and compare a canonical state hash.

insert into object_projections (
  tenant_id, object_type, object_id, projected_state, updated_at
)
values ($1, $2, $3, $4, now())
on conflict (tenant_id, object_type, object_id)
do update set
  projected_state = excluded.projected_state,
  updated_at = now()
where object_projections.projected_state is distinct from excluded.projected_state;

That protects the projection, not all side effects. Sending a customer email because the state changed still needs a transition key such as:

notify_customer:order_123:entered_shipped:v7

The version prevents a replay of an old shipped snapshot from sending the notification again.

[WORKFLOW JSON: Zapier or Make workflow that receives an internal event envelope, calls an idempotency-claim endpoint before any irreversible action, branches on existing-completed, existing-processing, and newly-claimed outcomes, then reports the final result.]

// [WORKFLOW JSON: Zapier or Make workflow that receives an internal event envelope, calls an idempotency-claim endpoint before any irreversible action, branches on existing-completed, existing-processing, and newly-claimed outcomes, then reports the final result.]

How do you keep event order when the provider does not?

You do not force universal order. You enforce the smallest order that preserves the business rule.

Global order is expensive and usually unnecessary. An update to Contact A does not need to wait for Contact B. Two updates to the same contact may need ordering. An invoice payment and a subscription cancellation for the same subscription definitely need a conflict rule.

Partition by business key:

tenant_id + object_type + object_id

Within a partition, use source version when available. A monotonic version is better than timestamp because clocks can differ and two changes can share the same timestamp precision.

select source_version
from object_projections
where tenant_id = $1
  and object_type = $2
  and object_id = $3
for update;

Then:

incoming version > stored version: apply
incoming version = stored version: duplicate or equivalent
incoming version < stored version: superseded
version missing: fetch current source state or apply an operation-specific rule

For transitions without source versions, model precedence.

const statusRank: Record<OrderStatus, number> = {
  pending: 10,
  paid: 20,
  allocated: 30,
  fulfilled: 40,
  cancelled: 50,
  refunded: 60,
};

A simple rank is dangerous when states branch. A cancelled order can be reopened. A refund can be partial. Use a real transition graph for complex domains.

const allowedTransitions: Record<OrderStatus, OrderStatus[]> = {
  pending: ['paid', 'cancelled'],
  paid: ['allocated', 'cancelled', 'refunded'],
  allocated: ['fulfilled', 'cancelled', 'refunded'],
  fulfilled: ['refunded'],
  cancelled: [],
  refunded: [],
};

When an unexpected event arrives, do not silently force it. Fetch current state, reconcile, or route to review.

A concrete example is a GoHighLevel opportunity update from multiple sources. A sales rep moves the opportunity to Won while a delayed nurture event tries to move it back to Contacted. Arrival order says the nurture update is newer. Business authority says the manual Won state should win.

Ordering alone cannot solve source priority. You need a conflict policy:

manual sales action outranks nurture automation
authoritative billing status outranks CRM custom field
fulfillment carrier state outranks internal estimate
explicit cancellation outranks delayed onboarding

Store the source and reason for every projection update.

alter table object_projections
add column authority text,
add column source_event_id uuid,
add column transition_reason text;

This is where a generic queue stops being enough. The queue delivers work. Your domain logic decides whether the work is still valid.

How should the dispatcher call Zapier or Make?

The dispatcher should send a normalized internal envelope, claim work with database locks, and treat the visual platform as a worker that can be unavailable or repeat its callback.

Claim ready events using for update skip locked.

with ready as (
  select id
  from webhook_events
  where status in ('ready', 'retryable')
    and coalesce(next_attempt_at, now()) <= now()
  order by first_received_at
  for update skip locked
  limit 25
)
update webhook_events e
set status = 'dispatched',
    dispatch_attempts = dispatch_attempts + 1,
    locked_at = now()
from ready
where e.id = ready.id
returning e.*;

The batch size and worker concurrency must match your plan, vendor limits, and downstream capacity. Do not copy a number from an article.

Send an envelope that is stable even if the provider payload changes.

{
  "schema_version": 3,
  "internal_event_id": "uuid",
  "tenant_id": "uuid",
  "event_type": "crm.contact.updated",
  "object": {
    "type": "contact",
    "id": "provider_contact_id",
    "version": 18
  },
  "occurred_at": "2026-08-08T10:00:00Z",
  "correlation_id": "uuid",
  "data_url": "https://internal.example.com/events/uuid/payload",
  "callback_url": "https://internal.example.com/events/uuid/result"
}

For sensitive payloads, do not send everything into workflow history. Send a short-lived signed data URL or let the workflow call a scoped internal endpoint. Make's scenario setting can keep processed data confidential, but its documentation warns that this limits troubleshooting. Design data minimization rather than relying only on log settings.

Use a dispatch signature so the visual workflow can verify that the call came from your service. Zapier and Make may not give you the same raw-body controls as your ingress, so choose a verification method compatible with the receiver. A random secret header over TLS can be acceptable for a private hook, but rotate it and scope it per environment or tenant group.

The dispatcher needs a timeout policy. A timeout does not mean the workflow did nothing. Mark the event dispatch_uncertain, then query the workflow result endpoint or wait for the signed callback before resending.

A visual workflow should claim the event or operation before side effects. It should not trust that receiving the envelope grants unlimited permission forever.

POST /operations/claim
  -> 201 newly claimed
  -> 200 already completed with result
  -> 409 currently processing
  -> 410 event superseded or cancelled

The workflow branches on that status.

When it finishes, it posts evidence:

{
  "internal_event_id": "uuid",
  "operation_id": "uuid",
  "status": "completed",
  "effects": [
    {
      "type": "crm_update",
      "external_id": "crm_record_id",
      "provider_request_id": "request_id_when_available"
    }
  ]
}

Do not let a workflow mark completion with only { "success": true }. Store enough evidence to inspect and reconcile the side effect.

How do you make downstream actions idempotent too?

The inbox prevents duplicate source processing. It does not make a non-idempotent downstream API safe.

Every irreversible action needs one of four protections:

provider-supported idempotency key
unique external reference accepted by provider
lookup and recover by deterministic business key
internal operation ledger plus reconciliation

Use provider-supported idempotency when available. The key should represent the business effect, not the attempt.

create_invoice:{tenant_id}:{order_id}:v1
send_welcome_sms:{tenant_id}:{customer_id}:{activation_version}
create_location:{tenant_id}:{subscription_id}

If the provider accepts your external reference and enforces uniqueness, use it. For example, create a CRM record with an immutable external ID custom field, then search by that exact ID on uncertain retry.

If the provider offers neither, use the operation ledger and a recovery state.

create type operation_status as enum (
  'pending',
  'processing',
  'uncertain',
  'completed',
  'retryable',
  'failed',
  'cancelled'
);

create table effect_operations (
  id uuid primary key default gen_random_uuid(),
  tenant_id uuid not null,
  operation_type text not null,
  business_key text not null,
  status operation_status not null default 'pending',
  provider text not null,
  request_payload jsonb,
  response_evidence jsonb,
  attempt_count integer not null default 0,
  last_attempt_at timestamptz,
  completed_at timestamptz,
  unique (tenant_id, operation_type, business_key)
);

The uncertain state is critical. It means the request may have succeeded, but you lack confirmation. Do not immediately repeat a non-idempotent action from this state.

A concrete example is sending an SMS. The API accepts the message, then your connection drops before the response. Repeating can send a second message. If the provider supports a client message ID, query by it. If not, record the exact destination, template version, and narrow time window, then use provider logs or manual review according to risk.

The safe retry policy varies by effect.

Your visual workflow should know the policy category. Do not let every HTTP module use the same automatic retry count.

Make's rate-limit guidance and error handlers can retry platform steps. Zapier can replay held or errored runs. Those features are safest when the effect endpoint is idempotent. Otherwise, automated recovery can multiply the original damage.

What does safe replay look like?

Safe replay is an explicit command against stored evidence. It is not “rerun everything from here.”

Make's current replay feature is useful because it can run a current scenario version with prior trigger data. Zapier also supports replay for affected runs. I use both for debugging and controlled backfill. I do not let platform replay be the only recovery interface for high-impact operations.

A replay request should specify:

event or operation ID
reason
requested by
selected processing version
allowed effects
whether dry run is required
expected current object version

Store it.

create table replay_requests (
  id uuid primary key default gen_random_uuid(),
  tenant_id uuid not null,
  target_type text not null,
  target_id uuid not null,
  requested_by text not null,
  reason text not null,
  processing_version text not null,
  mode text not null check (mode in ('dry_run', 'execute')),
  allowed_effects text[] not null,
  status text not null default 'pending',
  created_at timestamptz not null default now(),
  approved_by text,
  approved_at timestamptz,
  completed_at timestamptz,
  result jsonb
);

The replay engine re-evaluates current truth. It does not blindly reuse old authorization.

Suppose an old event says “send trial-ending reminder,” but the customer has already converted. Replaying the event today should classify the reminder as superseded. The old payload is input. Current subscription state is authority.

A dry run should show intended effects:

{
  "target_event_id": "uuid",
  "decision": "execute_partially",
  "effects": [
    {
      "type": "crm_projection_repair",
      "action": "apply",
      "reason": "projection_missing"
    },
    {
      "type": "customer_email",
      "action": "skip",
      "reason": "notification_already_completed"
    }
  ]
}

Require approval for payments, refunds, account deletion, customer messaging, and bulk replay. The approval threshold can depend on effect type and count.

Replay must use the same idempotency layer as live processing. Never build a separate “admin replay” path that bypasses operation claims because an engineer wants speed during an incident.

Bulk replay needs rate shaping and tenant limits. An outage can leave thousands of events ready at once. Releasing all of them into Make or Zapier can recreate the outage as a retry storm.

Use a replay batch with a fixed concurrency and pause control.

create table replay_batches (
  id uuid primary key default gen_random_uuid(),
  tenant_id uuid,
  requested_by text not null,
  status text not null default 'paused',
  max_concurrency integer not null,
  max_per_minute integer not null,
  filter jsonb not null,
  created_at timestamptz not null default now()
);

The values must be chosen from current platform and provider limits. Do not expose a free-form “max concurrency” field to support staff.

What happens when the provider is down for six hours?

Your inbox grows. That is the correct first behavior.

A durable system prefers backlog over loss. The hard part is recovering without overloading the provider or applying stale work.

Classify failures before scheduling retry. HTTP 429 and 503 are usually retryable. A 400 caused by a missing required field is not. An authentication failure may require connection repair before any event for that tenant can succeed.

Use exponential backoff with jitter and respect Retry-After.

function nextRetryDelayMs(attempt: number, retryAfterMs?: number): number {
  if (retryAfterMs && retryAfterMs > 0) return retryAfterMs;

  const base = Math.min(60_000 * 2 ** Math.max(attempt - 1, 0), 3_600_000);
  const jitter = Math.floor(Math.random() * Math.max(1_000, base * 0.2));
  return base + jitter;
}

These caps are examples, not recommended production values. Tune them to the provider and business deadline.

Add a circuit breaker. When repeated requests fail for the same provider or tenant connection, stop sending normal traffic for a cooling period. Continue accepting events. Probe with controlled requests. Resume gradually.

create table integration_circuits (
  tenant_id uuid not null,
  provider text not null,
  state text not null check (state in ('closed', 'open', 'half_open')),
  failure_count integer not null default 0,
  opened_at timestamptz,
  next_probe_at timestamptz,
  last_error jsonb,
  primary key (tenant_id, provider)
);

When service returns, do not process oldest-first without thinking. Some event types are cumulative and must all run. Others are snapshots where only the newest state matters.

For example, twenty contact-profile updates during the outage may collapse into one current-state fetch. Twenty order-line additions may not.

Define compaction policy by event type:

events:
  crm.contact.updated:
    recovery: fetch_current_state
    collapse_by: object_id
  order.line_added:
    recovery: process_each_event
    ordered_by: source_version
  analytics.page_view:
    recovery: bulk_insert
  notification.reminder_requested:
    recovery: reevaluate_current_policy

Make incomplete executions can preserve failed scenario state, and current documentation describes automatic or manual retry. That is useful inside Make. Your external inbox still lets you distinguish events that never reached Make from events waiting inside it.

A six-hour outage also tests credential expiry. OAuth tokens may expire while the queue waits. Refresh in the integration adapter before dispatch. Do not store refreshed tokens in a workflow node's static field when multiple workers can race. Use a locked credential record.

Finally, communicate backlog age. “Integration restored” is incomplete if six hours of work remain queued.

How does this behave under burst traffic?

Ingress should remain cheap while dispatch slows to downstream capacity.

That is the point of separating acceptance from work.

A burst test should measure four stages:

request acceptance latency
database commit latency
oldest ready event age
business-operation completion age

If acceptance is fast but oldest-event age rises, the queue is absorbing the burst. If database commit latency rises, ingress itself is at risk. If operation completion age rises only for one provider, its limit or adapter is the bottleneck.

Use indexes that match claiming and tenant inspection.

create index webhook_events_ready_idx
on webhook_events (first_received_at)
where status in ('ready', 'retryable');

create index webhook_events_tenant_status_idx
on webhook_events (tenant_id, status, first_received_at);

create index effect_operations_open_idx
on effect_operations (tenant_id, provider, last_attempt_at)
where status in ('pending', 'processing', 'uncertain', 'retryable');

Keep large payloads from bloating hot indexes. Store raw bodies in object storage when payload size is high, then keep a hash and location in Postgres. Ensure the object write and inbox record cannot silently diverge. One approach is writing the object first with a deterministic key, then committing the database pointer, with a cleanup job for orphaned objects.

Connection pooling matters in serverless environments. A burst of function instances opening direct Postgres connections can exhaust the database before event volume becomes large. Use a compatible pooler and set strict connection limits.

The dispatcher needs backpressure. Do not claim more events than workers can finish. A claimed event with a long lock timeout becomes invisible to other workers and difficult to inspect.

Use lease-based claims:

update webhook_events
set status = 'processing',
    locked_at = now(),
    lock_owner = $1,
    lock_expires_at = now() + interval '2 minutes'
where id = $2
  and (
    status in ('ready', 'retryable')
    or (status = 'processing' and lock_expires_at < now())
  )
returning *;

The lease duration must exceed normal processing time but recover from dead workers. Long operations should heartbeat the lease.

A burst also exposes noisy logs. Do not emit the full payload for every accepted event. Log structured IDs and classifications. Sample expected duplicates. Always log signature failures, dead letters, and operation uncertainty with enough context to investigate.

For Zapier, current docs say high webhook activity can be delayed and flood protection can be adjusted on paid plans. For Make, parallel webhooks can create high concurrency against downstream APIs. Your dispatcher should send at the rate each visual platform and connected app can accept, not at the rate the source can produce.

[DIAGRAM] Draw a burst graph with source arrival rate above downstream service capacity. Show the durable inbox absorbing the difference while dispatcher throughput remains bounded. Add queue-age thresholds and a circuit breaker.

[REAL NUMBER] Insert the largest verified burst your ingress accepted in a test, along with database commit percentile, oldest-event age, and final drain time. Do not present a provider benchmark as your own.

How do you stop one client from blocking every other client?

Partition capacity and isolate credentials.

A shared FIFO queue is simple until one tenant sends a large import, has an expired OAuth connection, or triggers a provider-specific error. Every retry from that tenant can consume the worker pool.

At minimum, enforce per-tenant concurrency.

create table tenant_dispatch_limits (
  tenant_id uuid primary key,
  max_inflight integer not null,
  max_per_minute integer not null,
  priority integer not null default 100,
  paused_at timestamptz,
  pause_reason text
);

The dispatcher checks inflight operations before claiming another event for that tenant. Do not rely on workflow folders or scenario names for isolation.

You can implement weighted scheduling:

select a tenant with ready work
respect tenant priority and current inflight count
claim one or a small batch
rotate to the next eligible tenant

For larger systems, use queue partitions by provider and tenant group. The exact queue technology can be Postgres, Redis, SQS, or another durable broker. Postgres is enough for many consulting implementations if you keep transactions short and indexes clean.

Credentials must be scoped to tenant and environment. A dispatch envelope should never contain a reusable OAuth refresh token. The integration adapter fetches the credential server-side using tenant context.

Use composite uniqueness everywhere:

tenant_id + provider + external_event_id
tenant_id + operation_type + business_key
tenant_id + object_type + external_object_id

An external ID may be unique only inside a provider account. Treating it as globally unique can attach one tenant's event to another tenant's record.

A concrete example is two agencies using GoHighLevel. Both can have a contact ID that is unique inside their locations or accounts but should never be resolved without the owning location and agency context. Your mapping table should include that scope.

create table external_identity_map (
  tenant_id uuid not null,
  provider text not null,
  external_parent_id text not null,
  external_object_type text not null,
  external_object_id text not null,
  internal_object_id uuid not null,
  primary key (
    tenant_id,
    provider,
    external_parent_id,
    external_object_type,
    external_object_id
  )
);

Isolation also applies to replay. A support user for Agency A must not search or replay Agency B's event. Use application authorization and database row policies where appropriate. Log every replay request.

When a tenant connection fails repeatedly, pause that tenant-provider pair, not the whole dispatcher. Keep accepting their events and show backlog. Healthy tenants continue.

What should you monitor before a customer reports it?

Monitor state gaps, not just HTTP errors.

The most useful alerts answer whether accepted work is reaching required outcomes.

I track these signals:

oldest ready event age by tenant and provider
oldest processing lease age
events accepted but never dispatched
operations in uncertain state
retry rate by error class
dead-letter count
signature failure rate
duplicate delivery rate
provider 429 and 5xx rate
visual workflow callback timeout rate
business operations missing required external evidence

Queue depth without age can mislead you. A thousand events that drain in seconds may be normal. Ten events waiting for two hours are not.

Create a mismatch query:

select
  e.tenant_id,
  e.provider,
  e.event_type,
  count(*) as stuck_count,
  min(e.first_received_at) as oldest_received_at
from webhook_events e
where e.status in ('ready', 'retryable', 'processing', 'dispatched')
  and e.first_received_at < now() - interval '15 minutes'
group by e.tenant_id, e.provider, e.event_type
order by oldest_received_at;

The threshold is domain-specific. A marketing sync can wait. A voice-call transfer event cannot.

Instrument correlation IDs across the full path. The same ID should appear in ingress logs, inbox row, dispatch request, Zapier or Make envelope, operation record, and downstream result.

Name Make runs with the internal event ID when possible. Make's current run-naming and replay capabilities improve history search. In Zapier, include the internal ID in fields that appear in run data. Do not put customer secrets in those names.

Build an operations view with three columns:

accepted
in progress
requires attention

The attention column needs actions such as inspect, fetch current source state, retry safe effect, mark superseded, and escalate uncertain side effect. Do not add a universal “retry” button.

Alert fatigue is real. Group repeated failures by tenant, provider, and root cause. One expired connection should create one actionable incident with an affected-event count, not a page for every event.

What does this architecture cost you?

You now own an integration service.

That means database migrations, backups, retention, secret rotation, worker deployment, alerting, support permissions, and repair procedures. You need to decide how long to retain raw payloads. You need a way to reprocess schema versions after provider changes. You need tests around signatures and dedupe keys.

For low-risk workflows, this is unnecessary. A Zap that posts a form response to an internal channel does not need a custom event system. Use the platform.

The architecture earns its cost when the workflow has one or more of these properties:

customer-facing side effects
money movement
account or resource creation
inventory or entitlement changes
bursty source traffic
strict audit needs
multiple retries or replay paths
multi-tenant credentials
high cost of duplicate execution
high cost of a missing event

There is also a latency tradeoff. Asynchronous dispatch adds queue delay. Usually that delay is small compared with external API calls, but it exists. If the source expects a synchronous answer, you need a different pattern.

A database inbox can become a bottleneck if designed poorly. Large JSON payloads, unbounded indexes, long transactions, and aggressive polling can hurt Postgres. Keep the hot path narrow. Archive old events. Use skip locked. Separate large raw bodies when needed.

The visual workflow becomes one worker rather than the center of truth. Operations staff may need a second interface for event state. That can feel less simple at first. The benefit arrives during failure, when the second interface tells them exactly what is safe.

You also have to resist rebuilding Zapier or Make in custom code. Do not move every mapping and branch into Node because the inbox exists. Keep human-readable orchestration where it remains useful.

My boundary is simple:

code owns acceptance, identity, concurrency, state, and replay authorization
visual tools own visible routing and bounded integration steps
provider systems own their domain truth
reconciliation checks all three

That division is not perfect. It is operable.

What would I do differently now?

I would add the inbox before the first serious burst, not after the duplicate-message incident.

I used to put dedupe near the end of a workflow because that was where the duplicate became visible. That is too late. Deduplicate at acceptance and again at the business-operation boundary.

I would never use search-before-create as the only duplicate protection. Search can help recovery. A unique operation claim prevents the race.

I would model uncertain from day one. Binary success and failure cannot represent a request that probably completed after the client timed out.

I would keep provider event IDs and business keys separate. One prevents duplicate event processing. The other prevents duplicate effects across different event types.

I would not enable automatic replay for every error. Rate-limit and transport failures can be retryable. Validation, policy, and uncertainty need different handling.

I would avoid global serial queues unless the entire workflow truly shares one ordered business stream. Per-object partitioning gives better throughput and fewer unrelated delays.

I would build the repair interface before launch. The first production failure should not require SQL typed into a live database. Safe actions need authorization, previews, and audit logs.

I would store the original event before transforming it. A mapper bug can be fixed. An event discarded after a failed mapping cannot.

I would also test old-event replay against current state. Reprocessing last month's payload through today's workflow can produce a valid API request and an invalid business outcome.

Finally, I would measure oldest-event age and uncertain-operation age from the first day. Error count alone stays quiet during the most dangerous failure, when the system is accepting everything and completing nothing.

[WORKFLOW JSON: complete internal Zapier or Make worker workflow with event claim, current-state check, operation claim, guarded downstream action, evidence callback, and explicit branches for duplicate, superseded, retryable, uncertain, and manual-review outcomes.]

// [WORKFLOW JSON: complete internal Zapier or Make worker workflow with event claim, current-state check, operation claim, guarded downstream action, evidence callback, and explicit branches for duplicate, superseded, retryable, uncertain, and manual-review outcomes.]

What is still easy to get wrong?

The hardest part is not storing the webhook. It is naming the business effect precisely enough that a unique key protects the right thing.

Make the key too narrow and duplicate actions slip through under different events. Make it too broad and you suppress legitimate repeated work. A welcome message may be once per customer activation, once per subscription, or once per location. The database cannot decide that for you.

That closes the loop from the forty-three CRM updates at the start. A queue would have spread them out. The correct operation key would have allowed one.

Even after you add the inbox, the dispatcher, and replay controls, you still have to answer the question most automation builds avoid: what, exactly, is allowed to happen once?