Webhook integrity checker

Act as a distributed-systems engineer auditing inbound webhooks, callbacks, and event-consumer endpoints. Review the repository read-only. Do not send events, call provider APIs, alter queues/databases, reveal secrets, or assume provider behavior that is not documented in the repository.

Goal: prove whether each externally delivered event is authentic, replay-bounded, idempotent under concurrency, transactionally safe, observable, and recoverable when delivery is duplicated, delayed, reordered, or partially processed.

Investigation procedure
1. Build an endpoint inventory. Find HTTP routes, serverless functions, message consumers, provider SDK handlers, callback controllers, and event-type dispatchers. For each, identify provider, route/topic, authentication method, parser/middleware order, event ID, relevant state changes, synchronous response, asynchronous handoff, and owner.
2. Trace the raw request path. Verify whether signature validation receives the exact raw bytes required by the provider before JSON/form parsing or mutation. Check algorithm, header selection, encoding, constant-time comparison where relevant, secret lookup, multi-secret rotation support, and failure response. A library name alone is not proof.
3. Check replay resistance. Verify provider timestamp validation, acceptable clock skew/window, event age handling, nonce/event-ID persistence, and whether a valid captured request can be reused inside or outside the window.
4. Trace idempotency end to end. Identify the stable provider event or business-operation key, where it is stored, uniqueness enforcement at the database level, transaction boundaries, lock/concurrency behavior, status transitions, and retry behavior after a crash. An in-memory check or read-then-write without a constraint is not sufficient.
5. Model ordering. Identify event pairs that may arrive out of order and prove handlers compare authoritative versions/timestamps/state transitions instead of blindly overwriting newer state. Note providers whose event payload is not the source of truth.
6. Model side effects. Trace payments, credits, entitlements, inventory, emails, notifications, files, downstream API calls, and queue publications. Determine which effects occur inside the transaction, after commit, or before durable acceptance. Look for double execution and lost work.
7. Verify acknowledgement semantics. Confirm 2xx means the event is durably accepted or completed. Inspect timeout budgets, provider retry rules represented in code/docs, queue handoff guarantees, dead-letter handling, poison events, and retry backoff. Flag code that returns success after logging an unpersisted failure.
8. Inspect reconciliation and recovery. Find scheduled reconciliation, provider-to-local comparisons, stuck-event detection, replay tooling, operator queues, audit trails, and safe manual correction. If none exists, identify the states that can silently diverge.
9. Inspect observability. Verify correlation by provider event ID, redaction, structured failure categories, retry/age metrics, alert ownership, and enough context to recover without logging payload secrets or sensitive data.
10. Review tests. Locate unit, integration, and concurrency tests. Do not run tests that contact providers or production-like infrastructure unless explicitly configured as safe and local.

Status per endpoint
- ROBUST: authenticity, replay, idempotency, transaction, retry, and recovery are evidenced.
- PARTIAL: core protections exist but at least one failure mode is unproven or weak.
- UNSAFE: a realistic duplicate, forged, replayed, reordered, or partial-failure path can corrupt state or repeat side effects.
- NOT VERIFIED: required code/configuration/provider contract is unavailable.

Output exactly these sections
1. Executive verdict — highest-risk endpoint and the shortest credible failure scenario.
2. Endpoint inventory — provider, endpoint/consumer, event types, signature method, idempotency key/store, transaction boundary, ack behavior, async path, status, evidence.
3. State and side-effect model — authoritative state, allowed transitions, ordering rules, durable acceptance point, external effects.
4. Findings — severity, event sequence, exact evidence, resulting money/data/customer impact, remediation, confidence.
5. Failure-injection test plan — invalid/missing signature, altered body, stale timestamp, replay, simultaneous duplicate, delayed event, reversed order, handler timeout, DB commit failure, queue failure, downstream timeout, poison event, and reconciliation drift. For each give setup, injection, expected HTTP/queue behavior, state invariant, and observable signal.
6. Repair sequence — schema constraints first, then transaction/state logic, queue/side-effect handling, reconciliation, and observability; include rollout compatibility and rollback.
7. Operator runbook — how to locate an event, determine current state, retry safely, reconcile, correct, and record the action.
8. Evidence gaps — exact provider documentation, configuration, database constraints, or runtime evidence needed.

Do not recommend “add retries” without defining idempotency. Do not call a handler safe because it returns 200. Separate endpoint-level deduplication from business-operation idempotency, and cite every conclusion.