Direct answer
Many production failures live outside the repository: settings in an identity provider, a connection pooler, a secrets manager or an OAuth app. Tests pass because they never touch those settings, and the failure shows up only in the deployed environment, often as a vague network error. The fix is to treat third-party configuration as part of the system: write it down as code or a checklist, validate it at startup, and run a smoke check after every change.
Why config failures get past every test
Tests run against mocks. Local development uses a staging tenant, a direct database connection and a hand-pasted .env file. Production uses a different tenant, a pooler, a secrets manager and settings someone clicked through once. None of that shows up in a diff.
In the production issues we reviewed across about 25 projects in October, failures caused by third-party configuration appeared in five. In each, the symptom pointed somewhere other than the setting.
Four cases we worked through
An SSO login that failed with “Failed to fetch”
A workspace app with web and desktop clients moved its sign-in to WorkOS. Production was a separate WorkOS environment from staging, so every dashboard setting had to be redone there. The one that bit was the CORS allowed web origins list, which is separate from the redirect URIs. Without the web origin on it, the browser’s code exchange failed with a bare “Failed to fetch.” A second setting, the token issuer the API expects, produced a different symptom when it didn’t match: the API rejected tokens that looked valid. The team now keeps a list of every production setting and a go-live check that exercises the callback, the session endpoint and a test webhook after each deploy.
A connection pooler that silently dropped locks
A regulatory research platform ran six replicas behind the PgBouncer pooler built into its managed Postgres service, in transaction pooling mode. Leader election, migrations and a scheduled singleton job all depended on session-level advisory locks. Under transaction pooling, each transaction can land on a different server connection, so a lock taken on one is gone by the next query. The fix split the database setting into two URLs: a pooled one for the ORM and short queries, and a direct one for the three code paths that hold locks. A deploy step now confirms the replicas settle on exactly one leader, which proves the lock still works.
A Kafka certificate broken by escaping
A robotics telemetry platform stored its Kafka client certificate in a cloud secrets manager. Pasting the PEM through the console’s JSON editor escaped its newlines, and both the stream consumer and the API lost their connection to the cluster. Nothing in the code had changed. The documented procedure now enters the certificate through the plain-text editor with real line breaks.
An OAuth flow with too many moving parts
An outreach tool connected mailboxes through Google OAuth, which depended on a client ID, a client secret, a redirect URI and a background job refreshing tokens, all configured outside the code. The team replaced it with a simpler credential that is verified against the mail server at the moment it is connected, and an authentication failure now marks the account and pauses its sends instead of failing quietly.
A fifth case, a partial configuration update that erased a tool endpoint on a voice platform, is covered in our article on fake success in AI-built apps.
Know your pooler’s limits
The pooler case is common and quiet. PgBouncer’s features page says transaction pooling “breaks client expectations of the server by design.” Its compatibility table lists session-level advisory locks, SET/RESET, LISTEN, WITH HOLD cursors, SQL-level PREPARE/DEALLOCATE and temp tables that outlive a transaction as never supported in that mode.
Managed Postgres services often put PgBouncer in front of the database for you. If your app, migration tool or job library uses any of those features, give that path a direct connection, and route staging through the same pooler in the same mode. A direct staging connection hides the bug every time.
How to make third-party config testable
- Keep config as code where the vendor allows it. Where a setting can’t be managed by code, keep a per-environment checklist in the repository and review it like code.
- Validate at startup. Parse certificates and keys when the process boots, check that the issuer and audience values are present and plausible, and refuse to start on placeholder values. A boot error is easier to read than a runtime one.
- Run a smoke check after every change. After a deploy or a dashboard edit, exercise the real path: a sign-in, a webhook delivery, a lock acquisition, a tool call.
- Make staging match production’s topology. Same pooler and mode, same secrets manager, same kind of identity tenant.
- Prefer full updates over partial ones. Read the whole object, change one field, write it back and read it again, so a partial update can’t drop the fields you didn’t send.
The preview works but the live site fails guide covers the most common version of this in AI-built apps.
Related Zenveus services: Elastic Infrastructure, the 48-Hour Verdict and the Production Readiness Standard