Direct answer
Anything that can take more than a few seconds, such as a long model call, a document extraction or a multi-step agent run, should start a background job, save its state outside the web process, and let the client follow progress. Holding that work inside an HTTP request ties its success to every timeout between the user and your server, and when one of them fires, the work and its result are usually lost.
Every layer has a clock
A request passes through several systems, and each one has its own limit. Cloudflare’s proxy returns a 524 error if the origin hasn’t responded within 125 seconds by default. Vercel Functions stop at 300 seconds by default and return a 504. Gunicorn, a common Python application server, kills a worker that stays silent for 30 seconds by default. Model calls on reasoning models can take minutes, which is why OpenAI offers a background mode so long tasks run “without having to worry about timeouts.”
A demo with one user and a short prompt rarely touches these limits. Production traffic, longer documents and slower models do.
What it looks like when it fails
The failures are rarely clean errors. In the production issues we reviewed across about 25 projects in October, long AI or I/O work held inside a request showed up in seven of them. Three examples:
- A voice agent went silent mid-call. The server ran a single synchronous worker. One caller’s slow webhook kept it busy, so a second caller’s tool request waited past its 20-second tool timeout and the assistant stopped talking while the call stayed open. Separately, the server’s 30-second worker timeout was shorter than a 16,000-token extraction, so the worker was killed and the in-memory result disappeared with it.
- A content pipeline hit a six-minute execution cap. A batch job ran on a serverless platform with a hard limit per execution and failed partway through long runs.
- A synchronous RAG assistant invited duplicate work. Answers took 20 to 30 seconds behind a blank screen, so users clicked again and sent the same expensive request twice.
None of these were model problems. They were request-shaped designs applied to job-shaped work.
The pattern that holds up
- Accept, record, return. The request validates the input, creates a job record with a unique ID and returns immediately.
- Do the work in a worker. A queue, a scheduled function or the provider’s background mode runs the task, independent of the user’s connection.
- Keep state outside the process. Status, partial results and the final output live in a database or store, keyed by the job ID, so a restart or redeploy doesn’t erase them.
- Make retries safe. Key each job on something stable so a retry continues or replaces the same job instead of creating a second one.
- Show progress. Let the client poll the job status or subscribe to updates, and show what is happening instead of a frozen screen.
- Recover stuck jobs. Mark jobs that stay “running” past a sensible limit as failed or retry them, so nothing waits forever.
The content pipeline above was fixed this way: each execution now handles one unit of work and schedules the next, the draft is saved before optional steps such as image generation, images retry on their own trigger, creation is keyed on the slug so retries are safe, and any job stuck for more than 45 minutes is recovered automatically. In another product, replacing result polling with realtime database updates removed up to two seconds of perceived delay.
Raising the timeout is not the fix
Raising a worker timeout or a function’s maximum duration buys headroom. It doesn’t make the work survive a restart, a redeploy or a dropped connection, and it keeps a worker occupied while other users wait. The voice agent’s first fix raised its timeout and added threads, which stopped the silent calls in a live two-caller test; its notes still describe that as a stopgap until the work moves to a queue with shared state.
A quick test for your app
- Find every request that calls a model, parses a document or runs an agent.
- Time the slowest real case, not the demo case, and compare it with each timeout in the path.
- Kill the server mid-task. If the result is gone, the state lives in the wrong place.
- Send the same request twice. If two jobs run, retries aren’t safe yet.
Related Zenveus services: Agentic AI Development and AI Prototype Hardening