Direct answer
Most chatbot loops and memory lapses start in the application code. The app never recorded what question it just asked, dropped turns from the history it sends, or let the model’s own text stand in for a tool result. Keep conversation state in code: the current step, the facts collected and the last question asked. Add turn limits and loop detection, refuse any claim without a matching tool result, and grade with fixed rubrics run several times.
Where does conversation state actually live?
In many AI-built assistants, the only memory is the transcript pasted into the next prompt. The model has to infer from raw text what step the user is on and what “yes” refers to. That works in a three-turn demo and breaks when history is trimmed, filtered or built incorrectly, with nothing in the code noticing.
In the production issues we reviewed across about 25 projects in October, conversation state and agent reliability problems appeared in four. Each time, state the system needed was implied by the transcript instead of stored.
What these failures looked like
- An insurance quoting chatbot looped. After listing a business’s risks, it asked whether the user wanted to see the coverage they needed. Each “yes” got the risks again. The widget never recorded which question it had asked, so “yes” fell through to the model with nothing to attach it to. The fix stored the last question in conversation state and handled the answer in code.
- The same chatbot forgot what the user had just said. Someone typed “I have a restaurant”, asked for more detail and was asked what type of business they ran. Messages were displayed but never added to the history sent to the model, and the business type was saved as the whole sentence. The fix recorded every turn and stored the extracted business type as a field.
- An audit of a regulatory question-answering assistant found that its history filter kept only the user’s turns. When the assistant asked “Texas or California?” and the user replied “Texas”, the follow-up resolver no longer saw the clarifying question, so it rebuilt the request as a question about “Texas” alone.
- In a multi-agent content pipeline, the orchestrating agent once ran its own quick checks in place of the quality review it was meant to dispatch, and reported a pass. The runner now verifies each stage’s required outputs and fails the stage when they are missing, whatever the agent says.
An assistant inventing a lookup result is the worst version; see fake success in AI-built apps.
How to stop loops and lost context
- Keep explicit state outside the prompt. Store the current step, the facts collected so far and the last question asked as structured fields. Pass them to the model, and let code decide what a short reply like “yes” or “Texas” answers.
- Build history on purpose. Include the assistant’s own clarifying questions, and test the history builder directly. In the regulatory audit, the existing tests passed and still missed the bug, because none exercised history filtering and resolution together.
- Detect loops and cap turns. Compare each reply with the last few. If it repeats itself or a step recurs too often, hand off to a person or a fixed path. Cap agent steps and retries, and keep the attempt count in stored state so it survives restarts.
- Require a tool result before any claim. With client tools, the model only requests the call: Anthropic’s tool use documentation describes the flow: Claude responds with a
tool_useblock, your code runs the operation and sends back atool_result. Reject any answer that states “found”, “sent” or “checked” without a matching result in that turn.
Why grader scores keep moving
Teams often treat one model-graded score as fact. Anthropic’s glossary entry on temperature says that even with temperature set to 0, results are not fully deterministic and identical inputs can produce different outputs. Treat a single grader run as one sample.
The content pipeline above hit this in its prompt grader: across three runs on the same inputs, three of 21 scores varied. Any score used as a gate is now the median of three runs, calibration requires at least two of three runs within 0.15 of each other, and the grader must catch at least six of seven deliberately planted faults before its scores count. Editing a graded file voids its old score. The same project found that a deterministic contrast checker caught a text-color problem that two model-based visual reviews had missed.
So write rubrics as countable checks with worked examples, run graders more than once, test them against known faults, and use code wherever the check can be computed. Our article on building AI evals before tuning prompts covers the test set.
What to check this week
Open a real conversation log. Can you see, outside the transcript, which step the user is on? Does every claim trace to a tool result in the same turn? Does a rerun of your grader on yesterday’s outputs give the same scores? Any “no” is where to start.
Related Zenveus services: Agentic AI, the AI Agent Production Readiness Assessment and the 48-Hour Verdict