Managing Chatbot Conversation Context: Context Windows, Truncation and Summarization

An AI chatbot looks as if it remembers the whole conversation, but at the API level the model keeps nothing between turns: the application resends the history each time. The longer the conversation, the closer that history gets to the context window limit, raising cost and lowering quality.

This article draws on current Anthropic and OpenAI documentation, an Anthropic engineering post and a long-context research paper, checked on October 23, 2026.

It covers what the context holds, why long conversations cost more and perform worse, what each way of staying within the limit trades away, and how to measure. Click Star already has articles on token optimization and RAG, so this one skips pricing and how caching is billed. Click Star sells AI integration services, so it has an interest in the topic.

Key points

• At the API level the model does not remember between turns. Anthropic states the Messages API is stateless and you always send the full history; OpenAI states each request is independent.
• The context window holds everything in the request plus what the model generates: system prompt, tool definitions, messages, tool results, images, documents and thinking.
• Longer is not automatically better: Anthropic describes "context rot", and the Lost in the Middle study found performance often lowest when information sits in the middle.
• There are 5 ways to stay within the limit: truncating, clearing old tool results, summarizing, external memory and retrieval, each with a trade-off.
• Anthropic's summary block is readable text, while OpenAI's compaction item is encrypted and not meant for people to read.
• Summaries and memory inherit the history's personal data. Set retention limits and permissions, and build a retention test set before turning compression on.

What the context window is and why the model does not remember on its own

The context window is all the text a model references when generating a response, including that response. Anthropic's "Context windows" page (no date shown, read October 23, 2026) calls it the model's "working memory", distinct from the training data.

Anthropic's "Using the Messages API" page (read October 23, 2026) says the API is stateless, so you always send the full history. OpenAI's "Conversation state" guide (read October 23, 2026) says much the same: each request is independent, and multi-turn conversations are built by passing more messages.

OpenAI adds 2 ways to have the provider store state: the Conversations object and the previous_response_id parameter. The guide is explicit that with previous_response_id all earlier input tokens in the chain are still billed as input tokens.

What takes up space in the window

According to Anthropic, everything in the request counts toward the window: the system prompt, every message (including tool results, images and documents) and the tool definitions. What the model generates, including extended thinking, counts too. Prompt caching changes the price of those tokens, not whether they take up space.

Per Anthropic's token counting page, from Claude Opus 4.7 onward a newer tokenizer yields roughly 30% more tokens for the same text, so recount for the exact model.

Overflow differs by provider. For Anthropic, if the input alone exceeds the window the API returns a 400 error. On Claude 4.5 and later, if input plus max_tokens exceeds the window the request is accepted and stops with stop_reason model_context_window_exceeded.

For OpenAI, the Responses API reference lists truncation (marked deprecated) as defaulting to disabled, with a 400 error for an oversized request; with auto, the model drops items from the start of the conversation.

How long conversations get costly and worse

Cost grows faster than the turn count because each turn resends the whole history. Illustrative arithmetic, not a measurement: assume the history grows 500 tokens per turn, each request sends the whole history up to that turn, and ignore the system prompt and caching, so cumulative input is about 27,500 tokens after 10 turns, 105,000 after 20 and 410,000 after 40.

Anthropic's engineering post (published September 29, 2025) describes "context rot": as the token count rises, the ability to recall information accurately falls. It calls this a gradient rather than a cliff; reasons include n² pairwise relationships for n tokens and models seeing shorter sequences more often in training.

The "Lost in the Middle" study (arXiv, submitted July 6, 2023, revised November 20, 2023) tested multi-document question answering and key-value retrieval. Performance was often highest when the information was at the start or end and dropped significantly in the middle. Limit: 2023 models.

5 ways to stay within the limit

On models: server-side compaction needs Claude 4.6 or later; clearing works on all supported ones.

Truncating. Keep the last few turns and drop the rest. It is simple and needs no extra call, but everything outside the window is gone. Pin the facts to keep.

Clearing old tool results. Anthropic's context editing clears the oldest tool results once a threshold is passed (default 100,000 input tokens, keeping the 3 most recent tool uses) and replaces them with placeholder text, server side; your application keeps the full history. It is in beta. Clearing invalidates cached prompt prefixes, hence the clear_at_least parameter.

Summarizing (compaction). Anthropic replaces older turns with a summary written by Claude. The on-demand kind lets your application decide when; the threshold kind (the "Compaction at a token threshold" page) lets the API do it when input reaches a trigger (default 150,000 tokens, minimum 50,000).

Both are in beta. The on-demand kind can keep the latest turns verbatim, and its summarization call is billed like any other request.

External memory. Anthropic's memory tool lets Claude read and write files in a /memories directory, while your application executes the operations on storage you control. Anthropic's engineering post calls this structured note-taking.

Retrieval. Load only the relevant passages each turn, so the risk is loading the wrong one.

How Anthropic and OpenAI differ

OpenAI's "Compaction" page (read October 23, 2026) enables server-side compaction through context_management with compact_threshold on a /responses request. When the token count crosses the threshold, the server compacts and emits a compaction item in the same stream.

The big difference is readability. Anthropic's on-demand compaction block holds the summary as readable text with a signature and must be sent back unmodified. OpenAI describes its compaction item as encrypted and not meant for people to read. Click Star's inference: with Anthropic you can log and review summaries, with OpenAI you need other controls.

Compression has a price. OpenAI's Prompt caching guide says compaction can change the prefix and reduce cache reuse; Anthropic says clearing tool results invalidates cached prompt prefixes.

Risks: lost information and exposed personal data

Lost information. Anthropic's post warns that overly aggressive compaction can lose subtle context whose importance only shows later. Per the documentation, images, documents and fetched URLs in the summarized range disappear, and system-role instructions there stop applying, so restate them.

Silent failure. For on-demand compaction, Anthropic describes 2 mistakes that raise no error: summarized messages left after the block are sent again, and omitting the block later leaves the model without a summary. The chatbot "forgets" with no error logged, so test behavior.

Personal data. Chat history can hold names, phone numbers and addresses, so summaries and memory inherit them. OWASP lists sensitive information disclosure as LLM02:2025 and recommends data sanitization and least-privilege access.

For memory, Anthropic writes that Claude usually refuses to write sensitive information, but you should add a check before writing, cap file sizes and delete files not accessed for a long time.

OpenAI states responses are stored for 30 days by default (off with store set to false), while items in Conversations are not subject to that 30-day limit. This article does not analyze legal obligations on personal data.

What this means for Vietnamese businesses

Click Star found no Vietnam-specific measurements of typical conversation length, how often limits are hit, or Vietnamese summary quality. Token counts for Vietnamese also lack a source, so count on your own text.

This is a Click Star reference framework, not from outside sources. A short support bot on a website usually needs no compression: just pin the prompt and cap the turns.

Also part of this framework: a multi-day advisory bot should keep the customer's needs, commitments and status in the CRM, not the history.

In agents that call tools many times, results can pile up; Anthropic's engineering post calls clearing old tool results one of the safest, lightest forms of compaction. Click Star's inference: try it before summaries.

Click Star's view. Treat context as a budget, not a storage room. Facts that must never be forgotten (name, need, promises made, handling status) belong in structured data the business controls, because a summary can always miss something.

A 3-layer frame to consider (Click Star reference framework): latest turns verbatim, older history rolled into a summary, and a structured fact record stored outside. Click Star sells AI, CRM and CDP integration services, so it has an interest; this view covers only engineering and governance.

How to measure

The list below is a Click Star reference framework.

• Input tokens per turn, read from the usage field; watch high percentiles, not only the average.
• The share of conversations that hit the compression threshold or an overflow error, and cumulative input cost per conversation.
• Long conversations with fact-check questions at the start, middle and end, scored before and after compression.
• The share of summaries containing personal data or wrong facts, sampled by reading.

What to do: a 14-day plan

A Click Star reference framework, not from outside sources.

• Days 1 to 2: log input tokens per turn and see how many conversations are truly long.
• Days 3 to 4: list the facts that must not be forgotten, move them to structured data and check the personal data in the history.
• Days 5 to 7: build a set of long conversations with retention questions; score it before any compression.
• Days 8 to 10: try the simplest approach that works (truncating with pinning, or clearing old tool results) and rescore.
• Days 11 to 14: only if still short, try summarizing, read sample summaries, set retention and permissions, and recheck the documentation before going live.

Common mistakes

1. Believing the model remembers on its own, when the application stores and resends the history.
2. Stuffing in the full history because the window is large, ignoring cumulative cost and quality decline.
3. Turning on automatic summarization with no retention test set.
4. Letting important facts live only in the summary.
5. With on-demand compaction, forgetting to resend the summary block or leaving summarized messages in, so the chatbot forgets with no error.
6. Storing history, summaries and memory with no retention limit, permissions or personal-data cleanup, and changing models without recounting tokens.

What is not known

Click Star has no Vietnam measurements of conversation length, Vietnamese token counts or Vietnamese summary quality, and has not read Lost in the Middle-style results on current models. The article did not read Google documentation.

Anthropic's compression features are in beta, so parameters may change. Items marked "Click Star's inference" are untested. Legal obligations on storing chat history are not analyzed.