Lost in Context Compaction

When LLM Systems Drop Side Constraints

Zhiqi Wang1, Yichi Zhang1, Dongwon Lee1, Nathalie Baracaldo2, Yuchen Yang1

1The Pennsylvania State University   2IBM Research

Abstract

When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks. We identify a class of user-issued instructions, side constraints (SCs), such as “do not delete any emails until I confirm,” that are meant to constrain the model’s behavior for the remainder of a session but are silently dropped during compaction. To quantify this loss we introduce ConPACT, an evaluation suite that measures compactors across three long-context scenarios: multi-turn chat, agentic trajectory, and long-horizon research. Current compactors retain only 17% of injected SCs on average, and most reduce compliance compared with the uncompacted context containing the SC. Retention varies with compactor, prompt, context length, SC phrasing, and injection location, showing that the loss is systematic rather than tied to any single setting. We propose an SC-aware extractor that runs alongside the compactor as a plug-and-play module, reaching over 90% retention across all three scenarios without modifying the compactor or the LLM.

Failure mode

Three panels from an inbox-cleanup session: the user asks the agent to suggest deletions and adds the side constraint "Don't act until I tell you to", and the agent only suggests; the compactor's summary drops the constraint; after "Please continue working through the rest of my inbox." the agent calls delete_email and deletes 10 emails.
Figure 1. The agent honors the constraint, loses it to compaction, then violates it without signalling that anything was dropped.

Evaluation suite

ConPACT crosses four axes. Fifteen hand-written SCs span five categories; each is rendered under four framings (constraint strength by explicitness) and injected into a 100K-token context at one of four locations. SWE-Natural adds real GitHub issues whose authors already wrote an SC, so nothing is injected. The context is then compacted and scored two ways: retention, whether the SC still appears in the summary, and compliance, whether the model behaves according to it on a forced-choice probe.

Overview of CONPACT: the five-category SC taxonomy; the four framings and four injection locations (top, middle, bottom, repeated); SWE-Natural GitHub issues that already contain an SC; and the compactor followed by the retention and compliance evaluations.
Figure 2. The ConPACT evaluation suite.

Retention by compactor

The evaluated compactors retain 17% of injected side constraints on average. Open-weight compactors and non-LLM baselines use 100K-token contexts; GPT-5.4-mini uses 220K-token contexts. Retention depends on the dataset and compaction prompt, with GPT-5.4-mini ranging from 6.7% to 98.0%.

Published dot plot of retention and effective retention by compactor for Hermes Agent, OpenResearcher, and WildChat. GPT-5.4-mini is evaluated separately at 220K tokens.
Figure 3. Retention and effective retention, from Figure 4 of the paper. Retention measures whether the SC survives in the summary; effective retention measures the normalized compliance gain over the uncompacted no-SC baseline. Open-weight compactors and baselines: 100K tokens, N = 750. GPT-5.4-mini: 220K tokens, N = 300; effective retention is unavailable because the probing model cannot process its uncompacted context.

Where the loss comes from

Constraints stated later in the context generally survive better, tracking proximity to the compaction instruction. Retention also varies with what the constraint binds: process constraints are retained least across compactors, and no SC type exceeds 36% retention in the results averaged over the three datasets.

Published retention heatmap by top, middle, and bottom injection location on WildChat and Hermes Agent at 100K tokens.
Figure 4. Retention by injection location, from Figure 7 of the paper. R5: Recent-5; Lingua: LLMLingua-2; oss-A/oss-P: gpt-oss with Anthropic/Pi prompts; Qwen3-A: Qwen3 with the Anthropic prompt.
Published retention heatmap for preference, information, output, action, and process SCs, averaged over three datasets. The highest retention is 36 percent for preference constraints under gpt-oss with the Pi prompt.
Figure 5. Retention by SC type, averaged over the three datasets, from Figure 8 of the paper.

SC-aware extractor

The extractor separates constraint tracking from summarization. Qwen3.5-9B reads user turns, identifies persistent constraints, and maintains a registry that is appended to the compacted context. It is training-free and requires no change to the compactor or the agent LLM.

SC registry retention with Qwen3.5-9B: Hermes Agent 93.7 percent, OpenResearcher 95.1 percent, and WildChat 90.3 percent.
Figure 6. SC registry retention with Qwen3.5-9B, using the extractor retention values reported in Table 4 of the paper.

Processing user turns takes 0.03 seconds per 100K-token context on OpenResearcher and up to 12.93 seconds on WildChat.

Code

Dataset construction, experiment runners, analysis code, and the extractor are on GitHub. The README maps each table and figure to the command that reproduces it.

git clone https://github.com/ZhiqiEliWang/compaction-integrity
cd compaction-integrity
pip install -e .
bash exp_sh/populate_dataset.sh   # build the long-context datasets
bash exp_sh/rq1/run_main.sh       # reproduce the main experiments

Citation

@misc{wang2026lostcontextcompactionllm,
      title={Lost in Context Compaction: When LLM Systems Drop Side Constraints}, 
      author={Zhiqi Wang and Yichi Zhang and Dongwon Lee and Nathalie Baracaldo and Yuchen Yang},
      year={2026},
      eprint={2608.11242},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.11242}, 
}