Context Cache

How Comis keeps prompt costs low across long-running agent sessions

What it does. Reuses the parts of every prompt that do not change from turn to turn — system prompt, conversation history, tool schemas — so your provider charges the cheap “cached read” rate instead of the full input rate.

Who it is for. Anyone running long-lived agents in chat. The page leads plain-English; a developer reference table follows.

LLM providers charge 10–20x less for cached prompt reads than for cache writes. That difference only shows up when the bytes sent to the provider are byte-for-byte identical to the previous turn. Comis treats cache stability as a first-class goal — every feature in this document exists to protect that stability across the full lifetime of a conversation.

A 76-call Claude Opus session in production achieved a 16.9x cache read/write ratio: 94% of input tokens served from cache at $0.50/MTok instead of $15/MTok uncached. (Marketing benchmark — see the README for the exact session.) On a typical Claude Sonnet long-running channel agent, that ratio shaves a daily cost from roughly $26 uncached → $5 cached for the same conversation volume, just by keeping the prefix stable.

Comis ships 15+ specific cache optimizations across the executor, context engine, and provider adapters. They are listed below.

---

The 15+ optimizations at a glance

Each row is one named mechanism in code. The “Why” column explains what it prevents.

#	Optimization	What it does	Why it matters
1	Adaptive TTL escalation	Starts every session at “short” (5m); promotes to “long” (1h) only after 3+ turns of confirmed cache reads	Avoids paying 1h-write premium on sessions that would have ended in minutes
2	Fast-path large-write escalation	If turn 1 writes more than 20K cache tokens, escalates to “long” immediately on turn 2	Big system prompts deserve long retention up front
3	Prefix-instability detection	If cache reads stay flat at the system-prompt baseline for 5 consecutive turns, forces TTL back to “short”	Stops bleeding 1h writes when something keeps invalidating the prefix
4	Cache fence (lastBreakpointIndex)	Tracks the index of the highest cache_control marker; layers below the fence are skipped on subsequent passes	Prevents context-engine layers from rewriting cached bytes and blowing the cache
5	Sub-agent spawn staggering	Concurrent sub-agents are started with offsets so they reuse the parent’s cached prefix instead of all writing fresh entries	Concurrent fan-out without the N× write cost
6	Two-phase cache-break detection	Pre-call snapshot of system + tools + metadata; post-call AND-threshold (>5% drop AND >2K tokens) attributes the cause	Differentiates real breaks from normal cache-write noise
7	Cache-break diff writer + durable signal	When a real break is detected, writes a JSONL diff to ~/.comis/cache-breaks/ showing exactly what changed, AND persists a content-free cache_break row to the obs_diagnostics store (the 15-reason discriminator + an estimated lost-cache-read cost, never tool names or prompt text) plus a cache.break trajectory record	Lets you debug “why did my cache invalidate?” after the fact — and query “rate by reason over time, and what it cost” across sessions. Opt out via observability.persistence.cacheBreaks: false
8	Cache priming for sub-agents	Sub-agent spawn packets carry the parent’s cached prefix so the child starts warm	Child gets the parent’s deal without the parent paying twice
9	Cache-write suppression on sub-agents	Sub-agents read the parent’s cache but do not write fresh long-TTL entries of their own	No double-billing the same prefix
10	Per-zone retention	The recent-message zone always uses “short”; older zones get “long” after escalation	Recent content churns; older content is stable, so each gets the right TTL
11	Anthropic ephemeral cache	cache_control: { type: "ephemeral", ttl: "5m" or "1h" } markers placed at the right block boundaries	Native Anthropic cache, used by 100% of Claude calls
12	Gemini CachedContent	Explicit CachedContent API with SHA-256 content hashing + 50% TTL refresh on active sessions	Gemini’s per-object cache lifecycle, on by default via geminiCache.enabled (opt-out)
13	OpenAI completion-storage flag	store: true on Responses-API requests when storeCompletions is set	Lets OpenAI keep generated outputs available for replay/debugging
14	Microcompaction at write time	Tool results above the per-tool inline threshold are saved to disk and replaced with a tiny reference at the moment they are written	Keeps verbose tool output out of every future turn’s prefix
15	Schema stripping	After discover_tools injects a schema block, it is replaced in history with a one-line summary on subsequent turns	Recovers ~1.7K tokens per discovery, which would otherwise sit in the cached prefix forever
16	Tool deferral with discovery	Non-essential tool schemas are removed from the tools array and re-injected only when the agent calls discover_tools	Saves ~81% of the tokens the LLM would spend on tool definitions every turn
17	Anthropic prefix stabilizers	Strips replayed thinking from every assistant turn, defers transient memory-recall to an uncached tail block, and pins the cache anchor to a fixed position	Keeps the cache_control prefix byte-identical turn over turn so markers register reads instead of re-writing the whole prefix
18	OpenAI prefix stabilizers	Defers the current turn’s recall to a trailing (uncached, never-persisted) input item, and strips replayed reasoning items, on the OpenAI Responses auto-cached prefix	Stops the automatic prompt_cache_key prefix collapsing to the instructions+tools floor each time recalled memory or aged reasoning rotates

The numbered list lives across the executor and context engine: packages/agent/src/executor/adaptive-cache-retention.ts (rows 1–3, 10), packages/agent/src/context-engine/context-engine.ts (row 4), packages/agent/src/spawn/lifecycle-hooks.ts (rows 5, 8–9), packages/agent/src/executor/cache-break-detection.ts and cache-break-diff-writer.ts (rows 6–7), packages/agent/src/executor/gemini-cache-manager.ts + gemini-cache-injector.ts (row 12), packages/agent/src/executor/stream-wrappers/request-body/ — breakpoint-orchestration.ts (rows 11, 17 Anthropic markers), store-flag.ts (row 13 + the usesResponsesInputApi gate), tool-result-clearing.ts (rows 17–18 stabilizers), all wired by factory.ts, and packages/skills/src/builtin/... (rows 14–16 wire into the executor).

The rest of this page explains the most user-visible of these optimizations in detail.

---

Provider cache support

Comis caches against three different provider mechanisms, verified end-to-end across Anthropic (Claude), OpenAI (native openai, openai-codex, and Azure), and Google (Gemini). Each provider caches differently, so Comis adapts its strategy per provider — and, critically, stabilizes the prefix the same way for all three so the cache actually hits.

Provider	Mechanism	How Comis drives it	Knobs that apply
Anthropic	Explicit cache_control breakpoints — you mark prefix blocks and the API caches up to the marker	Places ≤4 breakpoints at zone boundaries (max 4/request, 20-block lookback), escalates TTL adaptively, and keeps the marked prefix byte-stable	cacheRetention, adaptiveCacheRetention, cacheBreakpointStrategy, the cache fence
OpenAI	Automatic prefix caching — no markers; the API caches the longest matching prefix, keyed by prompt_cache_key	Names prompt_cache_key from the session key (each agent and sub-agent gets its own namespace) and stabilizes the auto-cached prefix	none — TTL/breakpoint knobs do not apply (storeCompletions is replay storage, not caching)
Google	Explicit CachedContent server-side object + implicit prefix caching for the tail	Creates/refreshes a CachedContent object for the system instruction + tool schemas, keyed by a SHA-256 content hash	geminiCache.enabled (default true), geminiCache.maxActiveCaches

Anthropic — cache_control breakpoints + prefix stabilizers

Claude caches a prompt prefix only when the bytes up to the cache_control marker are byte-for-byte identical to the previous turn. Comis places the markers (zones, ≤4 breakpoints, 20-block lookback — rows 4, 10, 11) and actively keeps the prefix byte-stable (row 17) so the markers register cache reads instead of re-writing:

• Strip replayed thinking. Extended-thinking blocks are stripped from every replayed assistant message so the cached history matches the durable, no-thinking form and never mutates when a turn transitions active → historical.
• Defer transient recall to the uncached tail. The per-turn memory-recall block ([Relevant context from memory: …]) is moved off the cache-marked query block onto a trailing uncached block — still visible to the model, never cached — so the prefix does not change when that recall rotates out next turn.
• Pinned cache anchor. The semi-stable anchor breakpoint is pinned to a fixed position (the compaction summary, else the second user message) so it never re-anchors and re-writes the [0..anchor] span.

Reported per response as cache_read_input_tokens / cache_creation_input_tokens. A 5-minute write costs ~1.25× the input rate, a 1-hour write ~2×, and a read ~0.1×. The minimum cacheable prefix is model-dependent (~4096 tokens on Opus / Haiku 4.5, ~1024–2048 on Sonnet and older); below it the marker is silently ignored.

OpenAI — automatic prefix caching

The OpenAI Responses and Completions APIs cache automatically — there are no cache_control markers and no TTL knob. The API caches the longest prefix it has already seen, keyed by prompt_cache_key, which Comis derives from the session key so each agent and each sub-agent caches in its own isolated namespace. Running Anthropic’s cache_control machinery on an OpenAI request body would strip the type:"function" tool shape and 400 the request, so Comis disables breakpoints for OpenAI and instead protects the auto-cached prefix the same way it protects the Anthropic one (row 18):

• Defer recall to a trailing item. The current turn’s recall block is re-attached as a separate trailing input item (transient, never persisted) so each user turn is byte-identical to its future historical clean form — the auto-cached prefix no longer collapses to the instructions+tools floor when recall rotates.
• Strip replayed reasoning (native openai and Azure only). With store: false the SDK keeps reasoning for recent turns but drops it from aging ones — an early-index prefix mutation. Comis removes replayed reasoning consistently so the prefix stays byte-stable. openai-codex keeps its reasoning stable and was already optimal, so it is excluded.

Reported as cached_tokens in usage. With the stabilizers on, reads grow monotonically; an occasional single cached_tokens: 0 between otherwise-monotonic reads is OpenAI’s own best-effort eviction, not a Comis cache break — the request body is byte-stable by construction.

storeCompletions (→ store: true) is unrelated to caching — it asks OpenAI to retain the generated output for later replay/debugging. Leaving it false does not affect prefix-cache hit rates.

Google — explicit CachedContent

Gemini caching is on by default for Gemini agents (geminiCache.enabled: true). Comis creates a server-side CachedContent object holding the system instruction + tool schemas, injects its name into each request, and strips those fields from the body to avoid double-billing. Staleness is detected by comparing a SHA-256 content hash — not field presence. (A field-presence check would keep the cache from ever hitting: pi-ai leaves toolConfig undefined when no tool choice is set, which such a guard mistakes for a change, creating and evicting the cache on every call.) TTL is fixed at 3600s with a 50%-interval refresh on active sessions.

The explicit cache is shallow by design — it covers the system instruction + tools (a guaranteed, stable discount), while Gemini’s implicit prefix caching covers the conversation tail. Below the per-model minimum (~4096 tokens for gemini-3-pro / gemini-3.1-pro) the manager passes through uncached and implicit caching still applies. Reported as cachedContentTokenCount (constant across turns once the cache is live). Only Google AI Studio providers use the explicit path (not Vertex AI). Set geminiCache.enabled: false to opt out — e.g. a system prompt that churns every turn, which would thrash the per-object cache.

The cache fence, adaptive TTL escalation, and cacheRetention apply only to the Anthropic cache_control path. OpenAI prefix caching and Gemini CachedContent manage their own lifecycles, so those knobs have no effect there.

---

How context grows (and why it costs money)

Each conversation turn adds content to the context window your agent sends to the LLM. Here is what fills it:

• Conversation turns. Every user message and assistant reply is stored in a JSONL session file and replayed on the next call.
• Tool results. Every tool invocation appends a result block. A bash tool returning 50K characters of log output becomes a permanent fixture unless Comis intervenes.
• System prompt. Identity files (AGENTS.md), security rules, skill manifests, and workspace instructions are assembled once and sent every turn.
• Memory retrieval. Before each call, the RAG layer fetches up to five memory entries and injects them as additional context.
• JIT skill guides. Verbose operational guides for specific tools are withheld from the system prompt and injected on first use.

Comis never silently deletes conversation messages. The full history is always persisted to disk. What changes between turns is how much of that history the context engine chooses to forward to the model.

---

Adaptive retention

When Comis places a cache_control marker on a prompt block, it signals the provider to cache that block. But there are two TTLs to choose from: short (5 minutes) and long (1 hour). Writing at 1h TTL on a cold session wastes money if the cache entry expires before it is ever read. Adaptive retention solves this.

Here is what is actually happening under the hood:

1. Cold start. The first call writes at "short" (5m) TTL. This minimizes write cost on sessions that may not continue.
2. Escalation check. After each turn, Comis records the cacheReadTokens returned by the provider. Once confirmed reads arrive, the session is “warm.”
3. Promotion. After 3 turns of confirmed cache reads (or immediately on turn 2 if the first turn wrote more than 20K cache tokens — a signal of a large system prompt), retention escalates to "long" (1h).
4. Prefix instability detection. If cache reads stay at or below the system prompt baseline for 5 consecutive turns, the prefix is unstable and further 1h writes would go to waste. Comis forces retention back to "short" until the prefix stabilizes.

Sub-agents use static "short" retention. They complete in under 60 seconds and never accumulate enough cache reads to justify adaptive tracking overhead.

The config key adaptiveCacheRetention (default true) controls this behavior. Set it to false to use the static value from cacheRetention instead.

---

Cache fence

The context engine pipeline modifies conversation history before each LLM call. Without coordination, those modifications would land inside the cached prefix and invalidate it — defeating the purpose of caching.

The cache fence prevents this. After each call, Comis records the message index of the highest cache_control breakpoint as the fence index. Any context-engine layer that would normally modify content at or below that index skips those messages instead.

Concretely:

• The observation masker (which replaces old tool results with placeholders) skips messages below the fence.
• Schema stripping (which strips verbose discovery schemas from session history) skips messages below the fence.
• Microcompaction (which offloads oversized tool results to disk) skips messages in the first third of the session.

The fence index is stored in pre-trim space and correctly translated after the history window trims old messages from the front of the session, so protection survives across turns with long histories.

The cache fence is part of the Anthropic cache_control path. OpenAI uses automatic prefix caching keyed by prompt_cache_key, and Gemini uses the explicit CachedContent API — both manage their own prefixes (see Provider cache support). The fence has no effect on those providers; instead the OpenAI and Gemini prefixes are kept stable by the per-provider stabilizers described above.

---

Microcompaction

Microcompaction runs at write time — the moment a tool result is appended to the session file, not at the start of the next turn. This is different from the LLM compaction layer, which runs during context assembly.

When a tool result exceeds the per-tool inline threshold, Comis:

1. Saves the clean payload to tool-results/<toolCallId>.json in the session directory. External-origin results (MCP tools, web fetches — anything that arrived wrapped as untrusted content) are unwrapped before the write, so the file holds parseable payload bytes — an offloaded MCP JSON result is valid JSON on disk. The external origin is recorded in a <toolCallId>.origin.json sidecar.
2. Writes a compact reference into the session in its place: [Tool result offloaded to disk: ...] with a head/tail preview (1,500 and 500 characters respectively) cut from the clean payload, plus a recovery example that is verified against the written bytes (a json.load example only when the file actually parses as JSON; text-tool guidance otherwise). For external-origin results the preview is re-wrapped with the untrusted-content boundary — the security envelope is presentation-layer, not storage-layer.
3. The agent can recover the full content at any time: the read tool works for any offloaded file (recovery reads of tool-results/ files are exempt from re-offload up to the hard cap), and exec with python/jq keeps only the extracted data in context. When a recovery read targets an external-origin file, the content re-enters context re-wrapped as untrusted (keyed by the origin sidecar).

Per-tool inline thresholds:

Tool	Inline limit
Default tools	8,000 chars
MCP tools (mcp__*)	15,000 chars
read (file read)	15,000 chars
Hard cap (any tool)	100,000 chars

Results below the threshold are never offloaded. Above the hard cap, the clean payload is truncated to 100,000 chars on disk and the inline reference says so. Results that are already-offloaded are skipped on subsequent passes. The observation masker recognizes the [Tool result offloaded to disk: prefix and also skips those entries.

---

JIT guide injection

Verbose operational guides for specific tools — like the workspace customization guide for agents_manage, or the task delegation section — would consume significant tokens if included in the system prompt for every turn. Most turns never use these tools.

JIT guide injection defers them. The guides sit in memory, indexed by tool name. When a tool produces its first successful result in a session, Comis appends the corresponding guide as a text block at the end of that result:

---
[Tool Guide - shown once per session]
<guide content here>
---

The deliveredGuides set tracks which guides have been delivered in the current session. Once delivered, the guide is never injected again. On session reset, the set is cleared.

If a tool call errors on its first invocation, the guide is not consumed — the delivery slot stays open so a successful retry can fire it. Consuming the slot on a failed call would make the guide silently disappear for the rest of the session.

Two guide sources exist independently:

• Tool guides — per-tool operational instructions (keyed by exact tool name).
• System prompt guides — deferred system prompt sections triggered by a tool name, such as the privileged tools section which fires once on the first use of any of 10 privileged tools.

Learned-skill listing stability

When the Verified Learning reflection layer is enabled (default-off), the agent’s promoted, read-only learned procedures (<source>learned</source>) are listed inside the SAME <available_skills> block as the platform skills — but always appended AFTER the platform skills. The platform byte-prefix therefore never shifts when a learned skill is added or removed, so adding a learned procedure does not invalidate the cached prefix of the skills block.

The listing is captured by a per-session prompt-skills freeze (snapshotted once when the session starts). A newly-promoted procedure is therefore picked up on the next session, not mid-session — the in-session listing stays byte-stable, and a demoted/archived procedure likewise leaves the listing only at the next freeze. With the loop off (the default) the block is byte-identical to a platform-only install.

---

Tool deferral and parallelism

Deferral

With 50+ tools available, sending every schema to the LLM on every call wastes ~81% of the tokens spent on tool definitions. Comis defers non-essential tools using a BM25-backed discovery tool (discover_tools).

Deferred tools are removed from the tools parameter entirely. Their schemas are stored in a DiscoveryTracker. When the agent needs a deferred tool, it calls discover_tools with a natural-language query. Comis runs BM25 search over the deferred set and re-injects matching tool schemas mid-turn.

Tool deferral is configurable per-agent:

deferredTools.mode	Behavior
"auto" (default)	Rule + budget heuristics decide what to defer
"always"	All non-core tools are deferred
"never"	Deferral is disabled

Use deferredTools.neverDefer and deferredTools.alwaysDefer to pin specific tools regardless of mode.

Parallelism

The SDK can execute multiple tool calls concurrently within a single turn. This is safe for read-only tools but unsafe for tools that modify files, databases, or external services — ordering conflicts can corrupt state.

Comis classifies each tool as read-only or mutating. Read-only tools (file reads, web fetches, memory searches) run concurrently. Mutating tools share an async mutex and run one at a time, even when the SDK fires them in parallel. The classification is a static set; MCP tools default to the serialized path.

---

Schema stripping

When discover_tools loads a set of tools into the active context, it returns a <functions> block containing their full JSON schemas. Those schemas can be 1,720 tokens per discovery result. Once the schemas are loaded into the tools parameter for subsequent turns, the verbose block in the conversation history is pure redundancy.

After each turn, Comis scans the session history for discover_tools tool results containing <functions> blocks and replaces them with compact summaries:

[Discovery loaded: 4 tool(s) are now callable]
- tool_name_1
- tool_name_2
- tool_name_3
- tool_name_4

Already-stripped results (prefixed [Discovery loaded:) are detected and skipped. Results without a <functions> block pass through unchanged. Stripping respects the cache fence: entries at or below the fence index are not touched.

---

MCP disconnect cleanup

MCP servers can connect and disconnect at runtime. When a server disconnects, any tools it provided are no longer callable. If those tools remain registered in the DiscoveryTracker, the agent may attempt to call them and receive confusing errors.

Comis subscribes to two MCP lifecycle events:

• mcp:server:disconnected — removes all tools from the named server from every active session’s tracker.
• mcp:server:tools_changed — removes only the specific tools that were removed (not the entire server).

This cleanup runs automatically. You do not need to configure it. The agent’s active tool set reflects the current state of connected MCP servers within one turn of a disconnect event.

---

Configuration

These are the config keys most relevant to cache behavior. All live under the per-agent config block in your config.yaml.

Cache retention

Key	Default	What it controls
cacheRetention	"long"	Base TTL hint: "none", "short" (5m), or "long" (1h)
adaptiveCacheRetention	true	Whether to escalate from short to long after warm-up turns
cacheBreakpointStrategy	"auto"	Breakpoint placement: "auto" (→ multi-zone, recommended), "multi-zone", or "single". "single" places one breakpoint and can cause lookback-window cache misses (excessive cache writes) on long tool-using turns.
cacheRetentionOverrides	{}	Per-model-prefix overrides, e.g. "claude-haiku": "none"
advancedCacheOptimization.enableRecentZonePromotion	true	Promote recent-zone breakpoint from 5m to 1h in slow channels

Gemini explicit cache

Key	Default	What it controls
geminiCache.enabled	true	Enable Gemini CachedContent API (on by default for Gemini agents; set false to opt out)
geminiCache.maxActiveCaches	20	Max simultaneous cached content objects per agent

For Gemini agents, cacheRetention has no effect. Gemini uses the CachedContent API exclusively, managed by GeminiCacheManager. TTL is fixed at 3600 seconds with a 50%-interval refresh on active sessions.

Context engine

Key	Default	What it controls
contextEngine.observationKeepWindow	25	Tool results kept with full content (observation masker)
contextEngine.observationTriggerChars	120,000	Chars before observation masking activates
contextEngine.compactionCooldownTurns	5	Turns between LLM compaction triggers
contextEngine.compactionPrefixAnchorTurns	2	Head turns preserved during compaction for cache prefix stability
contextEngine.historyTurns	15	Recent user turns kept verbatim (pipeline mode)

Tool lifecycle

Key	Default	What it controls
toolLifecycle.enabled	true	Whether unused tools are demoted after N turns
toolLifecycle.demotionThreshold	20	Turns of non-use before a tool is schema-stripped and deferred

Compaction (session)

Key	Default	What it controls
session.compaction.reserveTokens	16,384	Tokens reserved for summary during SDK auto-compaction
session.compaction.keepRecentTokens	32,768	Recent-message tokens kept after SDK auto-compaction

---

Tips for long-running agents

If your agent drifts after many turns, the observation masker may be aggressively replacing tool results with placeholders before the model has used them. Try raising contextEngine.observationKeepWindow from 25 to 35 or 40.

If compaction runs too frequently, raise contextEngine.compactionCooldownTurns. The default of 5 turns is conservative. Sessions with slow-moving conversations (e.g., a Telegram channel where messages arrive every 10 minutes) can tolerate 10–15 turns between compactions.

If you see high cache-write costs on Haiku or Sonnet with small prompts, set adaptiveCacheRetention: false and cacheRetention: "short". This prevents the system from writing expensive 1h TTL entries on sessions that complete in under 5 minutes.

If your agent has many MCP tools, leave deferredTools.mode at "auto" and use deferredTools.neverDefer to pin the tools your agent uses on almost every turn. This avoids a discover_tools round-trip for high-frequency tools.

If a tool produces very large results (build logs, full file contents, API responses), the default 8K microcompaction threshold will offload them automatically. The agent can still read the full content on demand. If the offloaded path is causing confusion, raise maxToolResultChars per-agent to keep more content inline.

For Gemini agents, geminiCache is on by default — the CachedContent API caches the system instruction + tools server-side (guaranteed discount on cached input tokens) and is more predictable than relying on implicit prefix caching alone. Below the per-model minimum (~4096 tokens for gemini-3-pro / gemini-3.1-pro) the manager passes through uncached and Gemini’s implicit cache still covers the conversation tail. Set geminiCache.enabled: false to opt out (e.g. if your system prompt churns every turn, which would thrash the cache).