> ## Documentation Index
> Fetch the complete documentation index at: https://comis-feature-matrix-channel.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Monitor your agents' activity, costs, and message delivery from a single view

The Observe section in the sidebar gives you a bird's-eye view of everything happening in your Comis installation -- how active your agents are, how much they cost, whether messages are being delivered successfully, how prompt caching is performing, whether spend is within its ceilings, and -- when something goes wrong -- the deterministic root cause. Each sub-view is its own page: Overview, Context Engine, Context DAG, Billing, Delivery, Diagnostics, **Cache Health**, **Spend & Governance**, and the **Incident** drill-down. (Security-decision history lives in its own [Security & Audit](/security/audit) view.)

**Who it's for:** anyone watching costs, debugging delivery failures, measuring cache effectiveness, enforcing spend ceilings, or root-causing a degraded session.

## Where the metrics come from

All Observe data is sourced from Comis's own in-process collectors and persisted to SQLite (in `~/.comis/`). This native dashboard is the **zero-dependency lens** -- it needs no external services. If you prefer the standards lens, Comis also ships an **opt-in** OpenTelemetry/Prometheus export surface (off by default): a loopback `/metrics` endpoint plus Grafana dashboards-as-code and a one-command docker-compose stand-up -- see [Prometheus & Grafana](/operations/prometheus-grafana). The two render the same signals. The native dashboard reads them via JSON-RPC:

| Sub-view           | Primary RPC methods                                                                                                   |
| ------------------ | --------------------------------------------------------------------------------------------------------------------- |
| Overview           | `obs.delivery.stats`, `obs.billing.total`, `obs.billing.usage24h`, `obs.billing.byProvider`, `obs.delivery.recent`    |
| Billing            | `obs.billing.byProvider`, `obs.billing.byAgent`, `obs.billing.bySession`, `obs.billing.total`, `obs.billing.usage24h` |
| Delivery           | `obs.delivery.recent`, `obs.delivery.stats`, delivery-tracer detail RPC                                               |
| Diagnostics        | `obs.diagnostics`                                                                                                     |
| Cache Health       | `obs.cacheBreaks.byReason`, `obs.cacheStats.window`                                                                   |
| Spend & Governance | `obs.spend.snapshot`                                                                                                  |
| Incident           | `obs.explain`                                                                                                         |
| Context Engine     | `contextEngine.getLastMetrics`, `contextEngine.getLayerConfig`                                                        |
| Context DAG        | `context.dag.list`, `context.dag.tree`, `context.dag.search`, `context.dag.node`                                      |

Auto-refresh runs every 30 seconds; SSE event `metrics:update` triggers an extra refresh as soon as new data is recorded. A Prometheus `/metrics` endpoint is **not** exposed by default, but can be enabled via the opt-in [Prometheus & Grafana](/operations/prometheus-grafana) surface (loopback-bound on `127.0.0.1:9464`).

<Info>
  The Overview page at `/observe/overview` also contains five internal tabs -- **Overview**, **Billing**, **Delivery**, **Channels**, and **Diagnostics** -- that surface the same data as the standalone sub-views (plus a Channels tab unique to this page). The sidebar links route to the standalone components (e.g., `/observe/billing`); the Overview's internal tabs offer a unified view within a single page.
</Info>

## What You See

### Overview

Route: `/observe/overview` (also the default when clicking "Overview" in the sidebar)

The Overview is your at-a-glance summary. At the top, six stat cards show recent activity numbers:

* **Requests/min** -- agent request rate (recent throughput)
* **Error Rate** -- percentage of failed deliveries, color-coded green/yellow/red
* **Avg Latency** -- average delivery latency in milliseconds
* **Active Agents** -- count of agents that are currently running (not suspended)
* **Tokens (24h)** -- AI tokens consumed in the last 24 hours
* **Cost Today** -- estimated cost across all providers

Below the stat cards, a **24-hour activity chart** plots token usage over time so you can see when your agents are busiest, followed by per-agent and per-channel health grids.

The Overview also includes internal tabs for Billing, Delivery, Channels, and Diagnostics within the same page.

### Billing

Route: `/observe/billing` (standalone sub-view)

The Billing view breaks down your AI spending across drill-down levels:

* **By Provider** -- one row per AI provider (such as Anthropic, OpenAI, or Google) showing the provider name, total tokens used, number of API calls, cost, and cache savings. The cache savings column shows how much prompt caching has reduced your spending for that provider. A totals row at the bottom sums everything up.
* **By Agent** -- one row per agent showing the agent name, tokens used, what percentage of total usage that agent accounts for, and cost.

This helps you answer questions like "Which provider is costing the most?" and "Is one agent consuming a disproportionate share of resources?"

#### Per-tool and per-subagent cost

At the agent level, two additional tables surface finer-grained attribution when the data is available:

* **Per-tool cost** (labeled **best-effort**) -- attributes a turn's cost to the tool(s) that triggered it (from the `tool_tag` granularity). The per-tool shares are an even split across the distinct tools that fired in a turn, so they sum to the turn total -- they are a labeled estimate, never an exact per-tool charge.
* **Per-subagent cost** -- the corrected-dollar cost rolled up per subagent node, showing both the node's own cost and its **subtree cost** (the node plus all its descendants). This is exact within the agent graph.

These sections appear only when the daemon surfaces the granularity (governed by `observability.costGranularity.{perTool, subagentRollup}`); otherwise the view degrades silently rather than showing empty tables.

#### Filter and export

A filter bar at the agent level supports a **typed-query DSL** for narrowing the rows in place. Tokens are `key:value`, space-separated:

| Key               | Effect                                 |
| ----------------- | -------------------------------------- |
| `agent:<id>`      | only the named agent                   |
| `provider:<name>` | only the named provider                |
| `model:<id>`      | only the named model                   |
| `tool:<name>`     | only rows attributed to the named tool |
| `has:errors`      | only rows that had errors              |
| `minTokens:<n>`   | rows with at least N tokens            |
| `maxCost:<n>`     | rows costing at most \$N               |

Unknown keys are ignored (a hint names them) -- the query never fails. The filter is applied entirely in the browser over already-fetched rows; it is never turned into a database query.

The **Export CSV** and **Export JSON** buttons download the *currently-filtered* rows. The export carries a fixed set of cost columns only (agent id, tokens, percent-of-total, cost) -- never message bodies or any other content.

<Tip>
  The cache savings column helps you see the return on investment from prompt
  caching. Providers with high cache savings ratios indicate effective prompt
  reuse -- your system prompts and repeated context are being cached efficiently.
  The per-tool table's "best-effort" label is a reminder that a turn's cost is
  split across its tools, not measured per tool.
</Tip>

### Delivery

Route: `/observe/delivery` (standalone sub-view)

The Delivery view tracks whether messages are actually reaching their destinations. At the top, you will see three summary numbers:

* **Success rate** -- the percentage of messages delivered successfully (green if 99%+, yellow if 95-99%, red below 95%)
* **Average latency** -- how long deliveries take on average
* **Total deliveries** -- the total count in the selected time window

Below the summary, a **filter row** lets you narrow results by:

* **Channel type** -- filter to a specific platform (Telegram, Discord, etc.)
* **Status** -- show only successful, failed, or timed-out deliveries
* **Time range** -- choose between 1 hour, 6 hours, 24 hours, or 7 days

The trace table below shows each delivery with its timestamp, channel, a short message preview, status, latency, and number of steps. Click any row to expand it and see the full **delivery step waterfall** -- a step-by-step breakdown showing exactly where in the process a delivery succeeded or failed.

### Diagnostics

Route: `/observe/diagnostics` (standalone sub-view)

The Diagnostics view is an event log showing system-level activity. Each row has:

* **Time** -- when the event occurred
* **Category** -- what part of the system generated the event
* **Message** -- a description of what happened
* **Level** -- a severity badge (error, warning, or informational)

Events are sorted with the most recent at the top.

### Cache Health

Route: `/observe/cache` (standalone sub-view)

The Cache Health view answers "did prompt caching break this period, why, and what did it cost?" It reads `obs.cacheBreaks.byReason` and `obs.cacheStats.window`:

* **Stat cards** -- total cache breaks, **\$ lost** (the summed directly-lost cache-read saving across all reasons), and the hit/write ratio.
* **Break-rate-by-reason table** -- one row per [cache-break reason](/operations/observability) (e.g. `tools_changed`, `system_prompt_changed`, `ttl_expired`) with its count and estimated dollars lost. The dollar figure is `0` for an unknown-priced model (honest, never fabricated).

When the daemon surfaces no cache-break data, the view shows a "cache health not configured" empty state rather than a blank.

### Spend & Governance

Route: `/observe/spend` (standalone sub-view)

The Spend & Governance view is the native mirror of the Grafana Cost dashboard -- no Grafana required. It reads the **live** spend the kill-switch enforces (`obs.spend.snapshot`, the in-memory accumulator -- not a lagging SQL re-sum):

* **Headroom gauges** -- per-agent / per-tenant / global spend versus its ceiling, with the remaining headroom. A `null` ceiling renders a "spend governance not configured" state (honest -- the kill-switch ships off with null ceilings).
* **Pricing coverage** -- the three-state split (priced / free / unknown) so you can see how trustworthy the dollars are before acting on them.

See [Spend Governance (the cost kill-switch)](/operations/observability#16-spend-governance-cost-kill-switch) for the enforcement policy these gauges visualize.

### Incident (the root-cause drill-down)

Route: `/observe/incident?ref=<sessionKey|traceId>` (standalone sub-view)

The Incident view is the in-product twin of [`comis explain`](/reference/cli#comis-explain): it surfaces the deterministic, no-LLM [`obs.explain`](/reference/json-rpc#obs-explain) `IncidentReport` for a single session -- anomaly to root cause without leaving the product, and without Grafana. The same session always renders the same verdict.

It is reachable from **every per-session row** -- click a session in the Billing **By-Session** table, or the **Explain Incident** button in a [session's detail page](/web-dashboard/sessions-view) -- which navigates here keyed on that session's `sessionKey`. The view renders:

* **Header** -- the outcome severity (ok / degraded / failed) and the deterministic `likelyRootCause` code.
* **Stat cards** -- cost, tokens, duration, turns, cache-read ratio.
* **Failures table** and the **circuit-breaker timeline** (the normalized, newest-first failures + when each breaker opened/reset).
* **Optional, presence-conditional sections** -- rendered only when the session produced them: **Spend** (the kill-switch breach: scope + spent + ceiling), **Cache breaks** (per-reason count + \$ lost), **Recall** (memory-recall outcome counts), and **Audit** (per-kind security-event counts). The report is content-free -- counts, ids, dollars, and closed enums only, never a message body, query, or secret.

With no `ref` selected, the view shows a "select an incident to explain" empty state.

<Info>
  **Grafana relationship -- link, never embed.** When the opt-in [Prometheus & Grafana](/operations/prometheus-grafana) surface is enabled (`observability.prometheus.enabled: true`), the Incident view shows an **"Open in Grafana"** link to the matching dashboard (opening in a new tab). When Prometheus is disabled, no link is shown. The native dashboard **never embeds** Grafana (no `<iframe>`) -- it stays a zero-dependency SPA. One signal set, two lenses.
</Info>

## Understanding Your Costs

The Billing tab is your best tool for keeping spending under control. Here is how to read it effectively:

1. Start with the **By Provider** table to see which AI service is your biggest expense. The totals row at the bottom gives you the full picture.
2. Switch to the **By Agent** table to find which agents are consuming the most resources. The percentage column makes it easy to spot outliers.
3. Compare the two tables -- a high-cost provider combined with a high-usage agent tells you exactly where to optimize.

<Tip>
  Check the Billing tab weekly to catch unexpected cost increases early. A sudden spike in one agent's token usage often means it is stuck in a loop or receiving more traffic than expected.
</Tip>

## Tracking Message Delivery

When messages are not getting through, the Delivery tab helps you pinpoint the problem.

<Steps>
  <Step title="Switch to the Delivery tab">
    Click the **Delivery** tab at the top of the Observability view.
  </Step>

  <Step title="Filter by channel or status">
    Use the filter dropdowns to narrow results. To find problems quickly, set the status filter to **Failed** or **Timeout**.
  </Step>

  <Step title="Look for failed deliveries">
    Scan the trace table for rows with a red "failed" or orange "timeout" status badge.
  </Step>

  <Step title="Inspect the delivery waterfall">
    Click a failed delivery row to expand it. The step waterfall shows each stage of the delivery process with a green dot for successful steps and a red dot for the step that failed, along with the error message. This tells you exactly where the delivery broke down.
  </Step>
</Steps>

<Info>
  Data updates in real time via Server-Sent Events (SSE). Stat cards, tables, and charts refresh automatically as events arrive -- there is no manual polling delay.
</Info>

## Context Engine

Route: `/observe/context` (standalone sub-view)

The Context Engine view provides end-to-end observability for the DAG-based context pipeline. It refreshes automatically every 30 seconds and shows data for a selected agent or all agents at once via a filter dropdown.

### Pipeline Metrics

The top section shows key metrics in a 2-column grid:

* **Cache Hit Rate** -- percentage of prompt cache reuse (displayed as a gauge)
* **Budget Utilization** -- how much of the token budget is being consumed (displayed as a gauge)
* **Token Budget Segment Bar** -- a full-width segmented bar showing how the token budget is allocated across system prompt, tools, context, and messages
* **Eviction Chart** -- breakdown of context evictions by category

### Layer Waterfall

Below the metrics, a **layer waterfall** visualization shows the context assembly pipeline. Each layer represents a stage in context construction (system prompt, RAG results, session history, tool schemas, etc.) with timing information showing how long each layer took to assemble. This helps identify bottlenecks in context construction.

### DAG Compaction Panel

When DAG compaction has occurred, an additional section shows compaction statistics:

* **Total Compactions** -- how many compaction operations have run
* **Avg Summaries** -- average number of summary nodes per compaction
* **Max Depth** -- the deepest level in the DAG tree
* **Latest Duration** -- how long the most recent compaction took (in milliseconds)

## Context DAG Browser

Route: `/context` (standalone sub-view, listed under Observe in the sidebar)

The Context DAG Browser is an interactive tool for inspecting the directed acyclic graph (DAG) of context nodes that the context engine builds during conversations. It uses a two-column layout:

### Left Panel: Conversation List and Tree

* **Conversation list** -- shows all DAG conversations with their session keys. Select a conversation to load its tree structure.
* **Summary tree** -- an expandable tree view showing the hierarchical structure of the DAG. Nodes represent messages and summaries at various depth levels. Click a node to inspect it.
* **Search** -- a search bar performs server-side FTS5 full-text search within the selected conversation, highlighting matching nodes.

### Right Panel: Node Inspection

When you click a node in the tree, a slide-in detail panel shows:

* **Node type** -- message or summary
* **Content** -- the full text content of the node
* **Depth** -- the node's level in the DAG hierarchy
* **Kind** -- the node's classification (e.g., user message, assistant response)
* **Token count** -- how many tokens the node content contains
* **Parent IDs** -- which nodes this node was derived from
* **Child IDs** -- which nodes were derived from this node
* **Source message count** -- for summary nodes, how many original messages were compressed

This view is primarily useful for operators debugging context compaction behavior, understanding how summaries are generated, and verifying that the DAG structure matches expectations.

## Related Pages

<CardGroup cols={2}>
  <Card title="Channels View" icon="tower-broadcast" href="/web-dashboard/channels-view">
    Manage your connected messaging platforms and see their status.
  </Card>

  <Card title="Operations Monitoring" icon="gauge" href="/operations/monitoring">
    Server-side monitoring, health checks, and alerting.
  </Card>

  <Card title="Agents View" icon="robot" href="/web-dashboard/agents-view">
    View and manage your AI agents, their configuration, and activity.
  </Card>
</CardGroup>