Metrics & Tracing

Definitive reference for Isartor's OpenTelemetry traces, metrics, structured logging, and observability stack — from local development to Kubernetes.

Overview

Isartor uses OpenTelemetry for distributed tracing and metrics, plus tracing-subscriber with a JSON layer for structured logging.

When ISARTOR__ENABLE_MONITORING=false (default), only the console log layer is active — zero OTel overhead.

Architecture

┌─────────────┐                  ┌──────────────────┐
│  Isartor    │  OTLP gRPC      │  OTel Collector   │
│  Gateway    │─────────────────▶│  :4317            │
│             │  (traces +       │                   │
│             │   metrics)       │  Pipelines:       │
└─────────────┘                  │  traces → Jaeger  │
                                 │  metrics → Prom   │
                                 └───┬──────────┬────┘
                                     │          │
                          ┌──────────▼──┐  ┌────▼──────────┐
                          │   Jaeger    │  │  Prometheus   │
                          │   :16686    │  │  :9090        │
                          │   (UI)      │  │  (scrape)     │
                          └─────────────┘  └───────┬───────┘
                                                   │
                                           ┌───────▼───────┐
                                           │   Grafana     │
                                           │   :3000       │
                                           │  (dashboards) │
                                           └───────────────┘

Enabling Monitoring

ISARTOR__ENABLE_MONITORING=true
ISARTOR__OTEL_EXPORTER_ENDPOINT=http://localhost:4317
RUST_LOG=info,h2=warn,hyper=warn,tower=warn       # optional override

When ISARTOR__ENABLE_MONITORING=false (the default), Isartor uses console-only logging via tracing-subscriber with RUST_LOG filtering. No OTel SDK is initialised — zero overhead.

Telemetry Initialisation (src/telemetry.rs)

init_telemetry() returns an OtelGuard (RAII). The guard holds the SdkTracerProvider and SdkMeterProvider; dropping it flushes pending telemetry and shuts down exporters gracefully.

Service identity:

service.name    = "isartor-gateway"
service.version = env!("CARGO_PKG_VERSION")   # e.g. "0.1.0"

Distributed Traces — Span Reference

Every request gets a root span (gateway_request) from the monitoring middleware. Child spans are created per-layer:

Root Span

http.status_code and isartor.final_layer are recorded after the response returns (empty → filled pattern).

Layer 0 — Auth

Auth is lightweight; no dedicated span is created. Events are logged at debug/warn level.

Layer 1a — Exact Cache

Layer 1b — Semantic Cache

cosine_similarity — the best-match score formatted to 4 decimal places. This is the key attribute for tuning the similarity threshold.

Layer 2 — SLM Triage

Layer 2.5 — Context Optimiser

When L2.5 modifies the request body, it also sets the response header x-isartor-context-optimized: bytes_saved=<N>.

Layer 3 — Cloud LLM

Custom Span Attributes — Quick Reference

These are the Isartor-specific attributes (beyond standard OTel semantic conventions) that appear on spans and are useful for filtering in Jaeger / Tempo:

OTel Metrics (src/metrics.rs)

Four instruments are registered as a singleton GatewayMetrics via OnceLock:

Where Metrics Are Recorded

Request Dimensions

Unified prompt telemetry distinguishes:

• traffic_surface: gateway or proxy
• client: direct, openai, anthropic, copilot, claude, antigravity, etc.
• endpoint_family: native, openai, or anthropic

Token Estimation

estimate_tokens(prompt) uses the heuristic: max(1, prompt.len() / 4). This is intentionally conservative — the metric tracks relative savings rather than precise token counts.

ROI — isartor_tokens_saved_total

This is the headline business metric. Every request resolved before Layer 3 (exact cache, semantic cache, or local SLM) avoids a round-trip to the external LLM provider.

# Daily token savings
sum(increase(isartor_tokens_saved_total[24h]))

# Savings by layer
sum by (final_layer) (rate(isartor_tokens_saved_total[1h]))

# Prompt volume by traffic surface
sum by (traffic_surface) (rate(isartor_requests_total[5m]))

# Prompt volume by client
sum by (client) (rate(isartor_requests_total[5m]))

# Estimated cost savings (assuming $0.01 per 1K tokens)
sum(increase(isartor_tokens_saved_total[24h])) / 1000 * 0.01

Use this metric to justify infrastructure spend for the caching / SLM layers.

Docker Compose — Local Observability Stack

Use the provided compose file for local development:

cd docker
docker compose -f docker-compose.observability.yml up -d

Configuration files:

Pipeline Flow

Isartor  ──OTLP gRPC──▶  OTel Collector ──▶  Jaeger    (traces)
                                          └──▶  Prometheus (metrics)
                                                     │
                                                     ▼
                                                  Grafana

OTel Collector Configuration

The collector config is at docker/otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
  otlp:
    endpoint: "jaeger:4317"
    tls:
      insecure: true
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp, debug]
    metrics:
      receivers: [otlp]
      exporters: [prometheus, debug]

Prometheus Configuration

The Prometheus config is at docker/prometheus.yml:

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 5s
    static_configs:
      - targets: ['otel-collector:8889']

Prometheus scrapes the OTel Collector's Prometheus exporter on port 8889 every 5 seconds.

Per-Tier Setup

Level 1 — Minimal (Console Logs Only)

No observability stack is needed. Use RUST_LOG for structured console output:

ISARTOR__ENABLE_MONITORING=false
RUST_LOG=isartor=info

For debug-level output during development:

RUST_LOG=isartor=debug,tower_http=trace

Level 2 — Docker Compose (Full Stack)

The docker-compose.sidecar.yml includes the complete observability stack:

cd docker
docker compose -f docker-compose.sidecar.yml up --build

Services included:

The gateway is pre-configured with:

ISARTOR__ENABLE_MONITORING=true
ISARTOR__OTEL_EXPORTER_ENDPOINT=http://otel-collector:4317

Level 3 — Kubernetes (Managed or Self-Hosted)

For all options, point the gateway at the collector:

ISARTOR__OTEL_EXPORTER_ENDPOINT=http://otel-collector.isartor:4317

Grafana Dashboard Queries (PromQL)

Jaeger — Useful Searches

Trace Anatomy

A typical trace for a cache-miss, locally-resolved request:

isartor-gateway
  └─ HTTP POST /api/chat                       [250ms]
       ├─ Layer0_AuthCheck                       [0.1ms]
       ├─ Layer1_SemanticCache (MISS)            [5ms]
       ├─ Layer2_IntentClassifier                [80ms]
       │     intent=TEMPLATE, confidence=0.97
       └─ Layer2_LocalExecutor                   [160ms]
             model=phi-3-mini, tokens=42

Built-in User Views

For quick operator checks without a separate telemetry stack:

isartor stats --gateway-url http://localhost:8080
isartor stats --gateway-url http://localhost:8080 --by-tool

Add --gateway-api-key <key> only when gateway auth is enabled.

--by-tool prints richer per-agent stats: requests, cache hits/misses, average latency, retry count, error count, and L1a/L1b safety ratios.

Built-in JSON endpoints:

• GET /health
• GET /debug/proxy/recent
• GET /debug/stats/prompts
• GET /debug/stats/agents

Alerting Rules

Prometheus Alerting Rules

Create docker/prometheus-alerts.yml:

groups:
  - name: isartor
    rules:
      - alert: HighErrorRate
        expr: rate(isartor_requests_total{status="error"}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Isartor error rate > 5% for 5 minutes"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(isartor_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Isartor P95 latency > 2s for 5 minutes"

      - alert: LowCacheHitRate
        expr: >
          rate(isartor_requests_total{final_layer=~"L1.*"}[15m]) /
          rate(isartor_requests_total[15m]) < 0.3
        for: 15m
        labels:
          severity: info
        annotations:
          summary: "Cache hit rate below 30% — consider tuning similarity threshold"

      - alert: LowDeflectionRate
        expr: |
          1 - (
            sum(rate(isartor_requests_total{final_layer="L3_Cloud"}[1h]))
            /
            sum(rate(isartor_requests_total[1h]))
          ) < 0.5
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Isartor deflection rate below 50%"

      - alert: FirewallDown
        expr: up{job="isartor"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Isartor gateway is down"

Troubleshooting

See also: Configuration Reference · Performance Tuning · Troubleshooting