Performance Tuning

How to measure, tune, and operate Isartor for maximum deflection and minimum latency.

Table of Contents

1. Understanding Deflection
2. Measuring Deflection Rate
3. Tuning Configuration for Deflection
4. Tuning Latency
5. Memory & Resource Tuning
6. Cache Tuning Deep-Dive
7. SLM Router Tuning
8. Embedder Tuning
9. SLO / SLA Goal Templates
10. Scenario-Based Tuning Recipes
11. PromQL Cheat Sheet

Understanding Deflection

Deflection = the percentage of requests resolved before Layer 3 (the external cloud LLM). A request is "deflected" if it is served by:

The deflection rate directly maps to cost savings. A 70 % deflection rate means only 30 % of requests reach the paid cloud LLM.

Measuring Deflection Rate

Via Prometheus / Grafana

The gateway emits isartor_requests_total with a final_layer label. Use the following PromQL to compute the deflection rate:

# Overall deflection rate (last 1 hour)
1 - (
  sum(increase(isartor_requests_total{final_layer="L3_Cloud"}[1h]))
  /
  sum(increase(isartor_requests_total[1h]))
)

# Deflection rate by layer (pie chart)
sum by (final_layer) (rate(isartor_requests_total[5m]))

# Exact-cache deflection only
sum(increase(isartor_requests_total{final_layer="L1a_ExactCache"}[1h]))
/
sum(increase(isartor_requests_total[1h]))

Via the API

Send a test batch and count response layer values:

# Send 100 identical requests — expect 99 cache hits
for i in $(seq 1 100); do
  curl -s -X POST http://localhost:8080/api/chat \
    -H "Content-Type: application/json" \
    -H "X-API-Key: $ISARTOR_API_KEY" \
    -d '{"prompt": "What is the capital of France?"}' \
  | jq '.layer'
done | sort | uniq -c

Expected output (ideal):

   1 3       ← first request → cloud
  99 1       ← remaining → exact cache

Via Structured Logs

When ISARTOR__ENABLE_MONITORING=true, every request logs the final layer:

# grep JSON logs for final-layer distribution
cat logs.json | jq '.isartor.final_layer' | sort | uniq -c

Via Jaeger / Tempo

Filter traces by the isartor.final_layer tag:

Tuning Configuration for Deflection

Cache Mode

• exact — Only identical prompts hit. Good for deterministic agent loops.
• semantic — Catches paraphrases ("Price?" ≈ "Cost?"). Higher hit rate but adds ~1–5 ms embedding cost.
• both — Exact check first (< 1 ms), then semantic if no exact hit. Best of both worlds.

Similarity Threshold

How to tune:

1. Set ISARTOR__ENABLE_MONITORING=true.
2. Send representative traffic for 1 hour.
3. In Jaeger, search for cosine_similarity attribute on l1b_semantic_cache_search spans.
4. Plot the distribution. If most similarity scores cluster between 0.80–0.90, a threshold of 0.85 is good.
5. If you see many scores at 0.82–0.84 that should be hits, lower to 0.80.

Cache TTL

• Short TTL (60–120 s): Good for rapidly changing data, real-time dashboards.
• Medium TTL (300–600 s): Balanced for most workloads.
• Long TTL (1800+ s): Maximises deflection for static Q&A / documentation bots.

Cache Capacity

• Monitor eviction rate via cache.evicted span attribute on l1b_semantic_cache_insert.
• If eviction rate > 5 % of inserts, increase capacity or shorten TTL.
• Each cache entry ≈ 2–4 KB (prompt hash + response + optional 384-dim vector).

Tuning Latency

Target Latencies by Layer

Measure with PromQL

# P95 latency by layer
histogram_quantile(0.95,
  sum by (le, layer_name) (
    rate(isartor_layer_duration_seconds_bucket[5m])
  )
)

# P95 end-to-end latency
histogram_quantile(0.95, rate(isartor_request_duration_seconds_bucket[5m]))

Reducing Latency

Memory & Resource Tuning

Memory Budget

CPU Considerations

• Embedding generation runs on spawn_blocking (dedicated thread pool).
• Candle GGUF inference is CPU-bound; allocate ≥ 4 cores for embedded mode.
• The Tokio async runtime uses the default thread count (num_cpus).

Container Limits

# docker-compose example
services:
  gateway:
    deploy:
      resources:
        limits:
          memory: 512M    # minimalist mode
          cpus: "2"
        # For embedded SLM mode:
        # limits:
        #   memory: 4G
        #   cpus: "4"

Cache Tuning Deep-Dive

Exact vs. Semantic Cache Hit Analysis

# Exact cache hit rate
sum(rate(isartor_requests_total{final_layer="L1a_ExactCache"}[5m]))
/
sum(rate(isartor_requests_total[5m]))

# Semantic cache hit rate
sum(rate(isartor_requests_total{final_layer="L1b_SemanticCache"}[5m]))
/
sum(rate(isartor_requests_total[5m]))

Cache Backend: Memory vs. Redis

Switch with:

export ISARTOR__CACHE_BACKEND=redis
export ISARTOR__REDIS_URL=redis://redis.svc:6379

When to Disable Semantic Cache

• Traffic is 100 % deterministic (exact same prompts repeated).
• Embedding overhead is unacceptable (< 1 ms budget).
• Set ISARTOR__CACHE_MODE=exact.

SLM Router Tuning

Embedded vs. Sidecar

Model Selection

For intent classification (TEMPLATE/SNIPPET/COMPLEX in tiered mode, or SIMPLE/COMPLEX in legacy binary mode), smaller models (1–3 B params) are sufficient. Use the smallest model that meets your accuracy needs.

Tuning the Classification Prompt

The system prompt in src/middleware/slm_triage.rs determines classification accuracy. If too many COMPLEX requests are misclassified as TEMPLATE or SNIPPET (resulting in bad local answers), consider:

1. Making the system prompt more specific to your domain.
2. Adding examples to the prompt (few-shot).
3. Switching to a larger model.
4. Setting ISARTOR__LAYER2__MAX_ANSWER_TOKENS to allow longer SLM responses (default 2048).
5. Falling back to binary mode via ISARTOR__LAYER2__CLASSIFIER_MODE=binary if the three-tier split does not suit your workload.

Embedder Tuning

In-Process (candle)

The default embedder uses candle with sentence-transformers/all-MiniLM-L6-v2 (pure-Rust BertModel):

• 384-dimensional vectors
• ~90 MB model footprint
• 1–5 ms per embedding (CPU)
• Runs on spawn_blocking to avoid starving the Tokio runtime

Sidecar Embedder

For higher throughput or GPU acceleration:

export ISARTOR__EMBEDDING_SIDECAR__SIDECAR_URL=http://127.0.0.1:8082
export ISARTOR__EMBEDDING_SIDECAR__MODEL_NAME=all-minilm
export ISARTOR__EMBEDDING_SIDECAR__TIMEOUT_SECONDS=10

Embedding Model Selection

Use 384-dim models for production. 768-dim models double memory usage for marginal quality improvement in most use cases.

SLO / SLA Goal Templates

Developer / Internal SLO

Production / Enterprise SLO

SLA Template (for downstream consumers)

## Isartor Prompt Firewall SLA

**Availability:** 99.9 % monthly uptime (< 43.8 min downtime/month)
**Latency:** P95 end-to-end < 2 seconds
**Error Budget:** 0.1 % of requests may return 5xx
**Maintenance Window:** Sundays 02:00–04:00 UTC (excluded from SLA)

### Remediation
- Cache tier failure: automatic fallback to cloud LLM (degraded mode)
- SLM failure: automatic fallback to cloud LLM (degraded mode)
- Cloud LLM failure: 502 Bad Gateway returned, retry recommended

### Monitoring
- Health endpoint: GET /healthz
- Metrics endpoint: Prometheus scrape via OTel Collector on port 8889
- Dashboard: Grafana at http://<grafana-host>:3000

Alert Rules (Prometheus)

groups:
  - name: isartor-slo
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(isartor_requests_total{http_status=~"5.."}[5m]))
          /
          sum(rate(isartor_requests_total[5m]))
          > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Isartor error rate exceeds 1%"

      - alert: HighP95Latency
        expr: |
          histogram_quantile(0.95, rate(isartor_request_duration_seconds_bucket[5m]))
          > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Isartor P95 latency exceeds 3 seconds"

      - alert: LowDeflectionRate
        expr: |
          1 - (
            sum(rate(isartor_requests_total{final_layer="L3_Cloud"}[1h]))
            /
            sum(rate(isartor_requests_total[1h]))
          ) < 0.5
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Isartor deflection rate below 50%"

      - alert: FirewallDown
        expr: up{job="isartor"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Isartor gateway is down"

Scenario-Based Tuning Recipes

Scenario A: Agentic Loop (High-Volume Identical Prompts)

Profile: Autonomous agent sends the same prompt hundreds of times per minute.

ISARTOR__CACHE_MODE=exact           # Semantic unnecessary for identical prompts
ISARTOR__CACHE_TTL_SECS=3600       # Long TTL — agent prompts are stable
ISARTOR__CACHE_MAX_CAPACITY=50000  # Large cache for many unique prompts

Expected deflection: 95–99 % (after warm-up).

Scenario B: Customer Support Bot (Paraphrased Questions)

Profile: End users ask the same questions in different ways.

ISARTOR__CACHE_MODE=both
ISARTOR__SIMILARITY_THRESHOLD=0.80  # Lower threshold to catch paraphrases
ISARTOR__CACHE_TTL_SECS=1800       # 30 min — support answers change slowly
ISARTOR__CACHE_MAX_CAPACITY=10000

Expected deflection: 60–80 %.

Scenario C: Code Generation (Low Cache Hit Rate)

Profile: Developers ask unique, complex coding questions.

ISARTOR__CACHE_MODE=both
ISARTOR__SIMILARITY_THRESHOLD=0.92  # High threshold — wrong cached code is costly
ISARTOR__CACHE_TTL_SECS=600        # Short TTL — code context changes quickly
ISARTOR__INFERENCE_ENGINE=embedded   # Let SLM handle simple code questions

Expected deflection: 20–40 % (SLM handles simple extraction).

Scenario D: RAG Pipeline (Document Q&A)

Profile: Queries against a knowledge base; similar questions are common.

ISARTOR__CACHE_MODE=both
ISARTOR__SIMILARITY_THRESHOLD=0.83  # Moderate threshold
ISARTOR__CACHE_TTL_SECS=3600       # Documents change infrequently
ISARTOR__CACHE_MAX_CAPACITY=20000  # Large cache for document variation

Expected deflection: 50–70 %.

Scenario E: Multi-Replica Kubernetes

Profile: Horizontally scaled behind a load balancer.

ISARTOR__CACHE_BACKEND=redis
ISARTOR__REDIS_URL=redis://redis-cluster.svc:6379
ISARTOR__ROUTER_BACKEND=vllm
ISARTOR__VLLM_URL=http://vllm.svc:8000
ISARTOR__VLLM_MODEL=meta-llama/Llama-3-8B-Instruct
ISARTOR__CACHE_MODE=both
ISARTOR__SIMILARITY_THRESHOLD=0.85

Benefit: All replicas share the same cache → deflection rate applies cluster-wide.

PromQL Cheat Sheet

See also: Metrics & Tracing · Configuration Reference · Troubleshooting