Request Inspector

The Request Inspector is one of Luker's core backend modules, used to track the complete lifecycle of each AI generation request from initiation to completion, and record detailed token usage data. It serves as the infrastructure for generation diagnostics.

Problem Background

In SillyTavern, after an AI generation request is sent, the backend does not systematically record the request's token consumption. Users cannot know how many tokens each generation actually cost, and administrators cannot track resource usage in multi-user scenarios.

Luker implements a complete request lifecycle tracking system, covering text generation, image generation, and vector-embedding / rerank requests.

Core Capabilities

Request Lifecycle Tracking

Each AI generation request goes through the following state transitions:

1. Start — Records request metadata, marks the request as being tracked
2. Complete — Request returns successfully, records token usage
3. Fail — Request errors out, records error information
4. Abort — User actively cancels generation

Token Usage Statistics

The Request Inspector records detailed token data for each generation:

• Prompt Tokens — Tokens consumed by the input prompt
• Completion Tokens — Tokens consumed by the model's generated content
• Total Tokens — Total usage

This data is extracted from API responses and associated with user accounts for usage statistics and diagnostic analysis.

Token Statistics for Streaming Responses

For streaming (SSE) responses, token usage information is typically contained in the last SSE event. The Request Inspector extracts the usage field from the SSE event stream, ensuring accurate token consumption statistics for streaming generation as well.

Image Generation Request Tracking

Beyond text generation, the Request Inspector also tracks image generation requests, covering every image generation backend.

Embedding & Rerank Request Tracking

The Request Inspector also spans the vector subsystem, recording embedding, query, and rerank calls sent to every remote vector provider (OpenAI, Cohere, Jina, Ollama, VLLM, Voyage, and so on) and the KoboldCpp direct-embed bridge. Local-only inference sources that never leave the process are skipped so the ring buffer stays focused on actual upstream traffic.

Record Retention

Request records live in memory, capped at 200 per user. Each user keeps only their most recent records; once the cap is reached, the oldest record is dropped to make room.

Completed records are kept for two hours, then removed. A record that is still generating is not held to this clock: in-flight requests are kept for up to six hours so a long generation or a stalled stream is never swept before its completion callback runs. Expired records are swept every five minutes, which means a record survives at most two hours and five minutes past its start.

Record count and age alone do not bound memory: records hold full request and response bodies, and a chat with inlined images stores those bytes twice, before and after provider conversion. The inspector therefore also enforces a byte budget across all users: once the retained size passes the budget, the oldest completed records are evicted first. Records for requests that are still generating are never dropped by the budget, and a single record larger than the whole budget is kept rather than discarded.

All values can be tuned in config.yaml:

requestInspector:
  ttlMs: 7200000
  cleanupIntervalMs: 300000
  byteBudgetBytes: 268435456

• ttlMs: How long completed records are kept, in milliseconds
• cleanupIntervalMs: How often the expiry sweep runs, in milliseconds
• byteBudgetBytes: Soft cap on the heap bytes retained by request records across all users; when exceeded, the oldest completed records are evicted first

Relation to Storage Quotas

The token usage tracked by the Request Inspector is an independent statistics feature that helps users and administrators understand the resource consumption of AI generation. This is separate from the storage quota management in Auth & Quota:

• Token usage statistics: The Request Inspector records token consumption for each AI generation, providing usage visualization and diagnostic data
• Storage quota management: Manages the allocation and limits of file storage space

TIP

The Request Inspector starts automatically with the server. The defaults work out of the box; the retention settings above are there when you want to adjust them.