Semantic Cache Observatory — Tune Speed Against Wrong Answers
A deterministic cache lab where every millisecond saved has to survive a correctness audit: replay paraphrases, near-misses, changed facts, and context boundaries while tuning threshold, TTL, keys, lookup mode, and invalidation.
What this is
A deterministic, single-file response-cache laboratory. Replay the same multi-tenant workload under exact, semantic, or hybrid lookup; tune threshold, TTL, context keys, and fact-change invalidation; then inspect the candidate, score components, key boundary, entry age, answer, and correctness classification behind every hit or miss.
Why this is mind-blowing
The fastest-looking policy can be the least useful one. The observatory makes that failure measurable: stale and contaminated hits lower precision and contribute zero trusted savings, while an immutable tenant guard proves that aggressive reuse never becomes cross-tenant leakage. The “semantic” model is deliberately local and lexical, with every weight and penalty visible instead of hidden behind an imaginary embedding API.
Prompt
Build “Semantic Cache Observatory — Tune Speed Against Wrong Answers,” a polished single-file simulation of exact and semantic response caching. Use a deterministic, transparent local text-similarity model (token normalization plus explainable weighted similarity; do not pretend it is a real hosted embedding). Provide a seeded prompt/answer workload with paraphrases, near-misses, changed facts, and tenant/context boundaries. Users tune similarity threshold, TTL, context-key strategy, exact-vs-semantic mode, and invalidation; step/play workloads and inspect why each request hit/missed, matched candidate/score, stale/wrong-answer classification, latency/cost saved, hit rate, contamination count, and precision. Include strict tenant isolation invariant, reset/presets, responsive 390px, accessibility, keyboard/touch, reduced motion. Inline HTML/CSS/JS only, no external deps/assets/network/build, <=160KB.Paste into your favorite AI and adapt it to your context.
What I learned shipping it
- A semantic cache is a precision system, not just a hit-rate system: correct hits divided by all hits exposes the false economy that raw latency charts hide.
- Tenant isolation belongs before candidate generation, while locale, plan, fact version, TTL, and invalidation determine how much safe reuse remains inside that hard boundary.
- A small lexical model can teach threshold behavior honestly when its normalization, weights, overlap terms, and penalties are visible instead of being mislabeled as a hosted embedding.