loading
All projects
2026Independent research, sole author21 pages
When Semantic Caches Lie
Why one similarity threshold cannot make an LLM cache both safe and useful across domains
abstract
LLM systems increasingly reuse a stored answer whenever a new prompt looks close enough in embedding space, trading a model call for a cheap lookup. This paper asks what the literature mostly skipped: when that closeness test is wrong, what does it cost? I ground cache-equivalence in human-labeled prompt pairs across three domains and six embedding models, then measure how often a cache serves a confident wrong answer, what that false hit costs a downstream QA task, and how to set the similarity threshold so reuse stays safe.
Written, run, and typeset solo, unaffiliated with any institution, including the preregistration, the experiments, and every figure.
key findings
- On adversarial near-duplicates even the strongest neural encoder barely beats chance, and a classic lexical baseline quietly outscores all of them
- A false hit is nearly as damaging in retrieved context as in the final answer, and a naive fixed threshold can cost several times more than not caching at all
- A cost-aware threshold recovers near-baseline cost, but transfers across domains only in the conservative direction
- Preregistered and fully reproducible: every number and figure is generated from code, no hand entry, in a Dockerized pipeline
cite this work
APA
Patel, S. (2026). When Semantic Caches Lie: False Hits, Downstream Cost, and Calibration in Embedding-Based Caching for Large Language Models. https://doi.org/10.5281/zenodo.20532711
BibTeX
@misc{patel2026semanticcaches,
author = {Patel, Sunny Jayendra},
title = {When Semantic Caches Lie: False Hits, Downstream Cost, and
Calibration in Embedding-Based Caching for Large Language Models},
year = {2026},
doi = {10.5281/zenodo.20532711},
url = {https://doi.org/10.5281/zenodo.20532711},
note = {Zenodo}
}