Why Semantic Similarity Can Give AI the Exact Opposite Answer

A researcher built 40 short policy statements across ten domains and generated three textual variants for each: a factual sentence, a synonymic rewording, and a contradictory version using various negation forms. Six open‑source sentence‑embedding models—all‑MiniLM‑L6‑v2, bge‑base‑en‑v1.5, e5‑base‑v2, gte‑base, mxbai‑embed‑large‑v1, and Qwen3‑Embedding‑0.6B—were used to embed both questions and statements. When the similarity threshold typical of RedisVL’s semantic cache (cosine ≥ 0.90) was applied, the opposite‑meaning question was judged more similar than the genuine rewording in 77.5 % to 100 % of the items, depending on the model. Consequently, a cache would return the cached answer to a “doesn’t cover” query 97.5 % of the time while only serving 7.5 % of true rephrasings. Retrieval experiments showed the same pattern: models found the “covers” passage almost always, but retrieved the “doesn’t cover” passage only between 27.5 % and 92.5 % of the time. Adding BM25 rank‑fusion generally worsened performance, and stripping stop words (which removes “not”) reduced BM25 to chance. A lightweight cross‑encoder re‑ranker on the top ten candidates restored accuracy to 97.5 % or higher for all models.

These findings echo earlier work that flagged negation blindness in neural retrieval—NevIR (EACL 2024) and Cao (ECAI 2025) reported random‑level performance on negated pairs, while Yadav’s 2026 preprint noted AUROC near 0.6 for contradiction detection. The problem is now quantified across the most‑downloaded sentence‑similarity model (all‑MiniLM‑L6‑v2, ~200 M monthly downloads) and five peers, demonstrating that the issue is systemic rather than an outlier. As enterprises embed open‑source transformers into caches, deduplication layers, and agent memories, they implicitly trust cosine similarity as a proxy for semantic equivalence—a trust that these results show to be misplaced for any sentence that flips meaning with a single negation token.

If production systems continue to rely on raw cosine thresholds, they risk delivering factually opposite answers without invoking a language model, effectively bypassing any safety net. Practitioners should benchmark their own embeddings against negated pairs, consider hybrid pipelines that combine BM25 with cross‑encoders, and avoid aggressive stop‑word removal. Monitoring AUROC for rephrasing vs. contradiction can serve as an early warning metric, and dynamic thresholding or explicit negation detection may be required before deploying semantic caches at scale.