Calculate cosine similarity before choosing a threshold
Work through cosine similarity with small vectors, distinguish ranking from probability, and choose retrieval thresholds using labeled examples.

Cosine similarity measures the angle between two nonzero vectors. For text embeddings, it is often used to rank passages by their relationship to a query. A similarity of 0.9 does not mean that a passage has a 90 percent chance of answering the question. The score is a geometric quantity whose usefulness depends on the embedding model and the task.
You can see the distinction with two-dimensional vectors. Real text embeddings usually have more dimensions, but the same calculation applies. The small example below is synthetic; its coordinates are chosen to make the arithmetic visible.
Calculate the score by hand
Let the query vector be q = (1, 0). Compare it with three document vectors: a = (1, 1), b = (2, 0), and c = (-1, 0). Cosine similarity is the dot product divided by the product of the vector lengths:
cosine(q, d) = dot(q, d) / (norm(q) * norm(d))
cosine(q, a) = 1 / sqrt(2) = approximately 0.707
cosine(q, b) = 2 / 2 = 1
cosine(q, c) = -1 / 1 = -1
Vector b points in exactly the same direction as the query even though its length is twice as large. Cosine similarity ignores that length difference. Vector c points in the opposite direction. Vector a lies between the query direction and the vertical axis.
If both vectors are normalized to unit length, their dot product equals their cosine similarity. This can simplify an implementation, but normalization must be part of the stated contract. A raw dot product on vectors with different lengths can produce a different ranking. A zero vector has no defined direction, so the formula cannot safely divide by its norm.
The diagram lists the chosen coordinates and their computed scores. It does not claim that opposite vectors represent semantic opposites such as “approved” and “denied.” Embedding geometry does not provide that simple linguistic guarantee.
Why text can be nearby but still wrong
The Sentence-BERT paper describes sentence embeddings intended for efficient similarity comparisons. A retrieval system uses this type of representation to search without jointly running a large model over every query-document pair. The representation compresses text into a vector, and that compression can discard distinctions the application cares about.
Consider the question “What is the meal allowance for contractors?” A passage about the meal allowance for employees shares vocabulary and subject matter. It may rank highly despite applying to the wrong group. A passage about contractor reimbursement might use different terminology yet contain the answer. Similarity ranking helps choose candidates; eligibility and answer support still require checking.
This is why metadata can matter as much as vector distance. If the user asks about a particular jurisdiction, policy revision, or tenant, those conditions can restrict the candidate set. An authorization restriction should be enforced by the application or datastore before unauthorized material can reach the model. A later instruction to ignore it is too late to prevent exposure.
Use the actual model’s conventions
The OpenAI embeddings guide, retrieved on September 20, 2026, documents vector generation and similarity-search usage. Other embedding systems may have their own preprocessing, dimensionality, normalization, or separate query and document encoders. Read the documentation for the exact model you deploy.
Do not compare vectors produced by different embedding models merely because their arrays have the same length. The dimensions do not have a shared meaning across arbitrary models. Store an embedding-model identifier with each index, and rebuild or maintain separate indexes during migration. A dimension mismatch is an obvious error; a same-size but incompatible vector space can quietly return plausible-looking nonsense.
For a local arithmetic check, this Python function states its assumptions explicitly:
from math import sqrt
def cosine(left, right):
if len(left) != len(right):
raise ValueError("dimensions must match")
left_norm = sqrt(sum(x * x for x in left))
right_norm = sqrt(sum(x * x for x in right))
if left_norm == 0 or right_norm == 0:
raise ValueError("cosine is undefined for zero vectors")
return sum(x * y for x, y in zip(left, right)) / (left_norm * right_norm)
This is illustrative scalar code, not a performance recommendation for a production vector index. In a numerical library, you would also decide how to handle non-finite values, floating-point tolerance, and batches. The explicit checks are useful because silent input errors can otherwise look like poor model quality.
Choose a threshold from a decision
Suppose your application will answer only when retrieval appears strong enough. Build labeled query-passage pairs that include relevant passages, near misses, and unanswerable questions. Split them into a threshold-selection set and a final evaluation set. Examine how many relevant passages you retain and how many misleading passages you accept at each candidate threshold.
A threshold selected for short policy questions may not transfer to code search or multilingual support. Even within one domain, a new embedding model can change the score distribution. Store the chosen threshold with the model and dataset version, and retest it when either changes.
Inspect the mistakes behind a threshold
A small labeled fixture makes the choice tangible. Suppose eight query-passage pairs have scores 0.92, 0.88, 0.82, 0.79, 0.70, 0.62, 0.55 and 0.30. The first, third, fourth and sixth passages contain the needed answer under the correct policy conditions. The other four do not. These are hypothetical labels and scores, not outputs from a named embedding service.
pairs = [
(0.92, True), (0.88, False), (0.82, True), (0.79, True),
(0.70, False), (0.62, True), (0.55, False), (0.30, False),
]
def threshold_counts(threshold):
tp = sum(score >= threshold and relevant for score, relevant in pairs)
fp = sum(score >= threshold and not relevant for score, relevant in pairs)
fn = sum(score < threshold and relevant for score, relevant in pairs)
return tp, fp, fn
assert threshold_counts(0.80) == (2, 1, 2)
assert threshold_counts(0.60) == (4, 2, 0)
assert threshold_counts(0.90) == (1, 0, 3)
At 0.80, three passages pass: two relevant and one misleading. Precision is the relevant fraction among accepted passages, two thirds; recall is the accepted fraction of all four relevant passages, one half. At 0.60, all four relevant passages pass along with two misleading ones. Precision happens to remain two thirds in this fixture, while recall becomes one. Lowering the threshold did not make the accepted set uniformly better; it changed which errors the application must handle.
At 0.90, only the top passage survives. Precision is one on this tiny sample and recall is one quarter. That sample precision is not a guarantee about the next query. A near miss could receive an even higher score in another dataset. If no passage passes a threshold, precision has a zero denominator; report that state explicitly rather than silently presenting a perfect score.
The fixture evaluates individual query-passage pairs. A product that retrieves five passages and writes one answer also needs query-level checks. One correct passage among four wrong revisions may not produce a correct answer. Likewise, rejecting all passages can be an appropriate response to an unanswerable question. Define those outcomes separately instead of optimizing one pairwise number and calling the complete assistant reliable.
Jay Alammar’s The Illustrated Word2vec, published March 27, 2019, provides a visual introduction to representing items with vectors and comparing them in more than two dimensions. The arithmetic here makes the scaling boundary explicit: multiplying either nonzero vector by a positive factor leaves its cosine unchanged. Neither that property nor a compelling visualization establishes an application threshold.
As an exercise, change the 0.88 near miss to 0.95 without changing its label. Recompute the strictest threshold’s accepted set. It now contains one relevant and one misleading passage, so precision becomes one half. The exercise isolates the failure in treating a high geometric score as factual confidence, while leaving the cosine formula itself entirely correct.
For a first retrieval interface, returning the highest-ranked passages with their source titles can be more honest than displaying similarity as a confidence percentage. The reader can inspect the evidence, and the application can separately measure whether the retrieved passage contains the answer. That measurement is the basis for deciding when a generated answer is safe to show.