Querying and relevance

A database query returns the rows that match. A search query returns the documents that match ranked by how well, and the ranking is the feature: the difference between "here are 4,000 results" and "the one you wanted is first". This lesson is the query DSL's two halves — the part that filters and the part that scores — the formula behind the score, the levers that change it, and how to read the engine's explanation when the wrong document wins.

#Query context and filter context

Every clause in Elasticsearch runs in one of two contexts. In query context the clause answers "how well does this document match?" and contributes to the score. In filter context it answers "does this document match, yes or no?", contributes nothing to the score, and — the operational half — is cached as a bitset and reused across queries. Status codes, date ranges, tenant ids, categories: filters. The text the user typed: a query. Putting a status: PAID condition in query context wastes CPU scoring a term that is the same for every match; putting the user's words in filter context returns unranked results.

#The bool query

{
  "query": {
    "bool": {
      "must":     [ { "match": { "title": { "query": "kafka consumer lag", "operator": "and" } } } ],
      "should":   [ { "match_phrase": { "title": "consumer lag" } },
                    { "term": { "tags": "kafka" } } ],
      "filter":   [ { "term": { "status": "PUBLISHED" } },
                    { "range": { "publishedAt": { "gte": "now-1y" } } } ],
      "must_not": [ { "term": { "category": "internal" } } ]
    }
  }
}

must clauses must match and score; should clauses are optional and add to the score when they match (with minimum_should_match making some of them required); filter and must_not are filter context. The leaf queries: match analyses the query text with the field's analyser and matches any of the terms (operator: and requires all); match_phrase requires them adjacent, in order; multi_match runs one query across several fields with a type that decides how their scores combine (best_fields for "the one field that matches best", cross_fields for a name split across first and last); term matches one exact term — on a keyword field, always, and on a text field only by accident, because the query is not analysed and PAID will never equal the indexed paid; range, exists, prefix, wildcard (slow), fuzzy for typos within an edit distance. Almost every real search is a bool with the user's text in must, the boosts in should, and everything structural in filter.

#BM25: what the score is

The default scoring function is BM25, and its three inputs explain most surprising rankings:

• Term frequency: a document that mentions kafka five times scores higher than one that mentions it once — but with saturation, so the fiftieth mention adds almost nothing. That is the k1 parameter, and it is why keyword-stuffed documents stopped winning.
• Inverse document frequency: a term that appears in few documents is worth more than one that appears in most. kafka in a Kafka blog is nearly worthless; outbox is rare and decisive. Stop words are the limit of this: they appear everywhere and are worth nothing, which is why removing them changes little.
• Field length normalisation: a match in a 5-word title counts more than the same match in a 5,000-word body, because the short field is "about" the term more. The b parameter controls how strongly.

Scores are per shard, computed from that shard's term statistics, so a small index with few documents per shard can rank inconsistently between shards; search_type=dfs_query_then_fetch gathers global statistics at a cost, and a single-shard index avoids the problem for anything under a few million documents. Scores are also not comparable across queries: 12.4 for one query and 3.1 for another means nothing about either.

#Boosting: deciding what matters more

BM25 ranks by text alone. A product search wants in-stock items first, a knowledge base wants recent articles first, a marketplace wants the well-reviewed seller first — signals the text does not carry. The levers, cheapest first:

• Field boosts: multi_match with fields: ["title^3", "body"] makes a title match worth three times a body match.
• should clauses: an optional term on tags or match_phrase on the title adds score when it hits and costs nothing when it does not — the example above.
• boost on a clause, for weighting one signal over another.
• function_score (or the newer rank_feature and distance_feature queries): multiply or add to the score from a field — field_value_factor on popularity, a gauss decay on publishedAt so recency fades smoothly, a filter with a weight for in-stock. This is where business rules meet relevance, and where a small change reorders every result.
• rescore: rerank only the top N with an expensive query, so the cheap query finds candidates and the expensive one orders them.

The discipline around all of it: keep a set of judged queries — fifty real queries with the result a product owner says should be first — and run them as a test against every relevance change. Relevance tuning without that is moving a slider and hoping; the testing course's contract-test idea applies to search results too.

#Explain: why this document, and why not that one

GET /articles/_explain/42?q=... (or "explain": true on the search) returns the score's derivation for one document: each clause's contribution, and inside each, the term frequency, the IDF, the field length and the parameters that produced the number. Read it for two documents — the one that won and the one that should have — and the difference is usually one of: a term the losing document does not contain because analysis produced a different term (back to _analyze), a field-length penalty on a long document, an IDF that made a common word worthless, or a boost that fired for the wrong one. The _validate/query?explain=true endpoint shows how the query was rewritten before running, which is where a term query on a text field reveals itself.

The other diagnostic is the search profiler ("profile": true), which is EXPLAIN ANALYZE for search: time per query component per shard, and the wildcard or the unbounded range that is costing the milliseconds.