Inverted indexes and analysis

WHERE title LIKE '%kafka%' scans every row and finds nothing for "Kafka's" or "kafka-based". A search engine answers the same question in milliseconds over millions of documents because it did the work at index time: it broke each document into terms, normalised them, and built a structure that maps every term to the documents containing it. Understanding that structure — and the analysis that feeds it — is most of what separates a search that works from one that returns nothing for the query the user actually typed.

#The inverted index

A database index maps a key to rows. An inverted index maps a term to the list of documents it appears in, with positions:

term        postings (doc id: positions)
"kafka"     1:[3], 4:[0, 12], 9:[7]
"consumer"  1:[4], 9:[8]
"lag"       1:[5]

A query for kafka consumer looks up two postings lists and intersects them — document 1 and 9 — in time proportional to the lists' lengths, not the corpus size; a phrase query additionally checks that the positions are adjacent. That is the whole reason full-text search is fast, and it is also why the choice of what counts as a term is the choice of what can be found at all: if "Kafka's" was indexed as the term kafka's, a query for kafka will not match it.

Elasticsearch (and OpenSearch, the same engine forked) keeps one such index per field, in Lucene segments that are written once and never modified — a deleted document is marked, an updated one is a delete plus a new document, and segments are periodically merged. Two operational facts follow: an index is near-real-time, with a document searchable about a second after it is written (the refresh_interval), not immediately; and an update is not cheap, because it is a whole new document.

#Analysis: from text to terms

An analyser is the pipeline that turns a string into terms, applied to the document at index time and — this is the part people miss — to the query at search time, so that both sides agree. It has three parts:

1. Character filters: strip HTML, map characters (& to and).
2. A tokeniser: split into tokens. standard splits on Unicode word boundaries; whitespace only on spaces; keyword does not split at all; ngram and edge_ngram produce substrings for autocomplete; pattern splits on a regex.
3. Token filters: lowercase; stop removes "the", "a", "of"; stemmer reduces "consumers" and "consuming" to consum; asciifolding turns "café" into "cafe"; synonym expands "k8s" to "kubernetes".

"analysis": {
  "analyzer": { "en_tech": { "tokenizer": "standard", "filter": ["lowercase", "asciifolding", "english_stop", "english_stemmer"] } },
  "filter": { "english_stemmer": { "type": "stemmer", "language": "english" }, "english_stop": { "type": "stop", "stopwords": "_english_" } }
}

The _analyze API shows what an analyser does to a string — POST /_analyze { "analyzer": "en_tech", "text": "Kafka's consumers are lagging" } returns kafka, consum, lag — and it is the first thing to run when a query does not match: analyse the document's text and the query's text and compare the terms. Stemming is language-specific (the English stemmer mangles German), and stemming plus stop words trade recall for precision in ways a product team should decide, not a default.

#text versus keyword

The most consequential mapping decision, and the one every Elasticsearch beginner gets wrong once:

• A text field is analysed: split into terms, lowercased, stemmed. It is for full-text search — match queries, relevance scoring, phrase queries. It cannot be sorted on, aggregated on, or matched exactly, because the field's value is gone; only its terms remain.
• A keyword field is stored as one term, exactly as given. It is for identifiers, status codes, tags, email addresses, anything you filter with term, sort by, or aggregate over. A term query on a keyword field for PAID matches PAID; the same query on a text field matches nothing, because the indexed term is paid.

A field that needs both — a product name you search and sort by — is mapped as text with a keyword sub-field (name and name.keyword), which is what dynamic mapping does for every string, and which doubles the index for strings that only ever needed one of the two.

#Mappings

A mapping is the schema: each field's type and analyser. text, keyword, the numeric types, date with its formats, boolean, geo_point, nested for arrays of objects whose fields must be matched together (the MongoDB $elemMatch problem: without nested, an array of {sku, qty} objects is flattened into two independent arrays, and sku: A-1 AND qty > 5 matches across elements), and object for everything else. Set the mapping explicitly before the first document, with "dynamic": "strict" so that an unexpected field is an error rather than a guess.

Because a field's type and analyser decide what was indexed, a mapping cannot be changed for an existing field: changing status from text to keyword means reindexing every document into a new index, which is what the operations lesson's aliases are for. Design the mapping from the queries — the same access-pattern-first rule as the document-modelling lesson — and treat it as a migration when it changes.

#Dynamic mapping traps

With no mapping, the first document's shape becomes the mapping, guessed. The guesses that hurt: a string becomes text with a keyword sub-field (fine, and twice the size); a number that arrives as "42" in the first document becomes a keyword and rejects 42 in the second; a date that arrives as "2026-09-18" becomes a date and rejects "18/09/2026"; a field that is an object in one document and a string in another rejects the second forever; and a document with a thousand distinct keys (a map of user-defined attributes) creates a thousand fields, which is the mapping explosion that makes a cluster slow and is capped by index.mapping.total_fields.limit at 1,000 for exactly that reason. Every one of those is an indexing error at 3 am on a document that was fine in development. Explicit mappings, dynamic: strict, and flattened for the user-defined maps.