Lucene's default similarity function

Lucene's scoring Function is defined by the function

  1. tf(t in d) denotes the term's frequency, defined as the number of times the term t appears in the currently scored document d. Documents that have more occurrences of a given term receive a higher score.The default computation for tf(t in d) in DefaultSimilarity is:

  2. idf(t) stands for Inverse Document Frequency. This value correlates to the inverse of docFreq (the number of documents in which the term t appears). This means rarer terms give higher contribution to the total score. idf(t) appears for t in both the query and the document, hence it is squared in the equation. The default computation for idf(t) in DefaultSimilarity is:

  3. coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in coord(q,d) function by the Similarity in effect at search time.

    where, overlap is the number of query terms matched in the document and maxOverlap the total number of terms in the query.

  4. queryNorm(q) is a normalizing factor used to make scores between queries comparable. This factor does not affect document ranking (since all ranked documents are multiplied by the same factor), but rather just attempts to make scores from different queries (or even different indexes) comparable. This is a search time factor computed by the Similarity in effect at search time. The default computation in DefaultSimilarity produces a Euclidean norm:

    where, the sum of squared weights (of the query terms) is computed by the query Weight object as:
    sumOfSquaredWeights = q.getBoost()^ 2 * Sum t in q ( idf(t) * t.getBoost() ) ^2
    Lucene uses by default the value 1.0 for the factors q.getBoost() and t.getBoost().

  5. t.getBoost() is a search time boost of term t in the query q. Notice that there is really no direct API for accessing a boost of one term in a multi term query, but rather multi terms are represented in a query as multi TermQuery objects, and so the boost of a term in the query is accessible by calling the sub-query getBoost() .
    The boost is 1.0 by default !

  6. norm(t,d) encapsulates a few (indexing time) boost and length factors: