The Pivoted Document Normalization similarity function


The Pivoted document normalization Scoring Function [1] is defined by the function:


where
  1. tf(t in d) correlates to the term's frequency inside the document d. It is defined by:

    where ,
    freq (t,d) is the frequency of term t in the document d.
    avgFreq (d) is the average frequency of terms in the document d, defined by

    and Ud is the number of unique terms in the document d.

  2. idf(t) stands for Inverse Document Frequency. This value correlates to the inverse of docFreq (the number of documents in which the term t appears). This means rarer terms give higher contribution to the total score. idf(t) appears for t in both the query and the document, hence it is squared in the equation. The default computation for idf(t) in DefaultSimilarity is:
    idf(t) = 1+log( numDocs/ (docFreq + 1) )

  3. coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in coord(q,d) function by the Similarity in effect at search time.
    coord(q,d) = overlap / maxOverlap where, overlap is the number of query terms matched in the document and maxOverlap the total number of terms in the query.

  4. queryNorm(q) is a normalizing factor used to make scores between queries comparable. This factor does not affect document ranking (since all ranked documents are multiplied by the same factor), but rather just attempts to make scores from different queries (or even different indexes) comparable. This is a search time factor computed by the Similarity in effect at search time. The default computation in DefaultSimilarity produces a Euclidean norm: queryNorm(q)= queryNorm(sumOfSquaredWeights)= 1/ sumOfSquaredWeights ^1/2

    where, the sum of squared weights (of the query terms) is computed by the query Weight object as: sumOfSquaredWeights = q.getBoost()^ 2 * Sum t in q ( idf(t) * t.getBoost() ) ^2
    Lucene uses by default the value 1.0 for the factors q.getBoost() and t.getBoost().

  5. t.getBoost() is a search time boost of term t in the query q. Notice that there is really no direct API for accessing a boost of one term in a multi term query, but rather multi terms are represented in a query as multi TermQuery objects, and so the boost of a term in the query is accessible by calling the sub-query getBoost() .
    The boost is 1.0 by default !

  6. norm(t,d) = lengthNorm(d) , is defined by

    and pivot, is defined by the averaged document length given by

    D is the total number of documents in the collection, and slope is a parameter with a value in [0,1] depended from the collection.


For implementation details follow the link: How to run the Pivot similarity function using Lucene ?

References

  1. Amit Singhal, Chris Buckley, and Mandar Mitra, Pivoted document length normalization. In SIGIR '96: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 21-29, New York, NY, USA, 1996. ACM.
  2. Doron Cohen, Einat Amitay, David Carme, Lucene and Juru at Trec 2007: 1Million Queries Track, IBM Haifa Research Lab, TREC 2007


RETURN TO THE MAIN PAGE