Introduction to Apache Lucene

Have you ever ever been interested by what powers a number of the finest Search Functions equivalent to Elasticsearch and Solr throughout use instances such e-commerce and a number of other different doc retrieval programs which might be extremely performant? Apache Lucene is a strong search library in Java and performs super-fast searches on massive volumes of information. The indexing and search capabilities of Lucene supply the absolute best options for search engines like google. 

By the tip of this text, you should have mastered the basics of Apache Lucene even if you’re new to the sector of Search Engineering.

Studying Targets

• Be taught the basic ideas of Apache Lucene.
• See how Lucene powers search functions like Elasticsearch, Solr and so forth.
• Perceive how Indexing and Looking work in Lucene.
• Be taught various kinds of Queries supported by Apache Lucene.
• Perceive find out how to construct a easy search software utilizing Lucene and Java.

This text was printed as part of the Information Science Blogathon.

What’s Apache Lucene?

To grasp Lucene in depth, there are a couple of key terminologies and ideas. Allow us to take a look at every considered one of them intimately together with examples. Think about an instance the place now we have the next details about three completely different merchandise in our assortment.

{
  "product_id": "1",
  "title": "Wi-fi Noise Cancelling Headphones",
  "model": "Bose",
  "class": ["Electronics", "Audio", "Headphones"],
  "value": 300
}

{
  "product_id": "2",
  "title": "Bluetooth Mouse",
  "model": "Jelly Comb",
  "class": ["Electronics", "Computer Accessories", "Mouse"],
  "value": 30
}

{
  "product_id": "3",
  "title": "Wi-fi Keyboard",
  "model": "iClever",
  "class": ["Electronics", "Computer Accessories", "Keyboard"],
  "value": 40
}

Doc

A doc is a elementary unit of indexing and search in Lucene. A doc ID identifies every doc. Lucene converts uncooked content material into paperwork containing fields and values.

Area

A Lucene doc comprises a number of fields. Every discipline has a reputation and a worth. See instance under.

• product_id
• title
• model
• class
• value

Time period

A time period is a unit of search in Lucene. Lucene does a number of pre-processing steps on uncooked content material earlier than creating phrases equivalent to tokenization and so forth.

Inverted Index

The underlying knowledge construction in Lucene that allows tremendous quick searches is the Inverted Index. In an inverted index, every time period maps to the paperwork that include it, together with the place of the time period in these paperwork. That is known as a Postings Checklist.

Section

A index could be sub-divided by Lucene into a number of segments. Every phase is an index in itself. Section searches are normally accomplished serially. 

Scoring

Lucene calculates the relevance of a doc by scoring mechanisms equivalent to Time period Frequency Inverse Doc Frequency (TF-IDF). There are additionally different scoring algorithms equivalent to BM25 which enhance upon TF-IDF. 

Now allow us to perceive how TF-IDF is calculated.

Time period Frequency (TF)

Time period frequency is the variety of occasions a time period t seems in a doc.

Doc Frequency (DF)

Doc frequency is the variety of paperwork that include a time period t. Inverse Doc Frequency divides the variety of paperwork within the assortment by the variety of paperwork containing the time period t. It measures the individuality of a selected time period to stop giving greater significance to repetitive phrases like “a,” “the,” and so forth. The “1+” is added to the denominator when the variety of paperwork containing the time period t is 0.

Time period Frequency Inverse Doc Frequency (TF-IDF)

 The TF-IDF is the product of Time period Frequency and Inverse Doc Frequency. A better worth of TF-IDF implies that the time period is extra distinguishing and distinctive in relevance to the entire assortment.

Elements of a Lucene Search Utility

Lucene comprises two main elements that are:

• Indexer – Lucene makes use of the IndexWriter class for indexing
• Searcher – Lucene makes use of the IndexSearcher class for looking.

Lucene Indexer

The Lucene Index is answerable for indexing paperwork for the search software. Lucene does a number of textual content processing and evaluation steps equivalent to tokenization earlier than indexing the phrases into an inverted index. Lucene makes use of the IndexWriter class for indexing.

The IndexWriter requires the specification of a listing the place the index shall be saved as effectively an analyzer for the uncooked content material. Though it’s fairly easy to write down your personal customized analyzer, Lucene’s StandardAnalyzer does an important job at this.

Listing listing = FSDirectory.open(Paths.get(INDEX_DIR));
IndexWriterConfig indexWriterConfig = new IndexWriterConfig(analyzer);
IndexWriter indexWriter = new IndexWriter(listing, indexWriterConfig);

Lucene Searcher

Lucene does search utilizing IndexSearcher class. The IndexSearcher class requires us to specify a sound Question object. A person question string could be transformed into a sound Question object utilizing the QueryParser class.

Upon specifying the utmost hits (aka search outcomes) we would like for the question, the Lucene searcher will return a TopDocs object which comprises the highest hits for the question. Every topDoc comprises a rating for every of the doc IDs retrieved. 

searcher = new IndexSearcher(listing);
parser = new QueryParser("question", new StandardAnalyzer());
Question question = parser.parse(searchString)
searcher.search(question, numHits)

Varieties of Search Queries Supported by Lucene

Lucene helps a number of completely different question sorts. Allow us to take a look at 5 mostly used queries together with examples.

Time period Question

A time period question matches paperwork that include a selected time period.

Question question = new TermQuery(new Time period("model", "jelly"));

Boolean Question

Boolean queries match paperwork that maintain true for a boolean mixture of different queries.

BooleanQuery.Builder builder = new BooleanQuery.Builder();
builder.add(new TermQuery(new Time period("class", "Laptop Equipment")), BooleanClause.Happen.SHOULD);
builder.add(new TermQuery(new Time period("model", "Jelly")), BooleanClause.Happen.SHOULD);
Question question = builder.construct();

Vary Question

Vary Queries match paperwork which include discipline values inside a variety. The instance under finds merchandise the place the value is between 30 and 50.

Question question = NumericRangeQuery.newIntRange("value", 30, 50, true, true);

Phrase Question

A phrase question matches paperwork containing a selected sequence of phrases. 

Question question = new PhraseQuery("title", "Noise", "Cancelling");

Perform Question

Calculates scores for paperwork primarily based on a operate of the worth of a discipline. Perform Question can be utilized to spice up the rating of outcomes primarily based on a discipline within the doc. 

Question question = new FunctionQuery(new FloatFieldSource("value"));

Constructing a Easy Search Utility with Lucene

To date, now we have realized about Lucene fundamentals, indexing, looking, and the varied question sorts you should use.

Allow us to now tie all these bits collectively right into a sensible instance the place we construct a easy search software utilizing the core components of Lucene: Indexer and Searcher. 

Within the instance under, we index 3 paperwork the place every doc comprises the next fields.

Identify is added as a textual content discipline and E-mail is added as a string discipline. String fields don’t get tokenized by Lucene.

import org.apache.lucene.evaluation.Analyzer;
import org.apache.lucene.doc.Doc;
import org.apache.lucene.doc.Area;
import org.apache.lucene.doc.StringField;
import org.apache.lucene.doc.TextField;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.retailer.Listing;

import java.io.IOException;

public class MyIndexer {
    personal Listing indexDirectory;
    personal static closing String NAME = "title";
    personal static closing String EMAIL = "electronic mail";
    personal Analyzer analyzer;

    public MyIndexer(Listing listing, Analyzer analyzer) {
        this.indexDirectory = listing;
        this.analyzer = analyzer;
    }

    public void indexDocuments() throws IOException {
        IndexWriterConfig indexWriterConfig = new IndexWriterConfig(analyzer);
        IndexWriter indexWriter = new IndexWriter(indexDirectory, indexWriterConfig);
        indexNewDocument(indexWriter, "john", "[email protected]");
        indexNewDocument(indexWriter, "jane", "[email protected]");
        indexNewDocument(indexWriter, "ana", "[email protected]");
        indexWriter.shut();
    }

    public void indexNewDocument(IndexWriter indexWriter, String title, String electronic mail) throws IOException {
        Doc doc = new Doc();
        doc.add(new TextField(NAME, title, Area.Retailer.YES));
        doc.add(new StringField(EMAIL, electronic mail, Area.Retailer.YES));
        indexWriter.addDocument(doc);
    }
}

As soon as the paperwork are listed, we are able to question them utilizing Lucene queries. Within the instance under, we use a easy TermQuery to seek out and print the paperwork that match the time period “jane”.

import org.apache.lucene.evaluation.Analyzer;
import org.apache.lucene.evaluation.commonplace.StandardAnalyzer;
import org.apache.lucene.doc.Doc;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.Time period;
import org.apache.lucene.search.*;
import org.apache.lucene.retailer.Listing;
import org.apache.lucene.retailer.FSDirectory;

import java.io.IOException;
import java.nio.file.Paths;

public class SimpleSearchApplication {
    public static void primary(String[] args) throws IOException {
        String INDEX_DIRECTORY = "listing";
        Listing indexDirectory = FSDirectory.open(Paths.get(INDEX_DIRECTORY));
        Analyzer analyzer = new StandardAnalyzer();
        MyIndexer indexer = new MyIndexer(indexDirectory, analyzer);
        indexer.indexDocuments();

        // Search on the listed paperwork
        IndexReader indexReader = DirectoryReader.open(indexDirectory);
        IndexSearcher searcher = new IndexSearcher(indexReader);

        // Assemble a Time period question to seek for the title "jane"
        Question question = new TermQuery(new Time period("title", "jane"));
        int maxHits = 10;

        TopDocs searchResults = searcher.search(question, maxHits);

        System.out.println("Paperwork with title 'jane':");
        for (ScoreDoc scoreDoc : searchResults.scoreDocs) {
            Doc doc = searcher.doc(scoreDoc.doc);
            System.out.println("title: " + doc.get("title") + ", electronic mail: " + doc.get("electronic mail"));
        }
        indexReader.shut();
    }
}

The above code returns the next consequence:

Paperwork with title 'jane':
title: jane, electronic mail: [email protected]

Conclusion

Apache Lucene is a sturdy search library that allows the event of high-performance search functions. With the introduction of Lucene 9.9, vital enhancements in question analysis, vector search, and different options have enhanced its capabilities. All through this information, we’ve lined the basic elements of Lucene, the workings of indexers and searchers, and find out how to construct a easy search software in Java. Moreover, we explored the varied kinds of search queries supported by Lucene. Armed with this information, you must now really feel assured in your understanding of Lucene and be able to create extra superior search functions using its highly effective options.

Key Takeaways 

• Apache Lucene is a strong Java library that may carry out tremendous quick full-text searches.
• Lucene helps numerous question sorts that cater to completely different search use instances.
• Lucene kinds the spine of a number of excessive efficiency search functions equivalent to Elasticsearch, Solr, Nrtsearch and so forth.
• Lucene IndexWriter and IndexSearcher are necessary lessons that allow quick indexing and looking.

Often Requested Questions

The media proven on this article shouldn’t be owned by Analytics Vidhya and is used on the Creator’s discretion.