Enhancing RAG Methods with Nomic Embeddings

The intersection of synthetic intelligence and information processing has developed considerably with the rise of multimodal Retrieval-Augmented Era programs. Multimodal RAG goes past conventional fashions that focus solely on textual content. It integrates numerous information sorts like textual content, photographs, audio, and video. This enables for extra nuanced and context-aware responses. A key innovation is Nomic imaginative and prescient embeddings. They create a unified house for each visible and textual information. This allows seamless interplay throughout totally different codecs. Through the use of superior fashions to generate high-quality embeddings, multimodal RAG improves info retrieval. It bridges the hole between totally different content material varieties. The result’s richer and extra informative consumer experiences.

Studying Goals

• Perceive the basics of multimodal Retrieval-Augmented Era programs and their benefits over conventional RAG.
• Discover the position of Nomic Imaginative and prescient Embeddings in making a unified embedding house for textual content and pictures.
• Evaluate Nomic Imaginative and prescient Embeddings with CLIP fashions and analyze their efficiency benchmarks.
• Implement a multimodal RAG system in Python utilizing Nomic Imaginative and prescient and Textual content Embeddings.
• Learn to extract and course of textual and visible information from PDFs for multimodal retrieval.

This text was printed as part of the Knowledge Science Blogathon.

What’s Multimodal RAG?

Multimodal RAG represents a major development in synthetic intelligence. It’s construct upon conventional RAG programs by incorporating numerous information sorts equivalent to textual content, photographs, audio, and video. In contrast to typical RAG programs that primarily course of textual info, multimodal RAG is designed to deal with and combine a number of types of information concurrently. This functionality permits for extra complete understanding and era of responses which can be context-aware throughout totally different modalities.

Key Elements of Multimodal RAG

• Knowledge Ingestion: The method begins with ingesting numerous sorts of information via specialised processors for every format. This ensures that the system can validate, clear, and normalize incoming information whereas preserving its important traits
• Vector Illustration: Completely different modalities are processed utilizing respective neural networks (e.g., CLIP for photographs or BERT for textual content) to generate unified vector representations or embeddings. These embeddings preserve semantic relationships throughout totally different modalities.
• Vector Database Storage: The generated embeddings are saved in vector databases optimized with indexing strategies like HNSW or FAISS for environment friendly retrieval
• Question Processing: Incoming queries are analyzed and reworked into the identical vector house because the saved information to find out related modalities and generate applicable embeddings for search

Nomic Imaginative and prescient Embeddings

A big innovation on this discipline of multimodal embeddings is the incorporation of Nomic imaginative and prescient embeddings, which create a cohesive embedding house for each visible and textual information. 

Nomic Embed Imaginative and prescient v1 and v1.5 are each high-quality imaginative and prescient embedding fashions developed by Nomic AI, designed to share the identical latent house as their corresponding textual content embedding fashions, Nomic Embed Textual content v1 and v1.5, respectively. It operates inside the identical house as Nomic Embed Textual content, making it well-suited for multimodal duties equivalent to text-to-image retrieval. With a imaginative and prescient encoder comprising solely 92M parameters, Nomic Embed Imaginative and prescient is well-suited for high-volume manufacturing purposes, complementing the 137M parameters of Nomic Embed Textual content.

CLIP fashions undergo in unimodal duties

Multimodal fashions equivalent to CLIP exhibit exceptional zero-shot capabilities throughout totally different modalities. Nevertheless, CLIP’s textual content encoders wrestle with duties past picture retrieval, as seen in benchmarks like MTEB, which evaluates the effectiveness of textual content embedding fashions. Nomic Embed Imaginative and prescient goals to handle these limitations by aligning a imaginative and prescient encoder with the prevailing Nomic Embed Textual content latent house.

To deal with the difficulty of underperformance on unimodal duties, equivalent to semantic similarity, Nomic Embed Imaginative and prescient, a imaginative and prescient encoder, was skilled alongside Nomic Embed Textual content, a long-context textual content encoder. The coaching methodology concerned freezing the textual content encoder and coaching the imaginative and prescient encoder on image-text pairs. This method not solely produced optimum outcomes but in addition ensured backward compatibility with the embeddings from Nomic Embed Textual content.

Efficiency Benchmarks of Nomic Imaginative and prescient Embeddings

As talked about earlier, current multimodal fashions equivalent to CLIP exhibit spectacular zero-shot capabilities throughout totally different modalities. Nevertheless, the efficiency of CLIP’s textual content encoders is subpar outdoors of duties like picture retrieval, as evidenced by benchmarks like MTEB, which evaluates the standard of textual content embedding fashions. Nomic Embed Imaginative and prescient is particularly designed to handle these shortcomings by aligning a imaginative and prescient encoder with the prevailing Nomic Embed Textual content latent house. This alignment leads to a unified multimodal latent house that delivers robust efficiency on picture, textual content, and multimodal duties, as demonstrated by the Imagenet Zero-Shot, MTEB, and Datacomp benchmarks.

Arms on Python Implementation of MultiModal RAG with Nomic Imaginative and prescient Embeddings

On this tutorial, we’ll construct a multimodal RAG system that may effectively retrieve info from a PDF containing each textual and visible content material. We’ll construct this on Google Colab utilizing T4 GPU (Free tier).

Step 1: Putting in Crucial Libraries

Set up all required Python libraries, together with OpenAI, Qdrant, Transformers, Torch, and PyMuPDF.

!pip set up openai==1.55.3 httpx==0.27.2 
!pip set up qdrant_client
!pip set up transformers
!pip set up transformers torch pillow
!pip set up --upgrade nltk
!pip set up sentence-transformers
!pip set up --upgrade qdrant-client fastembed Pillow
!pip set up PyMuPDF

Step 2: Setting OpenAI API key and Importing Crucial Libraries

Arrange the OpenAI API key and import important libraries like PyMuPDF, PIL, LangChain, and OpenAI.

from openai import ChatCompletion
import openai
import os
from openai import AzureOpenAI
from PIL import Picture
import torch
import numpy as np
import fitz  # PyMuPDF
import os
import time
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI
from openai import ChatCompletion
import openai
import base64
from base64 import b64decode

os.environ["OPENAI_API_KEY"] = ''

Arrange the OpenAI API key and import important libraries like PyMuPDF, PIL, LangChain, and OpenAI.

#photographs

def extract_images_from_pdf(pdf_path, output_folder):
    pdf_document = fitz.open(pdf_path)
    os.makedirs(output_folder, exist_ok=True)
    #Iterating throught the pages within the PDF
    for page_number in vary(len(pdf_document)):
        web page = pdf_document[page_number]
        #Operate For Getting Photos From the PDF Pages
        photographs = web page.get_images(full=True)

        for image_index, img in enumerate(photographs):
            xref = img[0]
            base_image = pdf_document.extract_image(xref)
            image_bytes = base_image["image"]
            image_ext = base_image["ext"]
            image_filename = f"page_{page_number+1}_image_{image_index+1}.{image_ext}"
            image_path = os.path.be part of(output_folder, image_filename)
            with open(image_path, "wb") as image_file:
                image_file.write(image_bytes)
    pdf_document.shut()

Use PyMuPDF to extract textual content from all pages of the PDF and retailer it in a listing.

def extract_text_pdf(path):
    """Extracts textual content from a PDF utilizing PyMuPDF."""
    doc = fitz.open(path)
    text_results = []
    for web page in doc:
        textual content = web page.get_text()
        text_results.append(textual content)
    return text_results

Step 5: Saving Extracted Textual content and Photos From PDF

Save photographs within the “take a look at” listing and extract textual content for additional processing.

def get_contents(pdf_path, output_directory):
  """Extracts textual content and pictures from a PDF, saves photographs, and returns textual content and elapsed time."""

  extract_images_from_pdf(pdf_path, output_directory)
  text_results=extract_text_pdf(pdf_path)
  return(text_results)
  
pdf_path = "/content material/retailcoffee.pdf"
output_directory = "/content material/take a look at"
text_results=get_contents(pdf_path, output_directory)

We use this PDF that has each textual content and pictures or charts to check the multimodal RAG. 

We save the photographs extracted from the PDF utilizing the PyMuPDF library within the “take a look at” listing. Within the subsequent steps, create embeddings of those photographs in order to have the ability to retrieve info from them in future primarily based on a consumer question.

Step 6. Chunking Textual content Knowledge For RAG

Cut up extracted textual content into smaller chunks utilizing LangChain’s RecursiveCharacterTextSplitter.

text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=2048,
        chunk_overlap=50,
        length_function=len,
        is_separator_regex=False,
        separators=[
            "nn",
            "n",
            " ",
            ".",
            ",",
            "u200b",  # Zero-width space
            "uff0c",  # Fullwidth comma
            "u3001",  # Ideographic comma
            "uff0e",  # Fullwidth full stop
            "u3002",  # Ideographic full stop
            "",
        ],
    )

doc_texts = text_splitter.create_documents(text_results)

Step 7: Loading Nomic Textual content Embedding Mannequin and Nomic Imaginative and prescient Embedding Mannequin

Load Nomic’s textual content and imaginative and prescient embedding fashions utilizing Hugging Face’s Transformers library.

from transformers import AutoTokenizer, AutoModel

# Load the tokenizer and mannequin
text_tokenizer = AutoTokenizer.from_pretrained("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)
text_model = AutoModel.from_pretrained("nomic-ai/nomic-embed-text-v1.5", trust_remote_code=True)

def text_embeddings(textual content):
    inputs = text_tokenizer(textual content, return_tensors="pt", padding=True, truncation=True)
    outputs = text_model(**inputs)
    embeddings = outputs.last_hidden_state.imply(dim=1)
    return embeddings[0].detach().numpy()
    
from transformers import AutoModel, AutoProcessor
from PIL import Picture
import torch
mannequin = AutoModel.from_pretrained("nomic-ai/nomic-embed-vision-v1.5", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("nomic-ai/nomic-embed-vision-v1.5")

Step 8: Producing Textual content and Picture Embeddings For Our Knowledge

Convert textual content and pictures into vector embeddings for environment friendly retrieval.

#Textual content Embeddins
texts_embeded = [text_embeddings(document.page_content) for document in doc_texts]

#Picture Embeddings
image_embeddings = []
for img in image_files:
    strive:
        picture = Picture.open(os.path.be part of(output_directory, img))
        inputs = processor(photographs=picture, return_tensors="pt")
        with torch.no_grad():
            outputs = mannequin(**inputs)
        embeddings = outputs.last_hidden_state
        if embeddings.measurement(0) > 0:  # Make sure the batch measurement is non-zero

            image_embedding = embeddings.imply(dim=1).squeeze().cpu().numpy()
            image_embeddings.append(image_embedding)
        else:
            print(f"No Embeddings For {img}")

    besides Exception as e:
        print(e)

#SIZE OF Textual content & Picture Embeddings
text_embeddings_size=len(texts_embeded[0])
image_embeddings_size=len(image_embeddings[0])

Step 9: Storing Textual content Embeddings in Qdrant 

Qdrant is an open-source vector database and search engine designed to effectively retailer, handle, and question high-dimensional vectors.We save our embeddings on this vector DB.

from qdrant_client import QdrantClient, fashions

consumer = QdrantClient(":reminiscence:")

if not consumer.collection_exists("text1"): #making a Assortment
 consumer.create_collection(
        collection_name ="text1",
      vectors_config=fashions.VectorParams(
        measurement=text_embeddings_size,  # Vector measurement is outlined by used mannequin
        distance=fashions.Distance.COSINE,
    ),
 )
 
 consumer.upload_points(
    collection_name="text1",
    factors=[
        models.PointStruct(
            id=str(uuid.uuid4()),
            vector=np.array(texts_embeded[idx]),
            payload={
                "metadata": doc.metadata,
                "content material": doc.page_content
            }
        )
        for idx, doc in enumerate(doc_texts)
    ]
)

Step 10: Storing Picture Embeddings in Qdrant 

Save picture embeddings in a separate Qdrant assortment for multimodal retrieval.

if not consumer.collection_exists("images1"):
    consumer.create_collection(
        collection_name="images1",
        vectors_config=fashions.VectorParams(
        measurement=image_embeddings_size,  # Vector measurement is outlined by used mannequin
        distance=fashions.Distance.COSINE,
    ),
  )
  
# Make sure that image_embeddings aren't empty
if len(image_embeddings) > 0:
    consumer.upload_points(
        collection_name="images1",
        factors=[
            models.PointStruct(
                id=str(uuid.uuid4()),  # unique id
                vector= np.array(image_embeddings[idx])  ,
                payload={"image_path": output_directory+'/'+str(image_files[idx])}  # Picture path as metadata
            )
            for idx in vary(len(image_embeddings))  
    )
else:
    print("No embeddings discovered")

Step 11: Making a MultiModal Retriever For Retrieving Photos and Textual content

Retrieve probably the most related textual content and picture embeddings primarily based on a consumer question.

def MultiModalRetriever(question):

    question = text_embeddings(question)

    # Retrieve textual content hits
    text_hits = consumer.query_points(
        collection_name="text1",
        question=question,
        restrict=3,3
    ).factors
    # Retrieve picture hits
    Image_hits = consumer.query_points(
        collection_name="images1",
        question=question,
        restrict=5,
    ).factors

    return text_hits, Image_hits

Step 12: Making a MultiModal RAG utilizing LangChain

Use LangChain to course of retrieved textual content and pictures, producing context-aware responses utilizing GPT-4o.

def MultiModalRAG(context,photographs,user_query,mannequin):  
    # Helper operate to encode a picture as a base64 string
    def encode_image(image_path):
        if image_path:
            with open(image_path, "rb") as image_file:
                return base64.b64encode(image_file.learn()).decode()
        return None


    image_paths = photographs   
    #three photographs primarily based on retrived photographs
    img_base64 = encode_image(image_paths[0])        
    img_base641 = encode_image(image_paths[1])  
    img_base642 = encode_image(image_paths[2])  

    message = HumanMessage(
            content material=[
                {"type": "text", "text": "BASED ON RETRIEVED CONTEXT %s ONLY, ANSWER THE FOLLOWING QUERY %s. Context can be tables, texts or Images"%(context,user_query)},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{img_base64}"},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{img_base641}"},
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{img_base642}"},
                },
            ],)

    mannequin = ChatOpenAI(mannequin=mannequin)    
    response = mannequin.invoke([message])
    return response.content material


def RAG(question):
  text_hits, Image_hits=MultiModalRetriever(question)

  retrieved_images=[i.payload['image_path'] for i in Image_hits]
  print(retrieved_images)
  reply=MultiModalRAG(text_hits,retrieved_images,question,"gpt-4o")
  return(reply)

Querying the Mannequin

Allow us to now question our multimodal RAG system with totally different queries to check its multimodal functionality, 

RAG("Income of Starbucks in billion {dollars} of Meals in 2020?")

Output:

'Primarily based on the chart exhibiting Starbucks' income by product for 2020, the income from
meals is roughly $3 billion.'

The response to this question is barely current within the following chart (Fig 4) within the PDF and never in any textual content. So our, multimodal RAG is ready to retrieve this info precisely.

RAG("Clarify what the Ansoff Matrix is for Starbucks.")

Output:

'The Ansoff Matrix is a strategic software that helps companies like Starbucks analyze
their development methods. For Starbucks, it may be damaged down as follows: 
1. **Market Penetration:** Starbu cks focuses on growing gross sales of current
merchandise in present markets. This contains enhancing the client expertise, leveraging their cellular app for comfort, and selling current choices.
2. **Product Improvement:** Starbucks introduces new merchandise for current markets. Examples embody launching new beverage choices or introducing meatless breakfast
gadgets to adapt to altering shopper preferences.
3. **Market Improvement:** This includes Starbucks increasing into new geographical
areas or market segments with current merchandise. It selects high-traffic
areas and creates a constant model picture and retailer expertise to draw prospects.
4. **Diversification:** Introducing totally new merchandise to new markets. This might
contain Starbuck s exploring areas like providing alcoholic drinks to draw 
totally different buyer demographics. 
Total, the Ansoff Matrix helps Starbucks strategically plan tips on how to develop and adapt
in numerous market situations by specializing in both present or new merchandise and 
markets.

The response to this question as effectively is barely current within the following diagram (Fig 3) within the PDF and never in any textual content. So our, multimodal RAG is ready to retrieve this info precisely.

RAG("World espresso consumption in 2017")

Output:

'The worldwide espresso consumption in 2017 was 161.37 million luggage.'

The response to this question as effectively is barely current within the following chart (Fig 1) within the PDF and never in any textual content. So our, multimodal RAG is ready to retrieve this info precisely.

Conclusion

The mixing of Nomic imaginative and prescient embeddings into multimodal RAG programs represents a significant leap in AI, permitting seamless interplay between visible and textual information for enhanced understanding and response era. By overcoming limitations seen in fashions like CLIP, Nomic Embed Imaginative and prescient affords a unified embedding house, boosting efficiency on multimodal duties. This growth paves the best way for richer, extra context-aware consumer experiences in high-volume manufacturing environments.

Key Takeaways

• Multimodal Retrieval-Augmented Era (RAG) programs combine numerous information sorts, equivalent to textual content, photographs, audio, and video, enabling extra context-aware and nuanced outputs in comparison with conventional RAG programs targeted on textual content alone.
• Nomic imaginative and prescient embeddings play a key position by unifying visible and textual information right into a single embedding house, enhancing the system’s potential to retrieve and synthesize info throughout a number of modalities.
• The multimodal RAG system processes information via specialised ingestion, vector illustration, and storage strategies, guaranteeing environment friendly retrieval and significant responses throughout numerous content material codecs.
• Whereas CLIP fashions excel in zero-shot capabilities, they wrestle with unimodal duties like semantic similarity. Nomic Embed Imaginative and prescient addresses this by aligning imaginative and prescient and textual content encoders, bettering efficiency on a variety of duties.

Steadily Requested Questions

The media proven on this article will not be owned by Analytics Vidhya and is used on the Writer’s discretion.

Nibedita accomplished her grasp’s in Chemical Engineering from IIT Kharagpur in 2014 and is at the moment working as a Senior Knowledge Scientist. In her present capability, she works on constructing clever ML-based options to enhance enterprise processes.