Contextual Retrieval for Multimodal RAG on Slide Decks

Construct and Persist the Index

We’ll now embed these summarized chunks and retailer them in a vector retailer for retrieval. LlamaIndex can persist indices regionally or combine with 40+ exterior vector databases.

import os
from llama_index.core import (
    StorageContext,
    VectorStoreIndex,
    load_index_from_storage,
)

if not os.path.exists("storage_nodes_iconiq"):
    index = VectorStoreIndex(new_text_nodes, embed_model=embed_model)
    index.set_index_id("vector_index")
    index.storage_context.persist("./storage_nodes_iconiq")
else:
    storage_context = StorageContext.from_defaults(persist_dir="storage_nodes_iconiq")
    index = load_index_from_storage(storage_context, index_id="vector_index")

retriever = index.as_retriever()

Baseline Index (With out Summaries)

We’ll additionally construct a “baseline” index on the unique textual content nodes (with out the contextual summaries) to check the distinction in retrieval high quality.

if not os.path.exists("storage_nodes_iconiq_base"):
    base_index = VectorStoreIndex(text_nodes, embed_model=embed_model)
    base_index.set_index_id("vector_index")
    base_index.storage_context.persist("./storage_nodes_iconiq_base")
else:
    storage_context = StorageContext.from_defaults(
        persist_dir="storage_nodes_iconiq_base"
    )
    base_index = load_index_from_storage(storage_context, index_id="vector_index")

Construct a Multimodal Question Engine

We would like a RAG pipeline that:

• Retrieves related chunks of textual content.
• Additionally masses the web page photos.
• Sends each textual content chunks and pictures to a multimodal LLM (right here we illustrate utilizing an OpenAI-like GPT-4 multimodal endpoint, labeled gpt-4o).

import base64
import openai
import os
from typing import Optionally available, Record

from llama_index.core.query_engine import CustomQueryEngine
from llama_index.core.base.response.schema import Response
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.prompts import PromptTemplate
from llama_index.core.schema import NodeWithScore, MetadataMode

QA_PROMPT_TMPL = """
Under we give parsed textual content from slides, in addition to photos.

---------------------
{context_str}
---------------------

Given the context info and no prior information, please reply the question:

Question: {query_str}
Reply:
"""

QA_PROMPT = PromptTemplate(QA_PROMPT_TMPL)

def encode_image(image_path: str) -> str:
    """If you wish to inline an area picture in base64."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.learn()).decode("utf-8")


class MultimodalQueryEngine(CustomQueryEngine):
    """
    Customized multimodal Question Engine that retrieves textual content nodes,
    then sends them + picture(s) to the brand new Imaginative and prescient-capable API as documented.
    """

    def __init__(
        self,
        retriever: BaseRetriever,
        model_name: str = "gpt-4o",
        qa_prompt: Optionally available[PromptTemplate] = None,
    ) -> None:
        tremendous().__init__(qa_prompt=qa_prompt or QA_PROMPT)
        self.retriever = retriever
        self.model_name = model_name

    def custom_query(self, query_str: str) -> Response:
        # 1) Retrieve textual content nodes
        node_with_scores: Record[NodeWithScore] = self.retriever.retrieve(query_str)

        # 2) Construct context
        context_str = "nn".be part of(
[nws.node.get_content(metadata_mode=MetadataMode.LLM) for nws in node_with_scores]

        )

        # 3) Format the ultimate immediate
        formatted_prompt_text = self._qa_prompt.format(
            context_str=context_str,
            query_str=query_str,
        )

        # 4) Construct the consumer message with textual content + photos
        user_message_content = [
            {
                "type": "text",
                "text": formatted_prompt_text,
            }
        ]

        for nws in node_with_scores:
            image_path = nws.node.metadata.get("image_path", "")
            if image_path:
                base64_data = encode_image(image_path)
                image_url = f"information:picture/jpeg;base64,{base64_data}"
                user_message_content.append(
                    {
                        "kind": "image_url",
                        "image_url": {
                            "url": image_url,
                            "element": "auto"
                        },
                    }
                )

        messages = [
            {
                "role": "user",
                "content": user_message_content,
            }
        ]

        # 5) Name your Imaginative and prescient mannequin
        response = openai.ChatCompletion.create(
            mannequin=self.model_name,
            messages=messages,
            max_tokens=500,
        )

        # 6) Return a Response object
        return Response(
            response=response.selections[0].message.content material,
            source_nodes=node_with_scores,
            metadata={},
        )
        
        # 2) Create a question engine
query_engine = MultimodalQueryEngine(
    retriever=index.as_retriever(similarity_top_k=3),
    model_name="gpt-4o",   # or "gpt-4o-mini", "gpt-4-turbo", and so on.
)

base_query_engine = MultimodalQueryEngine(
    retriever=base_index.as_retriever(similarity_top_k=3),
    model_name="gpt-4o",
)

Attempting Out Queries

Let’s question our new pipeline about AI utilization by division.

response = query_engine.question(
    "Which departments use GenAI essentially the most and the way are they utilizing it?"
)
print(str(response))

A typical response may appear like this:

Based mostly on the parsed markdown textual content supplied, the departments/groups that use 
generative AI essentially the most are:
1. **AI, Machine Studying, and Information Science** with a rating of 4.5.
2. **IT** with a rating of 4.0.
3. **Engineering / R&D** with a rating of three.9.
These scores are derived from a survey the place respondents rated the extent of
generative AI utilization on a scale of 1-5.
By way of how these departments are utilizing generative AI:
- **AI, Machine Studying, and Information Science**: Whereas particular use instances for this
division are usually not detailed within the supplied textual content, it may be inferred that they're
doubtless utilizing generative AI for superior information evaluation, mannequin growth, and
enhancing AI capabilities inside the group.
- **IT**: The IT division is utilizing generative AI for a number of impactful use instances,
together with:
- Ticket administration
- Chatbots
- Buyer help and troubleshooting
- Information administration
- Case summarization
The details about the departments and their use instances comes from the parsed
markdown textual content. There aren't any discrepancies between the parsed markdown and the
context supplied, because the markdown textual content clearly outlines each the departments with
the best utilization scores and the precise use instances for the IT division.

Comparatively, if we run the identical question on the baseline index:

base_response = base_query_engine.question(
    "Which departments use GenAI essentially the most and the way are they utilizing it?"
)
print(str(base_response))

You’ll see the baseline may need fewer particulars or barely totally different retrieval outcomes. Contextual retrieval offers extra exact context across the IT utilization particularly. The response would appear like:

Based mostly on the parsed markdown textual content supplied, the departments that use Generative AI
(GenAI) essentially the most are:
1. **AI, Machine Studying, and Information Science** - This division has the best 
weighted common rating of 4.5 for GenAI utilization, indicating important adoption. The
particular use instances are usually not detailed within the parsed textual content, however given the character of the
division, it's doubtless concerned in creating and refining AI fashions and
algorithms.
2. **IT** - With a rating of 4.0, the IT division can be a number one consumer of GenAI.
The use instances for IT embrace inner productiveness enhancements and IT operations,
as indicated by the 61% adoption fee for inner productiveness and 42% ROI point out
in IT use instances.
3. **Engineering / R&D** - This division has a rating of three.9. Whereas particular use
instances are usually not detailed within the parsed textual content, it's affordable to deduce that GenAI is
used for product growth and analysis functions, as advised by the 69% 
adoption fee for core product efficiency enhancements and 50% for pure language
interfaces.
The knowledge is derived from the parsed markdown textual content, which offers an in depth
breakdown of GenAI utilization by division and particular use instances. There aren't any 
discrepancies between the parsed markdown and the uncooked textual content, because the markdown seems
to be a structured illustration of the identical information. The picture was not supplied, so
it was not utilized in forming the reply.

Observing the Advantages of Contextual Retrieval

Right here’s one other instance question, On this subsequent query, the identical sources are retrieved with and with out contextual retrieval, and the reply is right for each approaches. That is thanks for LlamaParse Premium’s capability to understand graphs.

question = "what are related insights from the 'deep dive on infrastructure' part when it comes to mannequin preferences, value, deployment environments?"

response = query_engine.question(question)
print(str(response))

Output

The "Deep Dive on Infrastructure" part from the ICONIQ Progress report offers
insights into the infrastructure points crucial for deploying AI options. 
Nevertheless, the parsed markdown textual content doesn't explicitly point out mannequin preferences or
prices on this part. As a substitute, it focuses on infrastructure tooling and deployment
environments.
From the parsed markdown textual content, we are able to collect the next insights associated to 
deployment environments:
1. **Deployment Environments**: Enterprises are primarily internet hosting generative AI
workloads on the cloud or utilizing a hybrid strategy. The popular deployment strategies
are:
- Cloud: 56%
- Hybrid: 42%
- On-prem: 2%
2. **Cloud Service Suppliers**: Probably the most utilized cloud service suppliers for
internet hosting AI workloads are:
- Amazon Internet Companies (AWS): 68%
- Microsoft Azure: 61%
- Google Cloud (GCP): 40%
These insights are derived from the parsed markdown textual content, particularly from the 
sections discussing "Cloud Deployment Methodology" and "Infrastructure Tooling." There's
no point out of mannequin preferences or value issues within the supplied textual content. If
there have been any discrepancies or further particulars within the picture or uncooked textual content, they
are usually not out there right here, so the reply is predicated solely on the parsed markdown textual content
supplied.

Now, lets strive with the baseline strategy:

base_response = base_query_engine.question(question)
print(str(base_response))

Output

The parsed textual content from the slides doesn't present particular insights relating to mannequin preferences, value, or deployment environments within the 'deep dive on infrastructure' part. The slide titled "Deep Dive on Infrastructure" (web page 24) solely incorporates the title, the ICONIQ Progress branding, and confidentiality and copyright notices. There isn't any detailed info or information offered within the parsed textual content for this part.

Subsequently, based mostly on the parsed markdown textual content supplied, there aren’t any related insights out there from the ‘deep dive on infrastructure’ part relating to mannequin preferences, value, or deployment environments. If there have been any photos related to this part, they weren’t supplied, and thus no further insights may very well be derived from them.

This conclusion is drawn from the parsed markdown textual content, which lacks any particular info on mannequin preferences, value, or deployment environments in that part. The picture confirms this, because it solely reveals the title and a graphic with out further particulars.

When you want insights on these subjects, you may wish to seek advice from different sections or slides that particularly handle mannequin preferences, prices, or deployment environments.

• Contextual Retrieval may fetch the pages that debate cloud deployment strategies, infrastructure tooling, and price references, resulting in a extra thorough response.
• The baseline strategy may (in some instances) fail to retrieve the right chunk or present much less element.

Evaluating each solutions helps reveal that these quick “contextual summaries” in your metadata typically result in extra related retrieval.

An enormous due to Jerry Liu from LlamaIndex for creating this wonderful pipeline.

Conclusion

On this tutorial, we explored the method of parsing a PDF slide deck utilizing LlamaParse to extract each textual content and pictures, enriching every textual content chunk with contextual summaries to reinforce retrieval accuracy. We demonstrated easy methods to construct a Multimodal RAG pipeline with LlamaIndex, integrating each textual and visible information into a robust mannequin like GPT-4, showcasing the potential of multimodal LLMs. Lastly, we in contrast outcomes from a baseline index to a contextual index, highlighting the numerous enhancements in retrieval precision and relevance achieved by means of the contextual strategy. This complete information equips you with the instruments and strategies to construct efficient multimodal AI options.

Key Takeaways

• Contextual retrieval improves chunk matching for queries that may not have a direct key phrase overlap.
• Multimodal RAG can incorporate not simply textual content but in addition photos, charts, or diagrams from slides.
• Immediate caching is important when chunk sizes are giant and also you’re producing a context abstract for every chunk—this could cut back value considerably.
• When you have web-based content material (like retailer listings, giant units of HTML pages), you should use ScrapeGraphAI to fetch that information, then feed it into the identical pipeline.

With these steps, you’ll be able to adapt the strategy to any PDF or exterior information supply—whether or not it’s an enormous enterprise information base, advertising and marketing supplies, or your organization’s inner documentation.

Steadily Requested Questions

This text was revealed as part of the Information Science Blogathon.

Hello! I am Adarsh, a Enterprise Analytics graduate from ISB, presently deep into analysis and exploring new frontiers. I am tremendous captivated with information science, AI, and all of the revolutionary methods they’ll remodel industries. Whether or not it is constructing fashions, engaged on information pipelines, or diving into machine studying, I like experimenting with the most recent tech. AI is not simply my curiosity, it is the place I see the long run heading, and I am at all times excited to be part of that journey!