Retrieval-Augmented Ranking: A Key Technique for High-Performing RAGs

Retrieval-augmented generation (RAG) models are a class of natural language processing models that leverage both retrieval and generation capabilities. They aim to produce high-quality text outputs by first retrieving relevant information from a large corpus of documents or knowledge sources, and then using that retrieved information to condition and augment the text generation process. The first step in a typical RAG model pipeline involves a retrieval component that encodes the input query and documents into dense vector representations. These vectors are then compared to find the top most relevant documents based on their similarity to the query vector. However, this initial retrieval step may not always surface the truly most relevant documents due to limitations in the vector encoding and similarity matching process.

The typical RAG process is as follows:

• 1. The user asks a question or provides an instruction.
• 2. The system queries a vector database to find information relevant to the user’s question or instruction (“retrieval”).
• 3. The user’s prompt and any relevant information from the vector database are supplied to the language model (“augmentation”).
• 4. The language model uses the information from the database to answer the user’s prompt (“generation”).

Normally we use cosine similarly vector search when retrieving the related context from the Knowledge base or embedding.

• Calculate the dot product of the two vectors: The dot product is calculated by multiplying the corresponding elements of the vectors and summing them up.
  • dot_product = sum(a * b for a, b in zip(A, B))
• Calculate the L2 norm (Euclidean length) of each vector: The L2 norm is calculated by taking the square root of the sum of the squared elements of the vector.
• norm_A = sum(x ** 2 for x in A) ** 0.5
• norm_B = sum(x ** 2 for x in B) ** 0.5
• Calculate the cosine similarity: The cosine similarity is calculated as the dot product of the two vectors divided by the product of their L2 norms.
  • cosine_similarity = dot_product / (norm_A * norm_B)

The cosine similarity metric doesn’t capture potential information loss or information mismatch between vectors during the initial encoding stage. It operates on the assumption that the vectors being compared accurately represent the underlying data or text. However, in practice, the process of encoding text or data into dense vectors using techniques like word embeddings or language models can introduce information loss or distortion. This can lead to situations where the cosine similarity between two vectors may not accurately reflect the true semantic similarity or relevance between the original text or data points.

Let’s consider an example to understand this issue: 

Suppose we have two sentences: 

“The quick brown fox jumps over the lazy dog.”

 “A quick brown fox jumped over the sleeping dog.” 

If we encode these sentences into dense vectors (let’s say 512-dimensional vectors) using a pre-trained language model, the resulting vectors may not capture the subtle differences in tense and article usage (“a” vs. “the”) due to the inherent limitations of the encoding process. Let’s represent the encoded vectors as 

Sentence 1: v1 = [0.1, 0.2, -0.3, …, 0.2] 

Sentence 2: v2 = [0.15, 0.25, -0.28, …, 0.18] 

Now, if we calculate the cosine similarity between these two vectors: 

cosine_similarity(v1, v2) = (v1 · v2) / (||v1|| * ||v2||) 

The cosine similarity value may be very high, let’s say 0.98, indicating that these two sentences are highly similar according to the vector representations. However, if we look at the original sentences, we can see that there are subtle differences in tense and article usage, which may be important for certain applications or downstream tasks.

Why Re-Rank?

No matter the architecture of your model, there is a substantial performance degradation when you include 10+ retrieved documents. In brief: When models must access relevant information in the middle of long contexts, they tend to ignore the provided documents. This issue highlights one of the key reasons why the initial retrieval step alone may not work effectively in Retrieval-Augmented Generation (RAG) models. The initial retrieval process aims to surface the top-k most relevant documents based on vector similarity matching between the query and document embeddings.

(Changing the location of relevant information (in this case, the position of the passage that answers an input question) within the language model’s input context results in a U-shaped performance curve—models are better at using relevant information that occurs at the very beginning (primacy bias) or end of its input context (recency bias). Performance degrades significantly when models must access and use information located in the middle of its input context.

Read: https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf

To overcome this we can re-rank the retrieved document and put the most relevant document in the beginning. There are several ways we can achieve this. we can use Bi-Encoder re-ranker, Cross-Encoder re-ranker, and LLM-based re-ranker.

There are several reasons why a re-ranker is beneficial in RAG models:

•  Improved relevance: The re-ranker can employ more advanced models, such as cross-encoders or sequence classification models, to better capture the contextual relevance between the query and the retrieved documents. This helps surface the truly most relevant documents, which leads to improved generation quality. 
• Handling complex queries: Simple vector similarity matching may struggle with complex, multi-faceted queries or queries involving logical reasoning. The re-ranker can better handle such cases by incorporating more advanced relevance signals. Mitigating retrieval errors: The initial retrieval step may sometimes fail to retrieve relevant documents due to encoding or matching errors. The re-ranker acts as a safeguard, potentially recovering relevant documents missed in the first step. 
• Efficiency: While advanced cross-encoder models can be computationally expensive for the initial retrieval step, using a lightweight bi-encoder or sparse embedding model followed by a re-ranker can strike a balance between efficiency and effectiveness.

Just an Intro To Bi-Encoder models and Cross-Encoder models:

Before we jump into the re-ranker let’s see what is Bi-Encoder and Cross-Encoder models are.

Bi-Encoder models:

Bi-encoder models are utilized to generate dense vector embeddings for data, which can subsequently facilitate search queries. These models enable the comparison of data vectors and query vectors through methods such as cosine similarity. Utilizing a Bi-Encoder model, each item in the database can be represented by a vector embedding.

(Representation of a Bi-Encoder model)

Cross-Encoder models:

Bi-Encoder models represent one approach to assessing similarity between pairs of data. Alternatively, Cross-Encoder models offer a distinct strategy. Unlike Bi-Encoders, Cross-Encoders don’t generate vector embeddings for individual data points. Instead, they employ a classification mechanism specifically designed for data pairs. Each input to a Cross-Encoder model comprises a pair of data elements, like two sentences, producing a similarity score ranging from 0 to 1. Consequently, Cross-Encoders cannot process single sentences; they necessitate paired items for analysis. For search purposes, employing a Cross-Encoder involves pairing each data item with the search query to compute similarity.

(Representation of a Cross-Encoder model)

Okay, that’s enough theory; let’s code.

Step 1: Install all the required packages we need.

!pip install –upgrade –quiet  sentence-transformers langchain_community chromadb langchain langchain_openai

For demonstration purposes, we’ve released the Langchain 0.1 version blog, which you can find here: https://blog.langchain.dev/langchain-v0-1-0/

Step 2: Import all required packages into your working Notebook

Step 3: Load the document and split it into chunks of 1000 with an overlap of 100.

Step 4: Define the embedding model, convert the document into embeddings, and add the documents into Cromadb.

Step 5: Test document retrieval without re-ranking.

pretty_print_docs(retriever.get_relevant_documents("What are the major
changes in v 0.1.0?"))

output:

You can see this by viewing the 0th document that was not related to the question we asked. Also, the data was not chunked properly but we left it for some other time.

Step 6: Define a naive RAG (Retriever Answer Generator) with an OpenAI chat model.

output:

Step 7: Let’s view the document with its re-ranking scores:

Now you can use the scores to re-arrange the document. Not let’s implement this in langchain.

Step 8:  langchain dose not support bge-reranker out of the box so you need to create your pipeline 

Step 9: Now let’s define our compression_pipeline

Now let's try the same query with compression_pipeline 

docs = compression_pipeline.get_relevant_documents(“What are the major changes in v 0.1.0?”)

pretty_print_docs(docs)

You can see there is a change in the document order now lets create RAG using our compression_pipeline as retrieval

Step 10: create RAG using our compression_pipeline as retrieval

Final output:

You see the different in the answer we get from the RAG.

Thank you for being with me until the very end! I know it’s a long blog, but in the end, we’re happy with the results we’ve achieved. ? Until next time, goodbye, and happy coding!