AI/ML, AWS, Cloud Computing

4 Mins Read

The Power of Cross-Encoders in Re-Ranking for NLP and RAG Systems

Voiced by Amazon Polly

Overview

Re-ranking is critical in many natural language processing (NLP) tasks, particularly in retrieval-augmented generation systems (RAG). It refines the selection of retrieved documents or passages before passing them to the generative model. Cross-encoders are one of the most effective tools for this task, as they present an advanced method of assessing the relevance of query-document pairs. In this blog, we will discuss how cross-encoders work, why they are important, and how you can use pre-trained models for re-ranking.

Pioneers in Cloud Consulting & Migration Services

  • Reduced infrastructural costs
  • Accelerated application deployment
Get Started

Cross encoder

A cross-encoder model encodes a query and a document jointly to compute a relevance score. Unlike dual-encoders, which encode the query and document separately and then compare their embeddings, cross-encoders take the query and document as one sequence of tokens and process them together.

This allows the model to capture fine-grained interactions between the two, which makes it particularly useful for tasks such as re-ranking, question answering, and semantic similarity measurement.

Why Use Cross Encoder for Re-ranking?

Cross-encoder re-ranking has several advantages:

  • High Accuracy: Because the cross-encoders process both the query and the document, they better capture subtle interactions between both, leading to more accurate relevance judgment.
  • Contextualized Scoring: The self-attention mechanism in transformers allows the model to account for the interaction between tokens in the query and document, providing a more context-aware relevance score.
  • Better suited for Fine-Grained Tasks: Cross-encoders do well on tasks requiring finer grain relevance, such as in RAG systems wherein documents need to be very well matched to the query before any generative model is allowed to produce high quality outputs.

Pretrained Cross-Encoder Models

Many pretrained cross-encoder models are available that you can use for re-ranking tasks without having to train one from scratch. These models are usually fine-tuned on large datasets such as MS MARCO or Natural Questions and have been optimized for relevance ranking and re-ranking tasks.

Here are some popular pre-trained cross-encoder models:

  • Sentence Transformers’ Cross-Encoders:
    • The Sentence Transformers library has several pre-trained cross-encoder models fine-tuned for semantic search and re-ranking. Example models include:
      • cross-encoder/ms-marco-MiniLM-L-6-v2: Fine-tuned on the MS MARCO dataset for semantic search.
      • cross-encoder/ms-marco-TinyBERT-L-6-v2: A lighter, more efficient model optimized for faster inference.
      • cross-encoder/ms-marco-electra-base: An ELECTRA-based model fine-tuned on the MS MARCO dataset for better ranking performance.
  • Hugging Face Transformers:
    • Hugging Face maintains an impressive selection of transformer-based models already capable of adapting to cross-encoding needs: BERT, RoBERTa, DeBERTa, and more.
  • Other Specialized Models:
    • Many other pre-trained models are available from vendors, such as OpenAI (GPT-3.5, GPT-4), that can be utilized for ranking tasks. However, they do not qualify strictly as cross-encoders. Still, these models may be useful in re-ranking by scoring query-document relevance.

How do you use pre-trained cross encoders for re-ranking?

You can easily use pre-trained cross-encoder models to re-rank query-document pairs. Here’s a small demo of how to do this:

  • Install Sentence Transformers: First, install the sentence-transformers library:

encode

  • Load and Use a Pre-trained Model: Load a pretrained cross-encoder model and use it to predict the relevance of query-document pairs:

encode2

The final output will consist of scores based on which the pre-re-ranked responses can be ranked.

Example: Demonstrating Re-ranking Using a Cross-Encoder

Let’s consider a simple scenario with the following documents:

  1. Document 1: “Running is a popular form of physical exercise enjoyed by millions worldwide.”
  2. Document 2: “Running regularly improves cardiovascular health, boosts mental well-being, and helps with weight management.”
  3. Document 3: “Many athletes use running as part of their training to improve endurance and performance.

Query: ” What are the health benefits of running?”

Initially, simple retrieval might rank the documents like this:

Retrieved Documents (Before Re-ranking):

  1. Document 3: “Many athletes use running as part of their training to improve endurance and performance.”
  2. Document 1: ” Running is a popular form of physical exercise enjoyed by millions worldwide.”
  3. Document 2: ” Running regularly improves cardiovascular health, boosts mental well-being, and helps with weight management.”

Explanation of Document 3’s Initial Rank: Document 3 was initially chosen because it mentions “improve endurance,” which loosely relates to health benefits due to basic keyword matching.

After the cross-encoder is applied to re-rank the documents based on their context relevance to the query:

Re-ranked Documents (After Re-ranking):

  1. Document 2: ” Running regularly improves cardiovascular health, boosts mental well-being, and helps with weight management.” (Most relevant)
  2. Document 3: ” Many athletes use running as part of their training to improve endurance and performance.” (Still relevant but less so)
  3. Document 1: ” Running is a popular form of physical exercise enjoyed by millions worldwide.” (Least relevant)

Explanation of the Re-ranking:

  • Document 2 ranks first after re-ranking because it directly addresses the health benefits of running, such as cardiovascular health, mental well-being, and weight management, which closely align with the query.
  • Document 3 ranks second. While it mentions “endurance and performance,” these benefits are more related to athletic training than general health.
  • Document 1 ranks last because it focuses on the popularity of running and does not mention any specific health benefits.

Conclusion

Cross-encoders are powerful tools for re-ranking tasks, especially when you need precise relevance scoring between a query and a document. Their ability to jointly process both inputs allows them to capture detailed interactions, making them suitable for high-accuracy retrieval and re-ranking. However, depending on your specific requirements (e.g., large-scale retrieval, domain-specific tasks), other re-ranking methods like dual encoders or learning-to-rank may be more suitable. By understanding the strengths and weaknesses of each approach, you can select the best model for your re-ranking needs.

Drop a query if you have any questions regarding Cross-encoders and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

  • Reduced infrastructure costs
  • Timely data-driven decisions
Get Started

About CloudThat

CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.

FAQs

1. What is re-ranking in information retrieval?

ANS: – Re-ranking is taking a set of initially retrieved documents and reordering them based on their relevance to a specific query, typically using more advanced methods like cross-encoders.

2. How does a cross-encoder work for re-ranking?

ANS: – A cross-encoder jointly processes the query and each document to produce a relevance score, allowing it to evaluate the context and relationship between the two, which helps in more accurate ranking.

WRITTEN BY Venkata Kiran

Share

Comments

    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!