The Power of Cross-Encoders in Re-Ranking for NLP and RAG Systems

Overview

Re-ranking is critical in many natural language processing (NLP) tasks, particularly in retrieval-augmented generation systems (RAG). It refines the selection of retrieved documents or passages before passing them to the generative model. Cross-encoders are one of the most effective tools for this task, as they present an advanced method of assessing the relevance of query-document pairs. In this blog, we will discuss how cross-encoders work, why they are important, and how you can use pre-trained models for re-ranking.

Cross encoder

A cross-encoder model encodes a query and a document jointly to compute a relevance score. Unlike dual-encoders, which encode the query and document separately and then compare their embeddings, cross-encoders take the query and document as one sequence of tokens and process them together.

Why Use Cross Encoder for Re-ranking?

Cross-encoder re-ranking has several advantages:

• High Accuracy: Because the cross-encoders process both the query and the document, they better capture subtle interactions between both, leading to more accurate relevance judgment.
• Contextualized Scoring: The self-attention mechanism in transformers allows the model to account for the interaction between tokens in the query and document, providing a more context-aware relevance score.
• Better suited for Fine-Grained Tasks: Cross-encoders do well on tasks requiring finer grain relevance, such as in RAG systems wherein documents need to be very well matched to the query before any generative model is allowed to produce high quality outputs.

Pretrained Cross-Encoder Models

Many pretrained cross-encoder models are available that you can use for re-ranking tasks without having to train one from scratch. These models are usually fine-tuned on large datasets such as MS MARCO or Natural Questions and have been optimized for relevance ranking and re-ranking tasks.

Here are some popular pre-trained cross-encoder models:

• Sentence Transformers’ Cross-Encoders:
  • The Sentence Transformers library has several pre-trained cross-encoder models fine-tuned for semantic search and re-ranking. Example models include:
    • cross-encoder/ms-marco-MiniLM-L-6-v2: Fine-tuned on the MS MARCO dataset for semantic search.
    • cross-encoder/ms-marco-TinyBERT-L-6-v2: A lighter, more efficient model optimized for faster inference.
    • cross-encoder/ms-marco-electra-base: An ELECTRA-based model fine-tuned on the MS MARCO dataset for better ranking performance.
• Hugging Face Transformers:
  • Hugging Face maintains an impressive selection of transformer-based models already capable of adapting to cross-encoding needs: BERT, RoBERTa, DeBERTa, and more.
• Other Specialized Models:
  • Many other pre-trained models are available from vendors, such as OpenAI (GPT-3.5, GPT-4), that can be utilized for ranking tasks. However, they do not qualify strictly as cross-encoders. Still, these models may be useful in re-ranking by scoring query-document relevance.

How do you use pre-trained cross encoders for re-ranking?

You can easily use pre-trained cross-encoder models to re-rank query-document pairs. Here’s a small demo of how to do this:

• Install Sentence Transformers: First, install the sentence-transformers library:

• Load and Use a Pre-trained Model: Load a pretrained cross-encoder model and use it to predict the relevance of query-document pairs:

The final output will consist of scores based on which the pre-re-ranked responses can be ranked.

Example: Demonstrating Re-ranking Using a Cross-Encoder

Let’s consider a simple scenario with the following documents:

1. Document 1: “Running is a popular form of physical exercise enjoyed by millions worldwide.”
2. Document 2: “Running regularly improves cardiovascular health, boosts mental well-being, and helps with weight management.”
3. Document 3: “Many athletes use running as part of their training to improve endurance and performance.“

Query: ” What are the health benefits of running?”

Initially, simple retrieval might rank the documents like this:

Retrieved Documents (Before Re-ranking):

1. Document 3: “Many athletes use running as part of their training to improve endurance and performance.”
2. Document 1: ” Running is a popular form of physical exercise enjoyed by millions worldwide.”
3. Document 2: ” Running regularly improves cardiovascular health, boosts mental well-being, and helps with weight management.”

Explanation of Document 3’s Initial Rank: Document 3 was initially chosen because it mentions “improve endurance,” which loosely relates to health benefits due to basic keyword matching.

After the cross-encoder is applied to re-rank the documents based on their context relevance to the query:

Re-ranked Documents (After Re-ranking):

1. Document 2: ” Running regularly improves cardiovascular health, boosts mental well-being, and helps with weight management.” (Most relevant)
2. Document 3: ” Many athletes use running as part of their training to improve endurance and performance.” (Still relevant but less so)
3. Document 1: ” Running is a popular form of physical exercise enjoyed by millions worldwide.” (Least relevant)

Explanation of the Re-ranking:

• Document 2 ranks first after re-ranking because it directly addresses the health benefits of running, such as cardiovascular health, mental well-being, and weight management, which closely align with the query.
• Document 3 ranks second. While it mentions “endurance and performance,” these benefits are more related to athletic training than general health.
• Document 1 ranks last because it focuses on the popularity of running and does not mention any specific health benefits.

Conclusion

Cross-encoders are powerful tools for re-ranking tasks, especially when you need precise relevance scoring between a query and a document. Their ability to jointly process both inputs allows them to capture detailed interactions, making them suitable for high-accuracy retrieval and re-ranking. However, depending on your specific requirements (e.g., large-scale retrieval, domain-specific tasks), other re-ranking methods like dual encoders or learning-to-rank may be more suitable. By understanding the strengths and weaknesses of each approach, you can select the best model for your re-ranking needs.

Drop a query if you have any questions regarding Cross-encoders and we will get back to you quickly.

About CloudThat