Generating Synthetic Data for RAG Systems Evaluation Using Amazon Bedrock

Evaluating your Augmented Retrieval Generation (RAG) system to ensure it meets your business requirements is crucial before implementing it in production environments. However, this requires acquiring a high-quality dataset of real-world question-and-answer pairs, which can be a daunting task, especially in the early stages of development. This is where synthetic data generation comes into play. With Amazon Bedrock, you can generate synthetic datasets that emulate real user queries, allowing you to efficiently and at scale evaluate the performance of your RAG system. With synthetic data, you can optimize the evaluation process and gain confidence in your system’s capabilities before releasing it into the real world.

This post explains how to use Anthropic Claude in Amazon Bedrock to generate synthetic data for evaluating your RAG system. Amazon Bedrock is a fully managed service that offers a selection of high-performance base models (FMs) from leading AI companies such as AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon, through a single API, along with a wide range of capabilities to build generative AI applications with security, privacy, and ethics.

Fundamentals of RAG Evaluation

Before diving into how to evaluate an RAG application, let’s recap the basic components of a simple RAG workflow. The workflow consists of the following steps:

1. In the ingestion stage, which occurs asynchronously, data is split into separate chunks. An embeddings model generates embeddings for each of these chunks, which are stored in a vector store.
2. When the user asks a question to the system, an embedding of the question is generated and the most relevant chunks are retrieved from the vector store.
3. The RAG model augments the user input by adding the relevant retrieved data to the context. This stage uses prompt engineering techniques to effectively communicate with the large language model (LLM). The augmented prompt allows the LLM to generate an accurate response to user queries.
4. An LLM is prompted to formulate a helpful response based on user questions and retrieved chunks.

Amazon Bedrock Knowledge Bases offers a simplified approach to implementing RAG on AWS, providing a fully managed solution to connect FMs to custom data sources.

Minimum Components of an RAG Evaluation Dataset

To properly evaluate an RAG system, you need to collect an evaluation dataset of typical user questions and answers. Additionally, you should ensure to evaluate not only the generation part of the process but also the retrieval part.

A typical RAG evaluation dataset consists of the following minimum components:

• A list of questions that users will ask the RAG system.
• A corresponding list of answers to evaluate the generation phase.
• The context or a list of contexts containing the answer for each question to evaluate the retrieval.

In an ideal world, you would take real user questions as the basis for evaluation. While this is the optimal approach, as it directly resembles end-user behavior, it is not always feasible, especially in the early stages of building an RAG system. As you progress, you should aim to incorporate real user questions into your evaluation set.

Generating Synthetic Data and Its Evaluation

To illustrate the process, we employ a use case that builds a shareholder letter chatbot for Amazon, allowing business analysts to gain insights into the company’s strategy and performance over the years.

First, we upload PDF files of Amazon’s shareholder letters as our knowledge base. By implementing RAG, the knowledge retriever could use a database supporting vector search to dynamically look up relevant documents that serve as knowledge sources.

We use Anthropic’s Claude model to extract questions and answers from our knowledge base. For orchestration and automation steps, we employ LangChain, an open-source Python library designed to build applications with large language models.

Best Practices and Conclusion

Generating synthetic data can significantly enhance the evaluation process of your RAG system. However, it is essential to follow best practices to maintain the quality and representativeness of the generated data, combining synthetic data with real data, implementing robust quality control mechanisms, and continuously refining the process.

Despite limitations, synthetic data generation is a valuable tool to accelerate the development and evaluation of RAG systems, contributing to the development of higher-performing AI systems.

We encourage developers, researchers, and enthusiasts to explore the mentioned techniques and experiment with generating synthetic data for their own RAG applications.