Dataset on RAG Pipeline Evaluation for Retrieval and Generative Response Accuracy Testing

Citation Author(s):: Pruthvi Raj Venkatesh (NIT Warangal)

Radha Krishna P (NIT Warangal)
Submitted by:: Pruthvi Raj Venkatesh
Last updated:: Sat, 08/31/2024 - 17:01
DOI:: 10.21227/hg2s-f350
Data Format:: *.json;

610 views

Categories:

Artificial Intelligence

Keywords:

ACCESS DATASET CITE

Abstract

This dataset has been meticulously curated to evaluate the efficiency of Retrieval-Augmented Generation (RAG) pipelines in both retrieval and generative accuracy, with a particular focus on scenarios involving overlapping contexts. The dataset comprises two primary components: Motor data and Employee data. The Motor dataset includes master data of various motor models along with their corresponding manuals, linked by the motor's model name. Similarly, the Employee dataset encompasses employee master data and associated policy documents, linked by department. By providing a diverse and contextually rich set of information, this dataset serves as a comprehensive resource for testing RAG pipelines' capabilities in handling complex queries and generating precise responses in domains where context overlap is prevalent.

Instructions:

Overview

This dataset is designed to facilitate testing and evaluation of Retrieval-Augmented Generation (RAG) pipelines. It is divided into two main sections: Employee and Motor.

Dataset Files

IeeeEmployeeData.json: Contains employee information.
IeeeEmployeePolicyData.json: Contains departmental policies.
IeeeMotorData.json: Contains motor information.
IeeeMotorManualData.json: Contains motor manuals.

How to Use

Load the JSON Files: Parse the JSON files to access the data.
Link Data: Use the DepartmentName in IeeeEmployeeData.json to link with Department in IeeeEmployeePolicyData.json, and ModelName in IeeeMotorData.json to link with Model in IeeeMotorManualData.json.
Test RAG Pipelines: Use the overlapping context in the manuals and policies to test and evaluate RAG models for accurate context retrieval and question answering.