Benchmark Dataset for Generative AI on Edge Devices
- Citation Author(s):
-
Zeinab Nezami (University of Leeds)Maryam Hafeez (University of Leeds)Karim Djemame (University of Leeds)Syed Ali Raza Zaidi (University of Leeds)Jie Xu (University of Leeds)
- Submitted by:
- Zeinab Nezami
- Last updated:
- DOI:
- 10.21227/7d08-8655
- Research Article Link:
451 views
- Categories:
- Keywords:
Abstract
The benchmarking dataset, GenAI on the Edge, contains performance metrics from evaluating Large Language Models (LLMs) on edge devices, utilizing a distributed testbed of Raspberry Pi devices orchestrated by Kubernetes (K3s). It includes performance data collected from multiple runs of prompt-based evaluations with various LLMs, leveraging Prometheus and the Llama.cpp framework. The dataset captures key metrics such as resource utilization, token generation rates/throughput, and detailed inference timing for stages such as Sample, Prefill, and Decode. These metrics enable benchmarking of LLM performance in constrained, distributed environments, offering valuable insights into the feasibility and efficiency of deploying generative AI on edge computing devices.
Instructions:
To use the dataset, follow these steps:
1. Download the dataset: Obtain the dataset files, which are named based on the LLM model and evaluation run number.
2. Load the data: Open the CSV files using a data analysis tool such as Python’s pandas library, Excel, or any other software that supports CSV file format. Example in Python:
import pandas as pd
# Specify the path to the CSV file
file_path = 'gemma/inference_metrics_run_1.csv'
# Read the CSV file into a DataFrame
df = pd.read_csv(file_path)
# Display the first few rows of the DataFrame
print(df.head())
3. The dataset includes a suite of analysis scripts designed to facilitate data exploration, analysis, and visualization.
For further information, please refer to the README file included in the dataset. The README contains detailed instructions on how to use the dataset, along with explanations of the structure and content of the dataset.
