Benchmark Dataset for Generative AI on Edge Devices

Citation Author(s):
Zeinab
Nezami
University of Leeds
Maryam
Hafeez
University of Leeds
Karim
Djemame
University of Leeds
Syed Ali Raza
Zaidi
University of Leeds
Jie
Xu
University of Leeds
Submitted by:
Zeinab Nezami
Last updated:
Thu, 12/12/2024 - 09:27
DOI:
10.21227/7d08-8655
Research Article Link:
License:
0
0 ratings - Please login to submit your rating.

Abstract 

The benchmarking dataset, GenAI on the Edge, contains performance metrics from evaluating Large Language Models (LLMs) on edge devices, utilizing a distributed testbed of Raspberry Pi devices orchestrated by Kubernetes (K3s). It includes performance data collected from multiple runs of prompt-based evaluations with various LLMs, leveraging Prometheus and the Llama.cpp framework. The dataset captures key metrics such as resource utilization, token generation rates/throughput, and detailed inference timing for stages such as Sample, Prefill, and Decode. These metrics enable benchmarking of LLM performance in constrained, distributed environments, offering valuable insights into the feasibility and efficiency of deploying generative AI on edge computing devices.

Instructions: 

To use the dataset, follow these steps:

1. Download the dataset: Obtain the dataset files, which are named based on the LLM model and evaluation run number.

2. Load the data: Open the CSV files using a data analysis tool such as Python’s pandas library, Excel, or any other software that supports CSV file format. Example in Python:

import pandas as pd

# Specify the path to the CSV file

file_path = 'gemma/inference_metrics_run_1.csv'

# Read the CSV file into a DataFrame

df = pd.read_csv(file_path)

# Display the first few rows of the DataFrame

print(df.head())

3. The dataset includes a suite of analysis scripts designed to facilitate data exploration, analysis, and visualization.

For further information, please refer to the README file included in the dataset. The README contains detailed instructions on how to use the dataset, along with explanations of the structure and content of the dataset.

Funding Agency: 
The Communications Hub for Empowering Distributed Cloud Computing Applications and Research (CHEDDAR), supported by the EPSRC through UKRI’s Technology Missions Fund
Grant Number: 
EP/Y037421/1

Documentation

AttachmentSize
File README.pdf143.24 KB