google-ai-edge/eval
Google releases a CLI tool that benchmarks on-device AI models the same way as regular ones
ai-edge-eval is a command-line evaluation tool from Google that lets you test both LiteRT LM models, Google's lightweight on-device model format, and standard models like HuggingFace checkpoints using the same interface. It runs text and vision+text benchmarks such as ifeval, mmlu, piqa and mmmu through existing evaluation frameworks lm-eval and lighteval. Users can also register their own datasets and scoring functions to build custom evaluation tasks.
What it does
- Provides one CLI that evaluates both LiteRT LM models and native models like HuggingFace under a shared workflow
- Supports two backend frameworks, lm-eval and lighteval, covering both text-only and multimodal (vision + text) tasks
- Offers --limit and --sample-range flags to run quick smoke tests on a small slice of data instead of the full benchmark
- Lets users define custom tasks by supplying a JSON Lines dataset and a Python scoring function via a registration file
- Runs on Linux, macOS, and WSL2, and is released under the Apache 2.0 license
Why it matters
It gives developers building small on-device language models a standardized way to measure performance across different model formats. It also flags that each benchmark dataset carries its own license, so users must check usage terms themselves.
Terms in this repo
- LiteRT LM · Google's lightweight model format built for running on-device
- lm-eval · A benchmark evaluation harness for language models made by EleutherAI
- lighteval · An alternative evaluation framework from the HuggingFace ecosystem
- multimodal · Handling both images and text together as input
- JSON Lines (.jsonl) · A file format storing one JSON object per line
Repository description (English)
google-ai-edge/eval
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install