google-ai-edge/ai-edge-quantizer
Google's toolkit shrinks AI models for phones and edge chips without retraining them
AI Edge Quantizer (AEQ) is Google's tool for compressing LiteRT (formerly TensorFlow Lite) models after they're already trained, so they run faster and smaller on phones, GPUs, and NPUs. It lets developers pick different precision levels for different layers, or skip quantizing sensitive layers entirely, using simple recipes. It also includes built-in checks to measure how much accuracy is lost after compression.
What it does
- Compresses already-trained vision models, LLMs, and GenAI pipelines into smaller integer formats (like INT8 or INT4) for LiteRT and LiteRT-LM deployment on mobile CPUs, GPUs, and NPUs
- Lets developers target specific layers by name pattern (selective quantization) and mix different precision levels across layers (e.g., INT4 in some layers, INT8 in others)
- Includes advanced techniques like blockwise quantization, Hadamard transforms to reduce accuracy loss at very low bit-widths, and GPTQ/OCTAV optimization algorithms
- Offers three quantization modes: dynamic (weights only, calibration-free), weight-only (best quality but no speed gain), and static full-integer (fastest, but needs sample data for calibration)
- Ships with a command-line tool (aeq) and Python API, plus built-in accuracy validation (MSE, SNR, cosine similarity, KL divergence) and integration with Model Explorer for visualizing quantization effects
Why it matters
Post-training quantization is one of the main ways to make large models practical on resource-limited devices like phones, without the cost of retraining. Having selective and mixed-precision control plus built-in validation makes it easier to balance speed and accuracy for real deployment scenarios, including modern LLMs.
Terms in this repo
- Post-training quantization (PTQ) · Compressing a model into lower-precision numbers after training is already done, without retraining
- LiteRT · Google's runtime for running AI models on mobile and edge devices, formerly called TensorFlow Lite
- NPU · Neural Processing Unit, a chip specialized for running AI computations efficiently
- Static quantization / calibration · Quantizing both weights and activations using statistics gathered by running sample data through the model beforehand
- GPTQ / OCTAV · Optimization algorithms designed to reduce the accuracy loss caused by aggressive quantization
Repository description (English)
AI Edge Quantizer: flexible post training quantization for LiteRT models.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install