google-ai-edge/LiteRT-LM
Google open-sourced the engine that runs large language models directly on phones, laptops, and watches
LiteRT-LM is Google's open-source inference framework that lets large language models (LLMs) run directly on edge devices like phones, laptops, and Raspberry Pi, without needing a server. It supports models such as Gemma, Llama, Phi-4, and Qwen, with GPU/NPU acceleration, image and audio input, and agentic function-calling. It's already running in real Google products including Chrome, Chromebook Plus, and Pixel Watch.
What it does
- What: built an orchestration layer that runs LLMs directly on Android, iOS, web, desktop, and IoT devices instead of relying on cloud servers
- How: runs on top of Google's lightweight inference engine LiteRT, offers APIs in Python, Kotlin, Swift, JavaScript, and C++, and speeds things up with CPU/GPU/NPU acceleration and techniques like Multi-Token Prediction (MTP)
- Result: using MTP with Gemma4 models made inference up to 3x faster according to Google's blog, and models can be downloaded and run with a single command line even on a Raspberry Pi or laptop
- The latest v0.16.0 release shipped versioned C API prebuilt shared libraries for all supported platforms and added an experimental YNNPACK delegate for Linux arm64 builds
Why it matters
Running models on-device instead of sending data to a server means apps can work offline and keep user data local. For developers, it offers one consistent toolkit to add on-device AI to Android apps, web pages, and desktop software alike.
Terms in this repo
- LLM (Large Language Model) · An AI model trained on huge amounts of text to converse, summarize, or write code
- edge device · A device in the user's hands, like a phone or laptop, as opposed to a remote server
- NPU · A specialized chip built for AI computation
- Multi-Token Prediction (MTP) · A technique that predicts several tokens (words) at once to speed up generation
- function calling / Tool Use · A capability letting an AI model invoke external programs or tools to complete tasks
Repository description (English)
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install