jundot/omlx
A menu-bar app that runs and manages local LLMs on your Mac
oMLX is an open-source server for running large language models directly on Apple Silicon Macs, controlled from a native menu-bar app. It combines continuous batching, which handles many requests at once, with a two-tier cache that keeps reusable computation results in RAM and overflow on SSD. It speaks the OpenAI and Anthropic APIs, so existing tools can connect without changes.
What it does
- Serves not just text models but also vision-language models, OCR models, embeddings, and rerankers from a single server.
- Keeps previously computed context in a fast RAM tier and a larger SSD tier, so even mid-conversation context changes stay reusable, and it survives server restarts by restoring from disk instead of recomputing.
- Includes an experimental feature to split one large model across multiple Macs with different memory sizes over a network.
- With dedicated custom kernels enabled, models like GLM-5.2 and MiniMax M3 process about 845 tokens per second on an M3 Ultra, roughly 30x faster than the ~29 tokens per second fallback without them.
- An admin dashboard lets you download models, chat, run benchmarks, and tune per-model settings with a few clicks, no manual config editing needed.
Why it matters
It gives developers and researchers running LLMs locally on a Mac an easy way to manage memory and performance without touching the terminal. This matters especially for coding assistants, where reusing cached context instead of recomputing it saves real time and cost.
Terms in this repo
- Continuous Batching · Processing many concurrent requests together instead of one at a time, for efficiency
- KV Cache · A temporary store of a model's previously computed results so it doesn't repeat the same calculation
- Apple Silicon · Apple's own chips for Mac, such as M1, M2, M3, M4
- MCP (Model Context Protocol) · A standard way for AI models to connect to external tools or data
- LRU eviction · Automatically removing the least recently used item from memory first
Repository description (English)
managed from the macOS menu bar
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install