cactus-compute/needle
A 14MB AI model small enough to run tool-calling on a phone or watch, without internet
Needle 2 is a 45-million-parameter model built by Cactus Compute for calling tools and pulling structured data out of text, packed into a single 14MB file that runs a session in about 28MB of RAM. It is compressed to 2-bit precision and paired with its own lightweight inference engine, so it works fully offline on small devices. The Python package lets developers describe functions in code and have the model decide when and how to call them, plus fine-tune it with their own data.
What it does
- What: an open small language model specialized in tool calling, on-device use, and structured extraction (turning text into JSON), aimed at phones, wearables, smart home devices, and robots.
- How: built on the team's own 'Simple Attention Network' design (described in a linked paper) and compressed with a custom quantization method called Cactus Quants down to 2-bit weights, baked into a single self-contained engine with no internet access needed.
- Result: on the paper's benchmarks, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, while being 5x to 70x smaller and using 2-bit precision versus their 16-bit (f16) format.
- Extra features: responses come with a confidence score so apps can decide when to ask a human instead, a retrieval system narrows a large list of tools down to the top 5 relevant ones per turn, and a 256-token memory window keeps RAM usage bounded no matter how long a conversation runs.
- Workflow: install via pip, decorate a Python function as a tool, and call agent.run(); developers can also fine-tune the model with LoRA (a lightweight way to adapt a frozen base model) on their own tool data and export a new compact file.
Why it matters
Running AI on-device instead of in the cloud means it works offline, keeps data private, and avoids server costs — important for wearables, smart home gadgets, and robots that can't always reach the internet. This shows a working recipe for squeezing a capable, structured-output model into a footprint small enough for constrained hardware.
Terms in this repo
- LoRA · a lightweight fine-tuning method that trains a small add-on to a frozen base model instead of retraining everything
- quantization / 2-bit (CQ2) · compressing a model's numbers to use far fewer bits per value, shrinking file size at some cost to precision
- grammar-constrained decoding · forcing the model's output token-by-token to match a required JSON structure so it can't produce invalid data
- KV sink / sliding window · a technique to keep only a limited, fixed amount of conversation memory in RAM instead of it growing without bound
- GQA (grouped-query attention) · an efficient variant of the attention mechanism that reduces memory use in transformer models
Repository description (English)
14MB foundation model for tiny devices; phones, wearables, smart home, and robots.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install