antonygiomarxdev/synapse
An open-source project splits huge MoE AI models across multiple consumer GPUs so a single expensive datacenter card isn't required
Synapse is a Rust-based infrastructure that spreads the 'experts' of a Mixture-of-Experts (MoE) model across several ordinary machines instead of one huge GPU. The project reports that splitting the work this way produced results identical to running the model on one machine, with modest speedups from adding workers. It's aimed at researchers and small teams who have consumer GPUs like an RTX 4090 but not datacenter hardware.
What it does
- MoE models only activate a small fraction of their parameters per token (the project cites roughly 2%), so Synapse distributes those active parts ('experts') across separate worker machines instead of loading the whole model onto one GPU.
- The system has a Gateway that receives jobs, a Scheduler that assigns tasks, and Worker nodes that each hold a subset of experts loaded from GGUF model files; it offers two modes — a 'Speculative Network' for fast chat-style use and a 'Network DAG' for batch processing.
- On a test model (granite3.1-moe:3b), running experts across 2 workers gave a 1.35x speedup and 4 workers gave 1.43x compared to running everything on one machine, while producing mathematically identical output (cosine similarity 1.000000).
- The project also demonstrated that its own MoE computation matches an established reference implementation (llama.cpp/llama-cpp-python) closely (0.999 correlation), and that jobs still complete if a worker crashes, thanks to automatic retries.
- The README states this is a 'V0' milestone: the core idea is proven on a small model, and larger speedups are expected with bigger models (e.g., Mixtral, DeepSeek) spread across multiple physical machines.
Why it matters
Running today's large MoE language models usually requires multiple expensive datacenter GPUs; this project's approach could let individuals or small teams run such models on hardware they already own, like a single gaming GPU. It matters for the AI infrastructure/tooling space because it reframes MoE's architecture (only using part of the model per token) as a distribution opportunity rather than a scaling burden.
Terms in this repo
- Mixture-of-Experts (MoE) · AI model design where only a subset of its 'expert' sub-networks are used for each input, instead of the whole model
- GGUF · A file format commonly used to store and load AI model weights efficiently
- cosine similarity · A number from -1 to 1 measuring how similar two sets of results are; 1.0 means identical
- FFN (feed-forward network) · A basic building block inside AI models that processes data; here, the 'expert' parts being distributed
- axum · A Rust software library used to build web servers, used here for Synapse's Gateway
Repository description (English)
Distributed inference infrastructure for Mixture-of-Experts models. Run large MoE models on consumer GPUs.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install