lamhotsiagian/llm-system-design
A hands-on codebase that lets you inspect every piece of a production-style LLM system in plain Python
This repository has two parts: project, a pure-Python implementation of everything a real LLM serving stack needs — routing, retrieval-augmented generation (RAG), safety guardrails, and agent control — built with no heavy frameworks, and lab, a browsable Streamlit web app powered by LangChain. project ships 150+ offline tests across roughly 30 modules, while lab runs against local Ollama models (llama3.1, nomic-embed-text) or falls back to a deterministic offline mode. That means anyone can run and trace the internals of an LLM system without needing a live server or API keys.
What it does
- The project folder implements around 30 modules covering capacity/cost math, request pipelines, routing, caching, safety filters, training-infrastructure formulas, the inference engine, RAG, tool execution, agents, multi-agent orchestration, monitoring, autoscaling, and reliability engineering, all in pure Python standard library.
- The lab folder wraps LangChain and Streamlit into a web UI where you can click through routing, RAG, agents, guardrails, semantic caching, recommendations, and evaluation, and watch the execution trace live.
- lab defaults to an 'auto' backend: it uses real llama3.1 and nomic-embed-text models via Ollama if a local Ollama server is reachable, otherwise it automatically falls back to a deterministic offline mode with no server needed.
- project includes 150+ pytest-based offline tests, and lab tests can be run fully offline via the LAB_BACKEND=offline environment variable, making the whole system reproducibly testable.
- Optionally you can launch a FastAPI HTTP server or point the platform at any OpenAI-compatible endpoint such as vLLM, TGI, Ollama, or a vendor API.
Why it matters
It turns abstract LLM system-design concepts — routing, RAG, safety, agents, caching, observability — into runnable, inspectable code, making it useful as study material for system design learning, interview prep, or internal prototyping.
Terms in this repo
- RAG (retrieval-augmented generation) · answering a question by first retrieving relevant documents and grounding the response in them
- ReAct agent loop · a pattern where a model alternates between reasoning, calling tools, and observing results to solve a task
- KV cache · stored attention results from earlier tokens that speed up generating the next token
- LCEL · LangChain's syntax for chaining processing steps together like a pipeline
- guardrails · safety checks that filter dangerous inputs going in or unsafe outputs coming out of a model
Repository description (English)
A production-grade LLM System Design platform & interactive lab. Features a pure-Python LLM serving, RAG, routing, safety, and agent control plane (150+ offline tests) plus a LangChain + Streamlit UI with local Ollama models.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install