scenepilotLLM/sparkfetch
An open-source API that turns any webpage into clean markdown or JSON that LLMs can actually use
SparkFetch is a self-hostable API that takes any URL and strips out ads, navigation, and other clutter, returning clean Markdown, structured JSON, or plain text. It offers single-page scraping, recursive crawling across a whole site with depth control, and a map feature that discovers all URLs on a domain. It's built as a TypeScript project on Node.js 24 and Express 5.
What it does
- The scrape endpoint fetches a single URL and returns Markdown content plus metadata (title, description, links) as JSON
- The crawl endpoint recursively walks a website with configurable depth and page limits, running as an async job you can poll for status
- The map endpoint instantly discovers all accessible URLs on a domain, and output cleaning automatically strips tags like nav and footer
- The API spec is defined with OpenAPI 3.1 and validated with Zod, with a Drizzle ORM layer for the database
- It's released under the MIT license and can be installed and self-hosted with pnpm on Node.js 18 or later
Why it matters
If you're building a RAG pipeline or research automation tool that feeds web content into an LLM, this gives you clean text without writing your own HTML parser. Because it's self-hostable, you keep control over your data pipeline instead of depending on a third-party scraping service.
Terms in this repo
- Markdown · a lightweight text format for headings and lists that's easy for LLMs to parse
- RAG pipeline · a system that retrieves external documents to ground an LLM's answers
- crawling · automatically following links to visit and collect multiple pages of a site
- OpenAPI 3.1 · a standard specification format for describing API requests and responses
- Zod · a TypeScript library for validating that input data matches an expected shape
Repository description (English)
🔥 Turn any URL into clean, structured, LLM-ready content. The open-source web fetching & extraction API.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install