Sparkfetch/sparkfetch
An open-source API that turns messy webpages into clean text ready for AI to read
SparkFetch takes a URL and strips out ads, navigation bars, and other clutter, returning the page as clean Markdown, plain text, or structured JSON. Besides grabbing a single page, it can crawl an entire site to a set depth or just list every URL found on a domain. It's built in TypeScript on Node.js and Express, and can be self-hosted since it's open source.
What it does
- Instead of raw HTML, it removes boilerplate like ads and navigation and returns content as Markdown, plain text, or JSON
- The scrape endpoint fetches one page, crawl follows links to pull multiple pages in one job, and map just lists the URLs on a domain quickly
- Responses include not just the content but metadata like title, description, and HTTP status code
- Written in TypeScript, runs on Node.js 24 and Express 5, and can be cloned and run on your own server
Why it matters
Feeding AI chatbots or search tools with live web data requires cleaned-up text rather than raw pages, and this saves developers from building that extraction logic themselves. Being open source also means teams can self-host it instead of sending data to a third-party service.
Terms in this repo
- Markdown · a lightweight text format that marks headings and lists with simple symbols
- Crawl · automatically following links from a page to visit and collect many pages
- RAG pipeline · a process where an AI retrieves outside documents to use while generating an answer
- Metadata · extra descriptive info about content, like title, description, and fetch time
Repository description (English)
🔥 Turn any URL into clean, structured, LLM-ready content. The open-source web fetching & extraction API.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install