SSDEVIL108/LLM-Guided-Crawler-Secret-Hunter
A tool that crawls websites and uses AI to spot secrets developers accidentally left exposed
This is a Python tool that recursively crawls a target website, extracts links and JavaScript file paths exactly as they appear, then sends that code to multiple large language models (LLMs) in parallel to hunt for security issues. It looks for things like staging server addresses, debug flags, and hidden API endpoints, saving findings live to a markdown report. It's built for bug bounty hunters and security auditors who want to automate the reconnaissance stage of a security review.
What it does
- Uses the BeautifulSoup4 library to pull real <a> links and <script> sources directly from HTML, so the LLM never has to guess or invent links that don't exist
- Sends extracted code chunks to multiple LLMs at once (NVIDIA NIM or OpenAI-compatible APIs) using Python's ThreadPoolExecutor for parallel processing
- Builds an in-memory knowledge graph connecting pages, loaded scripts, outgoing links, and AI-flagged security findings
- Filters out UI-related files like CSS and images to focus only on staging subdomains, debug parameters such as DEBUG=true, and hidden API paths like /v1/*
- Writes results live to a markdown report as pages finish scanning, and can resume an interrupted scan by reading its own previous report instead of re-crawling everything
Why it matters
It could save security researchers and bug bounty hunters a lot of manual time normally spent browsing site source code and hunting for leaked internal details by hand. That said, since an LLM is interpreting code to flag potential issues, its output should be treated as a lead to verify rather than a confirmed vulnerability report.
Terms in this repo
- LLM · Large Language Model, an AI system trained to understand and generate text or code
- BeautifulSoup4 · A Python library used to parse HTML and extract specific tags or data
- ThreadPoolExecutor · A Python feature that runs multiple tasks concurrently in parallel
- knowledge graph · A data structure that represents pieces of information as connected nodes to show relationships
- NVIDIA NIM · An NVIDIA service/interface for running AI models
Repository description (English)
High-Precision Parallel Multi-Model LLM Web Reconnaissance Engine & Knowledge Graph Builder for Bug Bounty & Security Auditing.
Open on GitHubTrending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install