nvidia/skillevaluator
A three-stage checkup for the 'skill folders' that plug new abilities into AI agents
SkillEvaluator is an open-source tool that checks agent skills, which are folders of instructions and files that extend AI agents. It runs a validation stage for safety and formatting, a deduplication stage to catch overlap with existing skills, and a live evaluation stage that tests whether a skill actually helps an agent perform. Built by NVIDIA under Apache 2.0, it's currently labeled as experimental, community-supported software.
- agent-evaluation
- agent-security
- agent-skills
- agentic-ai
- benchmark
- claude-code
- codex
- evaluate
- evaluation
- security-scanner
- skill-eval
- skill-evals
What it does
- A skill is a folder containing a SKILL.md file plus supporting materials that extend an AI agent, following the Agent Skills specification
- Tier 1 validation checks schema, personal data (PII), licensing, quality, and security issues, and several of these checks run without any API key
- Tier 2 deduplication uses embeddings, a technique that turns text into numeric vectors, to measure how much a new skill overlaps with existing ones
- Tier 3 live evaluation runs the skill with a real agent inside a Docker, local, or cloud sandbox to see how it changes agent behavior, and can auto-generate its own test dataset
- Each tier can be run independently, and the tool integrates with other open-source projects like NVIDIA's SkillSpector security scanner and the Harbor agent evaluation framework
Why it matters
As teams start attaching arbitrary skill folders to AI agents, there's a real risk of hidden security flaws, leaked personal data, or redundant instructions that hurt performance, so a standardized pre-deployment check matters. This gives skill authors and agent-pipeline teams a concrete way to vet quality and safety before shipping.
Terms in this repo
- Agent Skill · A folder of instructions and files, including a SKILL.md, that extends what an AI agent can do
- Quality gate · A checkpoint that blocks progress unless certain criteria are met
- Embedding · A technique that converts text meaning into numeric vectors so similarity can be measured
- Sandbox · An isolated environment for running code safely without affecting the real system
- PII scan · A check for personally identifiable information, like names or addresses, hidden in a document
Repository description (English)
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
Open on GitHubProject siteCoverage
Trending repos
- vorssaint/vorssaint-utilsOne free menu bar app replaces a dozen paid Mac utilities
- Alishahryar1/free-claude-codeA local proxy that lets coding AI agents run on 49 free or cheap model providers instead of one paid service
- freestylefly/awesome-gpt-image-2A library of 532 reverse-engineered prompts that turn GPT-Image2 into a predictable image-making tool
- block/buzzAn open-source workspace where humans and AI agents chat, code, and review in the same rooms
- NousResearch/hermes-agentNous Research's Hermes is an AI agent that gets smarter the more you use it
- virgiliojr94/book-to-skillA tool that turns technical book PDFs into on-demand reference skills for AI coding agents
- VoltAgent/awesome-agent-skillsA single hub collecting over 1000 'how-to' manuals that make AI coding assistants act like experts
- anthropics/claude-plugins-communityA shared shelf where anyone's Claude add-ons get listed for install