What We Learned From Scoring 765 Skills
This batch started with a simple question: what do 765 SKILL.md files actually contain? The answer is more consistent than expected, and less flattering.
We analysed every skill in the directory — body text, headings, tool references, trigger phrases, length, and six evaluation dimensions. Twenty posts later, three findings survive scrutiny.
1. Routing happens in one field
The description decides whether a skill is loaded. Everything below it is invisible to that decision.
The corpus has not internalised this:
- 25.6% of skills state a boundary somewhere; 0.5% state one in the description.
- Many skills are well-written documents that happen to be routed by a single sentence written last.
The fix is a one-sentence move, and it is the highest-leverage edit available to most authors. See A Quarter of Skills State What They Won't Do.
2. Safety is the thinnest layer
| Behaviour | Coverage |
|---|---|
| Verification | 42.9% |
| Failure handling | 7.8% |
| Confirmation before irreversible actions | 1.8% |
The skills that do handle safety well are instructive: two skills warn explicitly about prompt injection (and are the only two containing the canonical injection phrase), and all three mentions of rm -rf are scoped or guarded. The good patterns exist; they are just rare.
3. Most structural effects are below the noise floor
We measured the score gap associated with confirmation rules (+0.55), reference files (+0.48), verification sections (+0.36), and several others. Every gap is smaller than the evaluator's run-to-run variance of about ±1.0.
That does not mean the features are useless. It means we cannot demonstrate their effect with this instrument. The mechanism — a description-level exclusion changes routing; a confirmation gate prevents an irreversible action — is stronger evidence than a correlation of 0.2.
What the corpus is
Mostly documents, not commands. Read and Write appear in roughly a quarter of skills; Bash appears in 1%. The dominant skill reads an input and produces an artifact: a contract, a deck, a report, a video.
That explains both the ecosystem's strengths and its blind spots. It is good at producing things and underdeveloped at checking them.
What we are doing next
- Fixing data gaps we found during the analysis, including entries whose body text never made it from the import archive into the page.
- Human review of the small set of skills that reference destructive commands or APIs.
- Publishing the rubric and the raw distribution, so the scores can be read as what they are: a screening tool, not a certificate.
The honest conclusion
A directory of 765 skills from 65 authors is not 765 independent data points. It is a dozen large catalogues plus a long tail, and most ecosystem-level statistics are really statements about who published the most files. We tried to say so every time it mattered.
The skills themselves are better than the averages suggest. The median skill has a workflow, an output, and a clear purpose. What it often lacks is the last ten percent — the verification step, the failure branch, the confirmation gate — that separates a helpful skill from a dependable one.