Every Structural Feature We Measured Fits Inside the Evaluator's Noise
A table of ten correlations that all look meaningful until you put the error bars next to them.
We scanned 765 SKILL.md files for ten structural features and compared the average score of skills with and without each one:
| Feature | Present | Share | Avg with | Avg without | Gap |
|---|---|---|---|---|---|
| Confirmation rules | 14 | 1.8% | 8.99 | 8.44 | +0.55 |
References assets/ | 32 | 4.2% | 8.94 | 8.43 | +0.51 |
References references/ | 409 | 53.5% | 8.67 | 8.19 | +0.48 |
| Mentions verification | 328 | 42.9% | 8.66 | 8.30 | +0.36 |
| States what it won't do | 196 | 25.6% | 8.70 | 8.36 | +0.34 |
| Failure handling | 60 | 7.8% | 8.76 | 8.42 | +0.33 |
| Output section | 174 | 22.7% | 8.62 | 8.40 | +0.22 |
References scripts/ | 173 | 22.6% | 8.61 | 8.41 | +0.20 |
| Workflow section | 377 | 49.3% | 8.54 | 8.37 | +0.17 |
| "When to use" section | 190 | 24.8% | 8.54 | 8.42 | +0.11 |
Every gap is positive. That is the seductive part — it looks like a clean ranking of practices.
The error bar that ruins it
We measured our evaluator's variance directly: re-running the same evaluation on 11 skills two hours later moved 7 of 11 scores, with a maximum swing of a full point.
Against that, a 0.2-point difference between skills with and without a workflow section is not a finding. It is what the noise looks like.
Only the top two rows — confirmation rules (+0.55) and assets/ (+0.51) — approach the noise floor, and both are computed on tiny samples: 14 and 32 skills respectively. A handful of well-made skills can move either average by half a point on their own.
Why we are publishing it anyway
Because the alternative is a reader discovering the correlation table elsewhere and treating it as a ranking. The honest framing is:
- Direction: skills built with more deliberate packaging score higher. Ten out of ten features point the same way, which is unlikely to be pure chance.
- Magnitude: we cannot resolve the size of the effect. The data is consistent with each feature being worth anywhere from 0 to 0.6 points.
- Causation: none established. Confident authors may produce both the feature and the score.
What survives
One thing does. The features are cheap relative to the score band. Adding a confirmation rule or a verification step costs a paragraph. Even if the effect is 0.2 points, that is a paragraph well spent — and unlike most quality advice, this one has a mechanical argument behind it: an agent that confirms before writing cannot silently destroy something.
The rubric is at methodology. The variance measurement is at we ran the same evaluation twice.