What Changed After We Re-Evaluated 174 Skills
The placeholder scores were not wrong because the evaluator was biased. They were wrong because three fields were never written.
Earlier this month we found that 174 of 765 skills — 22.7% of the directory — displayed zeroes for structure, content, and engineering while still showing an overall score around 8. The cause was a three-dimension evaluator that only wrote quality, usability, and security. The other fields defaulted to zero, and the display papered over it with a total that looked normal.
We re-ran all 174 with the six-dimension rubric.
The result
| Check | Before | After |
|---|---|---|
| Skills with placeholder dimensions | 174 | 0 |
| Assessments with all six dimensions | — | 765 |
| Weighted-score consistency error | — | ≤ 0.05 |
The new distribution across all 765 skills:
| Band | Skills |
|---|---|
| 9.5+ | 19 |
| 9.0–9.5 | 182 |
| 8.0–9.0 | 402 |
| 7.0–8.0 | 127 |
| Below 7.0 | 35 |
Average: 8.44. Median: 8.60.
Update, same day: 11 skills whose page bodies had been truncated by an import gap were restored and re-scored against their original SKILL.md files. Their scores moved modestly — one fell from 8.5 to 5.6 — and the table above reflects the final values.
The counter-intuitive part
The number of skills below 7 went from 18 to 35.
That is not because re-evaluation made skills worse. The earlier count was taken across 591 fully-evaluated skills. The 174 newly evaluated ones had never been measured on three of the six dimensions; when they were, some of them turned out to be weak. Better measurement revealed the low scores; it did not create them.
This is the most useful lesson from the exercise: a missing field is not a neutral absence. It participates in every average, every ranking, and every recommendation. The placeholder skills were inflating the apparent quality of their categories simply by not being measured.
What we changed in the evaluator
The scoring pipeline now:
- Scores all six dimensions in one pass and returns JSON.
- Computes the weighted total — trigger 20%, structure 15%, workflow 25%, content 10%, engineering 10%, security 20% — and applies the security veto.
- Writes every dimension back, plus the English rationale for each.
- Verifies that the stored weighted total matches the recomputed one within 0.05.
- Accepts an explicit ID list, so a re-evaluation is not silently limited to the most recent 200 rows.
- Has request timeouts on every network call, so one unresponsive repo cannot hang the batch.
What we still do not know
Whether the new scores are right, only that they are complete and internally consistent. The evaluator's run-to-run variance is about ±1.0 point, and the content dimension still cannot verify domain expertise.
The honest version of the before/after is not "quality improved". It is "the measurement caught up with the directory". That is a less satisfying headline and a more accurate one.