What an LLM Judge Can and Cannot Tell You About 765 Skills
A score is a screening instrument. Treating it as a measurement is how directories end up confidently wrong.
Every score on this site comes from the same process: a six-dimension rubric scored by an LLM, weighted, with a security veto.
| Dimension | Weight | What it asks |
|---|---|---|
| Trigger quality | 20% | Will it fire on the right requests? |
| Structure | 15% | Can the file be parsed and loaded efficiently? |
| Workflow usability | 25% | Can an agent execute it without guessing? |
| Content | 10% | Is the domain advice correct and specific? |
| Engineering | 10% | Are scripts, assets, and dependencies sound? |
| Security | 20% | What can it break, leak, or execute? |
If security scores below a threshold, the total is capped at 3.9 regardless of everything else. The weights reflect what makes a skill useful in practice, not what makes it look impressive.
What it does well
It ranks consistently. Across re-evaluations, the same file lands in the same band, and the dimension profile is stable: a skill that scores 9 on workflow and 6 on security does so repeatedly.
It also catches gross defects reliably: empty sections, missing trigger phrases, copied boilerplate, dangerous commands without guards. The 174 placeholder skills we found earlier were all identified this way, and all 174 were real.
What it cannot do
It cannot resolve small differences. Our working estimate of run-to-run variance is about ±1.0 point. Every structural effect we measured in this corpus — confirmation rules, reference files, verification sections — was smaller than that, between +0.11 and +0.55. Those correlations are descriptions of the corpus, not evidence that adding a section raises a score.
It cannot verify domain correctness. It can tell whether a tax skill has a workflow. It cannot tell whether the tax advice is right. Content is only 10% of the weight for this reason.
It cannot judge intent. A test fixture and an attack look the same at the string level. We review those by hand.
It is not independent of the corpus. The rubric was refined while reading these files. That is a real source of bias, and the reason we publish the weights rather than a single opaque number.
How to use a score
- 8.5+: likely well-built; read the security profile before installing.
- 7–8.5: usable, with a specific weakness worth checking — usually workflow or trigger.
- Below 7: read the file first. It may be a thin import, a placeholder, or a skill with a genuine safety problem.
Never use the score to compare skills across categories. A 9.0 in data engineering and a 9.0 in copywriting are not the same achievement, and the rubric is not calibrated to equate them.
The honest summary
The evaluator is good at finding broken things and bad at fine-grained ranking. We use it that way: to route human attention, not to replace it. Any directory that presents an LLM score as an objective quality certificate is telling you more about its confidence than about its corpus.