# LLM-as-a-judge

> LLM-as-a-judge is using a language model with a written prompt to grade another system's output. Its verdicts are only as trustworthy as its measured agreement with people.

Published 2026-11-12 · Reviewed 2026-11-12. Canonical: https://attify.app/glossary/llm-as-a-judge

LLM-as-a-judge is using a language model, given a written prompt and a rubric, to grade another system's output: whether a support answer resolved the request, whether a meeting was a sales meeting, whether a booking matches what was agreed. Research has found strong LLM judges can agree with human preferences at levels close to human-to-human agreement on some tasks, and also documented biases such as favoring longer answers or their own outputs ([Zheng et al., 2023](https://arxiv.org/abs/2306.05685)). Whether a given judge is good enough is an empirical question for each task.

## How Attify uses a judge

- **Only where reading is needed.** Most checks are rules over records. The judge decides what only reading can, on random samples per company (200 claims for the main check): for support, whether the issue was resolved and whether the final reply was a handoff; for sales meetings, whether it was a sales meeting; for bookings, whether the appointment matches what was agreed.
- **A published prompt.** Each standard prints its judge prompt word for word; a test fails if the running prompt differs. Changing a word is a new minor version.
- **Measured against people.** No judged score publishes until the judge agrees with people at least 85% on at least 50 labelled records for that standard version ([human audit](https://attify.app/glossary/human-audit), [judge agreement](https://attify.app/glossary/judge-agreement)).
- **No leakage.** No human-labelled record is ever an example in a prompt.
- **Reproducible.** Judge calls are cached by prompt, model and transcript.
- **Unclear favors the claim.** For sales meetings and bookings, "unclear" passes and its share is reported. For support, "unclear" means no request, and those conversations leave both rates.

## The vendor's judge is not evidence

Many platforms grade their own agent's conversations with an LLM (a success evaluation, a resolution classifier). For support, Attify reads such a verdict as the vendor's claim and tests it; for sales meetings and bookings it is not a claim. It never counts as a check.

## Questions

### Doesn't an LLM judging an LLM just repeat its mistakes?

It can, which is why the judge's agreement with people is measured on records never used to write or change its prompt, published with an interval, and required before anything it decided is published.

### Can I read Attify's judge prompts?

Yes. Each standard publishes its judge prompt word for word in an appendix, and a test checks that the prompt the judge runs equals the published text.

### Is a voice platform's success evaluation an LLM judge?

Yes: a voice platform's success flag is an LLM's grading of the call. For support, Attify reads it as the vendor's own claim and tests that claim; for sales meetings and bookings it is not a claim. It is never evidence.

## Sources

- arXiv: [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685), read 2026-11-08
- Attify: [Attify Audited Resolution Standard v0.11: Appendix B (judge prompt)](https://attify.app/standards/support-resolution), read 2026-11-09
