LLM evaluation and observability platforms vs. outcome verification
Evaluation and observability platforms are instruments for the team building an agent: they trace its runs and score them with evaluators the team chooses (code checks, LLM judges, human review), offline on datasets and online on production traces. Outcome verification is a measurement for the agent's buyers and their buyers' trust: it tests the agent's claimed outcomes under a published rule, against what happened next in the customer's own systems.
What the platforms describe#
- Braintrust: datasets of examples, scorers and classifiers built with code or LLM-as-a-judge, online scoring of production traces, which relies on LLM judges because live requests have no ground truth (Braintrust).
- LangSmith: offline evaluation on datasets with reference outputs before deployment, online evaluation on live traces without them; human review, code, LLM-as-judge and pairwise evaluators (LangSmith).
- Langfuse: LLM-as-a-judge, code evaluators, datasets and experiments, online evaluation, user feedback and annotation queues (Langfuse).
- Arize AX: evaluators that return a label, score and explanation at span, trace or session scope, for monitoring, batch runs or experiments (Arize).
Side by side#
Connecting them#
Attify reads LangSmith and Langfuse accounts directly, and takes OpenTelemetry traces: one instrumentation serves both (how to send data). The agent's claim travels as proven.outcome; handoffs as proven.handoff.
Questions
Can Attify read my LangSmith or Langfuse traces?
- Yes. Connect the account with an API key (on Langfuse, a read-only Viewer key). If it is the vendor's own account, the data is witnessed and counts toward a verification only after the customer consents and reconciles it.
Should I put my eval scores in the trace for Attify?
- Send the agent's own claimed outcome as proven.outcome. Your evaluators' scores are yours; Attify tests the claim against what happened in the customer's systems.