Analyst reports and review sites vs. audited outcomes
Analyst reports score vendors on criteria the analyst sets, from questionnaires, briefings, demos and customer references; review sites collect moderated opinions from users. Both help build a shortlist. Neither is a count of one agent's outcomes with a denominator, a period and an interval.
What they say about their methods#
- Forrester Wave: the analyst drafts criteria and invites vendors; participating vendors answer a questionnaire, give briefings and demos, and name reference customers; the analyst scores against defined scales and weights; vendors see their scorecards and get a courtesy preview about five days before publication; the process runs about 18 weeks; Forrester reserves the right to evaluate vendors that do not participate (Forrester).
- G2 reviews: filtered automatically and checked manually; reviewer tags include validated reviewer, current user and incentivized; incentives are capped at $100 a review and labelled; employees of the product's company and of direct competitors may not review; G2 says it never edits a review's content (G2).
Side by side#
Using all three#
Reports and reviews for the market map and shortlist; a pilot for fit; verification for what the deployed agent achieves.
Questions
Do G2 reviews say how well an AI agent performs?
- They say what reviewers experienced and think. They have no denominator, so they cannot give a rate with an interval.
Is a vendor ranked well in an analyst report a safe choice?
- It is a well-evaluated vendor on the analyst's criteria. Whether its agent's outcomes hold up with your customers is a separate, measurable question.