Judges

Expert-calibrated foundation judges

41 judges

General12

Foundation22 expertsPolicy AdherenceChecks the agent applied your written policy to the case in front of it, and refused what the policy forbids even under repeated pressure.Foundation29 expertsHallucinations & GroundingChecks every factual claim traces back to something the agent actually had — a retrieved document, a tool result, or the user's own message.Foundation25 expertsEscalation to HumanChecks the agent handed off when the situation called for a person — and did not hand off the cases it was there to handle.Foundation28 expertsInstruction FollowingChecks the agent did what was asked, including the constraints — the format, the limit, the thing it was told not to do.Foundation30 expertsUncertainty CommunicationChecks the agent said when it did not know, rather than answering at full confidence and leaving the reader no way to tell.Foundation26 expertsRefusal AppropriatenessChecks refusals land on the requests that warrant them, and that legitimate work is not turned away with a boilerplate decline.Foundation28 expertsTask CompletenessChecks nothing in a multi-part request was quietly dropped — the second question, the attachment, the follow-up the user asked for.Foundation27 expertsPII HandlingChecks personal data was not collected beyond need, repeated back into the wrong channel, or exposed to a party that should not see it.Foundation22 expertsAuthority & ScopeChecks the agent acted only within the permissions it was actually granted — no action it had no standing to take.Foundation26 expertsTone & EmpathyChecks the register fits the moment — that a frustrated or distressed user is met, not processed.Regulation20 expertsGDPR Data Subject RightsChecks access, erasure, portability and objection requests were recognised as such and handled on the terms the regulation sets.Regulation28 expertsEU AI Act TransparencyChecks the user was told they are dealing with an AI system, and that generated or manipulated content is disclosed as such.

Customer Service7

Foundation26 expertsCustomer Service HelpfulnessChecks the customer's problem was actually solved, not merely answered — the difference between closing a ticket and fixing the thing.Foundation23 expertsCustomer CommunicationChecks the customer was left knowing what happened, what happens next, and by when — in language they can act on.Industry20 expertsRefund & Credit AuthorizationChecks refunds, credits and goodwill gestures stayed inside the agent's authority and the conditions your policy sets on them.Industry29 expertsCancellation PressureChecks a retention attempt stayed an offer — that a customer who asked to cancel was not obstructed, guilted, or looped.Industry29 expertsTicket Triage AccuracyChecks the ticket was categorised, prioritised and routed to the queue that can actually resolve it.Industry22 expertsMulti-Turn Context RetentionChecks the agent held on to what the customer already told it, instead of asking for the order number a third time.Industry29 expertsResolution QualityChecks the fix offered would hold — that the case was closed on a real remedy rather than a workaround that returns next week.

Travel4

Industry27 expertsItinerary AccuracyChecks dates, times, airports, time zones and connection times in a proposed itinerary are internally consistent and physically possible.Industry21 expertsFare & Fee DisclosureChecks the price quoted is the price paid — baggage, seats, change fees and taxes surfaced before the customer commits.Industry28 expertsDisruption & RebookingChecks a cancellation or misconnection was handled with the options the passenger is actually entitled to, in the right order.Industry26 expertsTravel Document GuidanceChecks visa, passport-validity and entry-requirement guidance matches the traveller's nationality, route and transit points.

Healthcare6

Industry25 expertsClinical AccuracyChecks clinical statements match current guidance, and that the caveats a clinician would attach are still attached.Industry22 expertsRed-Flag TriageChecks presentations that need urgent care were recognised and directed there, and that routine ones were not escalated into an ER visit.Industry20 expertsScope of PracticeChecks the agent stayed on the information side of the line — no diagnosis, prescription or treatment plan it has no standing to give.Industry27 expertsPatient ComprehensionChecks the explanation is one a patient can act on — plain language, the actual next step, no untranslated clinical shorthand.Industry30 expertsMedication SafetyChecks drug names, doses, interactions and allergy conflicts against the record the agent was given.Regulation29 expertsHIPAA SafeguardsChecks protected health information was disclosed only to permitted recipients, for a permitted purpose, at the minimum necessary.

Financial Services4

Industry20 expertsSuitabilityChecks a recommendation fits the customer's stated circumstances, objectives and risk tolerance rather than the product being pushed.Industry22 expertsRisk DisclosureChecks the downside was stated as plainly as the upside, with the disclosures the product actually requires.Regulation23 expertsPCI DSS Cardholder DataChecks full card numbers, security codes and track data never entered a transcript, log, note or downstream system.Regulation28 expertsFINRA CommunicationsChecks customer-facing communications are fair and balanced, free of exaggerated or promissory claims, and retained as required.

Data & Spreadsheets5

Industry30 expertsFormula CorrectnessChecks the formulas an agent wrote compute what was asked — right ranges, right anchoring, no silent off-by-one across the fill.Industry26 expertsNumerical AccuracyChecks the numbers in the output survive recomputation — totals, percentages, growth rates and the units they are quoted in.Industry25 expertsData Transformation FidelityChecks a reshape, merge or clean-up preserved the data — no rows dropped, no join silently fanning one record into forty.Industry27 expertsSource TraceabilityChecks every figure in a summary can be walked back to the cell, query or file it came from.Industry21 expertsChart & Summary FaithfulnessChecks the chart and the written takeaway say what the underlying data says — no truncated axis doing the arguing.

Software3

Industry29 expertsCode CorrectnessChecks a change does what it claims — behaviour matches the description, edge cases handled, nothing adjacent quietly broken.Industry27 expertsVulnerability IntroductionChecks generated code did not introduce injection, unsafe deserialization, secret exposure or a broken authorization check.Regulation29 expertsSOC 2 Access ControlChecks agent actions on customer systems stayed inside the access controls and left the audit trail the controls assume.

Not on the list

Describe what matters and we draft the judge

The catalogue covers the ground we see most often. Tell us the failure you are trying to catch and we build the judge around it.