AI evaluation, grounded in real human judgment
Expert-calibrated judges for evaluating, monitoring, and improving AI at scale
Expert-calibrated judges. Use ours or build your own.
Judges calibrated to our network of industry experts
Adopt judges off the shelf, built with practitioners who define the standards and reference cases behind every verdict. Use them as they are, or tailor them with your own policies and examples when your use case calls for it.
Browse judge catalogJudges calibrated to your internal team
Start from scratch and calibrate judges to the people who know your product best. Your reviewers set the standards and label reference cases, and our calibration tooling turns their judgment into judges you can run at scale.
Judges calibrated to our network of industry experts
Adopt judges off the shelf, built with practitioners who define the standards and reference cases behind every verdict. Use them as they are, or tailor them with your own policies and examples when your use case calls for it.
Browse judge catalogJudges calibrated to your internal team
Start from scratch and calibrate judges to the people who know your product best. Your reviewers set the standards and label reference cases, and our calibration tooling turns their judgment into judges you can run at scale.
Judges can be used wherever you’re evaluating or monitoring AI today
Every judge is hosted on Forum AI and available through our API. Call it from your existing eval suite, CI checks, or production monitoring, with no models to host and no pipeline to rebuild.
- Hosted on Forum AINothing to deploy, scale, or maintain
- One API, any stackWorks wherever you already evaluate
- Efficient decision modelsFast, low-cost scoring at production volume
Who Forum AI is for
A customer support agent can close a ticket but frustrate a user.
A workplace assistant can send an email without the critical information.
A coding agent can ship a change but add unnecessary complexity.
Forum AI partnered with experienced operators to develop template judges for a wide range of relevant factors, designed to be customized for your use cases.
Features
- 01
Calibrate judges with your team
Build judges from scratch with the people who know your product best. Built-in calibration flows align each judge to your reviewers and adapt as your standards change.
- 02
Pull industry-standard judges off the shelf
Start with judges our network of domain experts has already built and calibrated, ready to run on day one.
- 03
Customize industry-standard judges with your team
Tailor expert-built judges to your own policies and examples, then calibrate them against your reviewers.
Popular use cases
- Customer Communication6 judges
- Productivity4 judges
- Safety7 judges
- Coding3 judges
- Finance5 judges
AI models are powering increasingly consequential tasks.
They offer mental health support. They guide voters during elections. They support new parents.
Forum AI partners with leading domain experts to develop industry-standard evaluations that both capture the necessary nuance and offer third-party defensibility.
Offerings
- 01
Standardized evaluation reports
Recurring industry-standard evaluations to track progress, compare competitors, and surface risks.
- 02
Commissioned evaluations
Bespoke prompt sets, fact sets, and judges tailored to your priorities for internal hillclimbing.
Popular use cases
- Mental Health6 judges
- News & Politics7 judges
- Finance5 judges
- Education4 judges
- Primary Health5 judges
A recruiting team reads hundreds of applications for every opening.
An engineering team triages bug reports one at a time.
A data team checks records by hand before anyone can trust them.
Forum AI judges bring calibrated judgment to the data your team reviews by hand, scoring every item the way your best reviewers would, across the whole pipeline.
Features
- 01
Score every item in the queue
Run bug reports, applications, records, and tickets through judges that rate each one the way your reviewers would.
- 02
Calibrate to your reviewers
Your team labels a sample, and calibration aligns each judge to their calls and keeps it aligned as your standards change.
- 03
Plug into your pipelines
Call judges through one API from wherever your data already moves, and send the uncertain cases back to a person.
Popular use cases
- Bug Report Triage
- Candidate Screening
- Data Quality
- Support Ticket Routing
- Content Review
A teacher grades a stack of essays every week.
A course team scores thousands of short answers at the end of every unit.
An AI tutor gives feedback that students take at face value.
Forum AI works with experienced educators to build judges that assess student work against your rubric, consistently and at scale, with feedback teachers can stand behind.
Features
- 01
Grade against your rubric
Score essays, short answers, and projects against the rubric your teachers already use, with the reasoning behind every mark.
- 02
Calibrate with your educators
Teachers grade a sample, and each judge is tuned until it agrees with them, so grading stays consistent across every class.
- 03
Check AI tutors and feedback
Evaluate the explanations and feedback AI tutors give students for accuracy, tone, and whether they build understanding.
Popular use cases
- Essay Grading
- Short-Answer Scoring
- Lab Reports & Projects
- Feedback Quality
- AI Tutor Evaluation
Expert-calibrated judges for evaluating AI and more. Ready to use off the shelf or be customized.
Adopt, customize, and deploy expert-calibrated judges in one place
Latest research and insights
NewsBench
As AI informs voters, shapes policy, and drives real-world decisions, it must understand what's happening in the world around it. We partnered with world-leading experts to build a benchmark for high-stakes news coverage.
View benchmark
NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting
As AI becomes a primary source of news, what matters is not just whether models avoid bias but whether they are accurate, well-sourced, and fair. NewsBench reframes evaluation around editorial standards set by senior journalists, policy experts, and intelligence analysts — measuring frontier models on source quality, factuality, and neutrality.
Read paper
Distilling Expert Judgment at Scale
Frontier AI is being deployed where the stakes are high and expert judgment is required. We show how to encode that judgment — not just experts' conclusions, but their reasoning — into automated systems that scale, outperforming uncalibrated frontier models on every source-quality metric.
Read paper
How We Turn Expert Insight Into Action
From expert interviews to AI judges — how we transform domain expertise into scalable evaluation systems that improve AI where it matters most.
Read post
Not All Queries Are Created Equal
Engineering a classification system for LLM evaluation — why the type of question matters as much as the answer when measuring AI performance.
Read post
How We Pick the Right Experts to Evaluate AI
Four principles that guide our work — building the expert network that holds AI systems to the highest standards of accuracy and nuance.
Read post
Speed-Running Content Moderation
What fifteen years of social media safety teaches about evaluating AI — lessons from the front lines applied to a new generation of challenges.
Read post
Ex-Meta Executive, CNN Anchor Campbell Brown Launches Forum AI
The Wrap covers Forum AI's launch with $3 million in seed funding to bring expert judgment to AI evaluation.
Read articleIn the news
Forum AI named Fast Company Next Big Thing in Tech
Forum AI's Campbell Brown: AI Needs Public Quality Testing
Forum AI study finds chatbots fail on election accuracy and sourcing 90% of the time
Forum AI's Campbell Brown on AI's accuracy gap in news and information
Forum AI and Allstate CEOs discuss AI's next chapter
Forum AI hosts release of Stanford HAI's 2026 AI Index Report
Forum AI cofounder discusses AI's judgment at Eye on AI
Campbell Brown co-launches Forum AI
Ex-Meta Executive, CNN Anchor Campbell Brown Launches Forum AI With $3 Million in Funding
Join our team
Help us build the judgment layer for AI — scaling expert evaluation across the domains that matter most.
Apply