Content Analysis - Statement Detection

Automatically detect potential spoilers across a large volume of articles, link them to internal key secrets and surprises, and give editorial teams the ability to override any AI decision at any stage.

An editorial organization needed to scale a content review process that was entirely manual. The volume of articles exceeded what the team could review by hand, but the sensitivity of the content required human judgment at every decision point. They needed AI to handle the volume while humans retained control.

Challenge

The business rules governing what constituted a "spoiler" were complex, contextual, and constantly evolving. Different content categories had different sensitivity thresholds. The editorial team needed to see exactly why the AI flagged something, override it when the AI was wrong, and have those corrections improve future performance. Building a black-box detector was not an option.

What We Built

We broke the problem into five sequential stages: content parsing, statement detection, semantic similarity checking against a vector store, confidence scoring, and linking with structured output. AI makes suggestions at each step. A human override interface sits at the end, and every correction feeds back into a golden dataset, improving performance over time.

The architecture required deep immersion in the business before any code was written. Understanding how the editorial team defines and categorizes sensitive content shaped every design decision. Total effort was roughly 2,500 hours across the full team, with 850 hours on the AI side. A significant portion went into building golden datasets and evaluation flows that test each pipeline stage independently.

What Changed

The editorial team can now review content at scale without sacrificing the judgment calls that matter most. Complex editorial workflows benefit from decomposition into discrete AI-assisted stages. Budget time for evaluation infrastructure, it is what gives stakeholders confidence that the system is working correctly and improving over time.