Building an Agentic Newsroom for AI

TL;DR. AI Brief tracks what labs and researchers ship in AI, from their own announcements, and a human editor approves everything before it is published. Behind it I'm building an agentic newsroom: editorial agents that report, write and fact-check a weekly brief while code enforces the rules. The agents now write the Weekly's drafts, after beating a single-step pipeline on evals over real weeks. The Weekly isn't open to readers yet. This first post covers the principles behind the system. Later posts cover individual parts in depth.

---

The problem: nobody can keep up

If you work in AI, you know the feeling. A new model release lands on Tuesday. Wednesday brings three papers that claim to beat it, a framework release that changes how you would build on it, and a benchmark nobody had heard of the week before. By Friday, Tuesday's news is already old.

The usual answers don't hold up well:

• Following everyone on social media gives you speed with no filter. You see everything, including the hype, the repeats and the wrong takes.
• Newsletters written by people are good to read, but one person can only read so much, and the field publishes more than that every day.
• Asking a chatbot "what happened in AI this week?" gets you a fluent answer that you can't check. It may be right, it may be out of date, or it may be made up, and you have no way to tell.

I wanted something different: a brief that covers the field the way a newsroom would, that anyone can check, and that doesn't need a team of people to run. That is AI Brief.

The idea: a newsroom, not a summarizer

The easy way to build an "AI news" product is a pipeline: collect links, ask a model to summarize them, and publish the result. I built that first. It works, and it reads like what it is, a list of summaries.

A real newsroom works differently. Editors decide what the story is before anyone writes. Reporters go back to the sources. Writers write, editors cut, fact-checkers check, and someone with authority signs off. Each role has a narrow job and a clear handoff.

AI Brief copies that structure with agents:

The rest of this post covers the principles that make that structure trustworthy.

---

Principle 1: Go to the source

Most AI news is a summary of someone else's summary, and errors get added at every step. AI Brief starts from primary sources: the papers, release notes, changelogs and announcements that labs and open-source projects publish about their own work.

This is an editorial choice and a legal one. Stating facts is fine. Copying someone else's sentences is not. So the system has hard rules about this:

• Summaries are written in our own words. If a summary can't be produced, the story is not published. The system never quietly falls back to copying the source's opening paragraph.
• An automated originality check compares what we write against the sources it cites and flags any long passage we share with them, including in the reference data (model, tool and benchmark descriptions).
• Every source has a rights status, and any source can be taken down in one step, everywhere its content appears.

Launching with fewer sources and no copying risk is better than launching with more and hoping nobody notices.

Principle 2: Agents do the work, people make the decisions

"Fully automated" is a tempting claim. I deliberately didn't make it.

In AI Brief, nothing a model writes publishes itself. The weekly brief arrives as a draft: a human editor reads it, approves or rejects it, and only then does it go live. The newsroom of editorial agents (an editor-in-chief, reporters, desk writers, a fact-checker and an editor) writes that draft. It took over from a single-step pipeline by beating it on the evals below. The same pattern is planned for the reference data (models, tools, coding agents, benchmarks): agents find what is new or what has changed, research it, and put a proposal with its evidence into an inbox. A person approves it.

This is not a lack of confidence in the agents. It's a design choice about where accountability sits. Agents are very good at the tiring 95% of the work: reading everything, every day, without getting bored. The 5% that carries the reputational risk stays with a person, and that 5% is short enough to review well.

It also gives the system a measure that no benchmark can: how often the editor approves a draft, and how much they had to change before publishing it.

Principle 3: Models write, code checks

This principle shapes the codebase more than any other.

Language models are good at judgment and writing: deciding what the week was about, finding the angle, saying it clearly. They are unreliable at following rules: counting words, citing correctly, not repeating last week's lead story. So the work is split along that line.

In the newsroom, every model call goes through a declared step. A step has:

• an output contract (structured output that is validated in code, not trusted as free text),
• a code check that runs on every output,
• a retry limit, with feedback that tells the model exactly which field failed and why,
• a fallback, or a clear failure if there isn't one,
• and an entry in a call log: which step, which prompt version, how many attempts, the outcome, the cost.

A step can end in four ways: ok, repaired, fell back or failed. "Silently returned something odd" is not one of them.

A small trick that turned out to matter: prompts ask for less than the check allows. If a field can be at most N words, the prompt asks for noticeably fewer. Models count loosely, and a little slack turns a frequent retry into a rare one.

Principle 4: Every claim traces back to a source

Before the newsroom sends a draft for review, the finished issue goes through code-only checks:

• Every link must point to a source the system actually read during this run, and every story must cite at least one.
• Every link must resolve, with no redirects to somewhere unexpected.
• No story may repeat one from recent issues.
• The issue must fit its schema, length limits and style rules.

If any check fails, the run stops, records which piece failed and why, and sends nothing. A missing issue is a problem I can fix. A wrong issue that has already been published can't be taken back.

This is the main difference from asking a chatbot about the week: the reader can follow every link back to the source.

Principle 5: One story is one story

When a major model launches, the lab's announcement, the paper, the GitHub release and the discussion threads all arrive within hours of each other. A naive feed shows that as five items. A newsroom shows it as one story with five sources.

AI Brief groups items by meaning rather than by headline, using semantic similarity together with rules that encode editorial judgment. For example, a model launch and the policy response to it are related but separate stories. Two consecutive patch releases of the same library look almost identical to a similarity model but are not the same story either.

A related lesson I learned the hard way: one number should mean one thing. At one point, a single score field held both "how relevant is this to AI?" and "how much is it trending right now?", and the second overwrote the first. Once a story was a week old, its trending score had decayed to zero, so the system lost its record of how relevant that story was. Separating the two signals fixed ranking, the weekly brief's story selection, and the grouping of stories, all in one change.

Principle 6: Measure before you ship

Prompts, models and agent designs change all the time. Without measurement, every change is a guess.

AI Brief's newsroom has an evaluation harness built on frozen weeks: snapshots of a real week's content that can be replayed without touching the live database. Any change to a prompt, model or agent produces a new issue from the same input, and that issue is scored on:

• Pre-submit pass rate: does it pass the code checks?
• Faithfulness: is each claim supported by the sources it cites? This is graded by a fixed judge model that is never the model under test.
• Editorial quality: does the lead say what the week was about, and were stories chosen for importance rather than recency? I check the judge against my own blind picks of the same weeks.

Results are stored per configuration, so "is the new prompt better?" has an answer backed by data instead of a gut feeling. The same evals decided when the agents took over. The rule was set in advance: the newsroom becomes the default writer only if it passes the code checks as reliably as the single-step pipeline, is at least as faithful to its sources, and writes better. It took several rounds of fixes to get there, and each round was measured on the same frozen weeks. The pipeline stays as the baseline every later change is compared against.

Principle 7: Build the engineering process the same way

I'm applying the same idea to how the platform itself is tested. I've built separate agents for each kind of check:

• a QA agent reads the diff, writes or updates tests for it, runs them, and explains any failure;
• a security agent runs dependency audits, static analysis and secret scanning, and tries prompt-injection attacks against the AI-facing parts of the app;
• a performance agent runs load scenarios and flags any slowdown against a stored baseline;
• a management agent combines all of that into one release-readiness view and files issues, each with a suggested root cause.

Wiring them into one pipeline that runs on every push is still in progress: each agent lives in its own repository, and the orchestration has to bring them together. Penetration-testing and capacity-testing harnesses sit alongside them for deeper runs before a release.

The platform itself is built contract-first: the public site, the newsroom and the internal tools talk to each other only through written contracts, so any one of them can be rebuilt without breaking the others.

Principle 8: Keep the infrastructure boring

The most surprising thing about AI Brief may be how little infrastructure it needs. The public site runs on a single server with standard, proven parts and self-hosted monitoring. Model inference is a pay-per-use API, so costs follow usage rather than sitting on idle GPUs.

Spending little on infrastructure is what makes the rest affordable: the checks, the evals, the review step. Each of those costs model calls, and a lean base leaves room for them.

---

What's live at launch

• A primary-source AI feed: what labs and projects shipped, from their own announcements, release notes and papers, summarized in our own words and grouped by story.
• Research papers: new papers with links back to the original.
• Reference data: models, tools, coding agents, benchmarks and a timeline of AI milestones, each entry with its sources credited.
• Ranking by reader type: the same news, ordered for what different readers care about.

What's coming

• The Weekly. A summary of the week, written by the agent newsroom and approved by a person before it's published.
• The agent test pipeline on every push: QA, security and performance agents wired into CI.
• Agentic discovery. Agents keep the model, tool, agent and benchmark catalogs current. They notice new entries, price changes and deprecations, research them, and send evidence-backed proposals for a person to approve.
• More primary sources: more labs and projects announcing their own work, each added after a review of its terms.
• This blog, every week. Each post will cover one part of the system in depth.