← Back to Shared Signal

How Shared Signal grades

Methodology v1.1 · 16 October 2026

Shared Signal asks whether an article's evidence is checkable. It would be incoherent to ask that of others and refuse it of ourselves — so here is the rubric, the arithmetic, the blind spots, and the times it has been wrong.

What a grade is a claim about

Shared Signal grades construction, not truth. It reads the words in front of it and asks how well the piece supports what it says: whether claims are attributed, whether conclusions follow, whether a reader has the background needed to weigh it, and whether the language informs or provokes.

It does not check whether anything happened. It has no retrieval, no web access, and no way to look anything up. A low grade means the piece is weakly built as presented. It is not a finding that anyone lied, and it is not a judgement of the publication or the journalist.

Grades apply to one article. Shared Signal never averages a publisher.

The four dimensions

Each is graded independently, A to F. These are the actual criteria the grader is given, not a summary of them.

Observations and their ratings

Under each dimension the grader names one to three observations: something the article does, restated plainly so a reader can check it or argue with it. Since v1.1 each observation also carries its own rating, on a scale that belongs to its dimension, with one sentence saying why.

"Unsupported" means the article offers no adequate support for that point. It never means the thing did not happen. "Undetermined" is used when the text alone does not allow a judgement. Observation ratings explain a dimension grade; they do not feed the arithmetic below, which still uses the four letters alone.

Readers can attach evidence to any observation, saying whether it supports or challenges it, and can suggest something the grader missed. Suggestions are reviewed by a person before anyone else sees them.

The bar

A deliberately demanding scale. If most things scored B, the grade would carry no information.

How the overall grade is calculated

The model does not choose the overall grade. It grades the four dimensions; the overall is arithmetic, computed in code, from those four alone. Every dimension carries equal weight.

This is deliberate. An earlier version let the model write its own overall label and it produced "Fabricated Scenario" — a verdict about reality that nothing in the rubric authorised and no dimension supported. Deriving the overall in code means it can only ever reflect what was actually assessed.

Equal weighting is a choice, not a discovery. There is a reasonable argument that sourcing should count for more than tone. We have not made that argument because we have no evidence for a particular ratio, and inventing weights would be exactly the kind of unsupported precision this product exists to catch. If the reader-grade data eventually shows a defensible weighting, that becomes a versioned change to this page.

Factual Accuracy is not graded

There is a fifth dimension in the data, and it returns unverified. It is not shown as a grade and contributes nothing to the overall grade.

The reason is specific. A language model answering from memory cannot distinguish a false claim from an event more recent than its training data. In calibration, the grader assessed a real assassination as fabricated with 95% confidence. It was not confused about journalism; it simply did not know the event had happened.

Retrieval ships before that dimension does. Until then, Shared Signal will not assert that anything is true or false, and the guard against it is structural in the code, not a line in a prompt.

When we withhold a grade entirely

Removing the factual grade did not remove the belief behind it. The same mistaken premise leaked into every other field: Evidence graded F, Context complaining about missing proof, prose full of "supposedly" and "purported".

So every piece of generated text is screened before a reader sees it. If the grader's own writing suggests it doubts the described events occurred, the entire judgement is withheld — the overall grade, every dimension grade, every description and quote. What survives is the flag, the reason, and the extracted claims, which restate the article rather than judge it.

One thing a withheld grade does not mean. It is not evidence that a story is contested or disputed. The grader saw no conflicting source, because it saw no sources at all. Saying otherwise would be asserting something we have not measured — the exact failure this product exists to catch, committed by us.

What gets published, and what never does

Before any reading appears on the public feed, the URL is fetched again from our own server with no cookies and no session, to answer one question: can a stranger open this?

Pages that need a login are never published — intranet pages, shared documents, email, anything on a private network. Subscription news articles are not published either. The reader who ran one still gets the full reading, but it is shown only to them, because the text came from their own paid session. A shared document is a public URL that redirects to somebody else's sign-in host. Nothing is called paywalled unless the page it landed on is still the publisher's. Anything the check cannot categorise is refused.

The feed carries the headline, the grading, a link, and at most one verbatim quote of 25 words. It never carries article text. One card per article, never per person.

What this method cannot do

• It cannot tell you whether something is true. No retrieval, no sources, no lookups.
• It is unreliable on recent events — the more recent, the less reliable, and the failure presents as confidence.
• It reads text, not pictures. A misleading chart or a doctored photograph is invisible to it.
• It has not been audited for political bias. We have not yet run a stratified sample across the spectrum with blind human raters. Until we have, treat any pattern you think you see as unmeasured — including by us.
• It is not deterministic. The same article run twice can differ by a grade step on a dimension.
• One model, one vendor. Every reading is produced by a single model. Its dispositions are our dispositions.

How this page changes

Anyone can challenge the rubric, a specific reading, or anything on this page. Only the maintainers change it. Every change is versioned and dated here with the reason, and challenges we decline are recorded too — a page that only shows the arguments we accepted would be advertising.

Every reading records the methodology version that produced it, so a grade from v1.0 can always be read against the v1.0 rules rather than whatever the rules became later.

To challenge something, email hello@sharedsignalcommunity.com with the article link and what you think is wrong. Publishers get the same path readers do.