How Duplicate Stories Are Merged

The methodology page describes this step in a paragraph. This is the detail behind it.

The fixed limits

• Only reports published within 48 hours of each other are ever treated as the same event.
• Two reports from the same publisher domain are never merged on the strength of their resemblance to each other. A publication cannot corroborate itself — and because breadth counts domains rather than articles, a publication filing three updates still counts once.
• Nothing here uses a language model. The rules are lexical and deterministic: the same set of headlines always produces the same grouping.
• Merging never hides anything. Every report in a grouped story keeps its own headline, publication name, timestamp and link to the original.

Step 1 — do the words agree?

Two headlines are first compared on surface wording. Three overlaps are measured, each as a share of what the two headlines have between them:

• Significant words — half the weight. Filler such as the, said, will and new is discarded first.
• Adjacent word pairs — three tenths. This is what separates “Ontario bans the drug” from “drug ban hits Ontario”, where the words match but the phrasing does not.
• Named entities — two tenths. The people, places and organizations extracted from each headline.

The combined figure merges a pair at 0.42. Two surcharges apply on this route: a pair whose categories disagree must reach 0.47, and a pair whose verbs actively contradict each other must reach 0.57.

Step 2 — does the event agree?

Word overlap alone cannot carry heavily reworded coverage. “US missionary kidnapped in Niger is released” and “American aid worker freed after months in captivity” are one event and share almost no words. So each headline also gets an event fingerprint:

• Words are reduced to a common stem for matching only, never for display — kidnap, kidnapped and kidnapping become one key.
• News verbs are collapsed into synonym groups: released and freed describe the same act, as do arrested and detained, or dies and dead.
• Words are weighted by rarity within the batch of headlines being processed, so a distinctive word carries far more weight than a common one.
• Calendar words, nationalities and attribution boilerplate are ignored — Tuesday, American, police say, officials said. They describe when something happened, who was involved in general terms, and how it was sourced. They do not identify the event.

A pair may only take this relaxed route on strong evidence: at least 2 shared rare words, at least one of them a proper noun anchoring a place or an actor, and no contradicting verbs. It then merges at a fingerprint agreement of 0.42.

Step 3 — is one headline contained in the other?

Rewordings are often lopsided. One outlet writes a long headline carrying the person’s full name and how long the ordeal lasted; another writes a short one. Measured as a share of everything both headlines contain, the pair looks weak — measured against the shorter headline alone, it is nearly total. So a strong-fingerprint pair can also merge on containment, at a bar that depends on how much the verbs confirm:

• 0.65 when both headlines name the same act;
• 0.72 when only one names it, or neither does — near-total containment is then required;
• 0.62 when the pair shares 3 or more rare anchors — place, subject and object, not just a name — and names the same act.

Step 4 — obituaries

One person’s death is reported as “dies at 82”, “dead at 82” and “Obituary: remembered as”, and the only material those headlines share is the name. Left to the general rules they split into several pages. A dedicated rule merges them when all of the following hold: both headlines carry an explicit death signal, they share a personal-name anchor, and they do not disagree about any other act. Two different people dying on the same day share none of that.

What stops a story swallowing its neighbours

• Contradicting verbs veto. When both headlines name an act and share none, they describe different events however much their nouns overlap — a volcano erupting and flights resuming as the ash clears are not the same story.
• Chains are broken up. Grouping can otherwise link A to C through B. After grouping, each member must still resemble the story’s lead report at 0.25 or support a majority of the other members. Members that do neither are pushed back out into stories of their own.
• The same publisher never counts twice, on any route.

How the merge rules are tested

Every change to these rules is run against a labeled set of headline pairs, each pair marked by hand as the same event, a related but different event, or unrelated. The set is weighted toward deep paraphrases and near-miss traps, because those are where automated grouping fails.

Those pairs are written for the test — invented towns, companies, teams and people — not sampled from live coverage. A score on invented headlines says whether a change made the rules better or worse; it does not say how often CurrentWire gets real news right. So no accuracy percentage is published here, and none should be read into the gates below.

What the test does enforce is a contract, and the contract is deliberately lopsided. A wrong merge corrupts a story page by attaching reporting that does not belong to it; a missed merge only leaves two pages where there should be one. Precision is therefore held far above recall: on that labeled set, the build fails if precision falls below 0.98 or recall below 0.8. The real check on the rules is the corrections process — reports about actual stories, on actual headlines.

When it gets it wrong

Both failure modes happen: two pages for one event, or a report in a source list that does not belong there. Automated grouping without human review cannot avoid that entirely, and the honest position is to say so and fix what is reported. The corrections process explains how.

Fixing it does not break links. A story keeps its identity across refreshes, but its address can still change: the address is built from the headline the page leads with, and a higher-tier publication joining the coverage can take that lead. When that happens the old address keeps working and redirects to the current one. When two pages turn out to cover one event, the older address permanently redirects to the surviving page. Either way, a published story URL never breaks — CurrentWire does not delete story URLs.

Related references

• How CurrentWire ranks the news — all six signals and their weights.
• What coverage breadth measures — what the grouped sources are then worth.
• Publisher authority tiers — which grouped report becomes the one you read.
• Editorial standards — the accuracy, attribution and labeling rules around all of this.