Methodology

What this ledger measures

SooLedger records public safety and disorder incidents in Sault Ste. Marie as a lifecycle: an incident is made public, someone may be arrested, charges may be laid, a court may reach an outcome, and council may respond. Most sources only ever count the first step. We track all of them, and we say plainly where our records stop.

Every record is located to a hundred-block (for example, "200 block Lake Street"), never an address. We never publish names, ages, descriptions, licence plates, or the names of businesses — even when the source document contains them. The exact time of day is never stored; where a release states a time, only a six-hour band (overnight, morning, afternoon, evening) is kept, and it does not appear on this site. A charge is an allegation until a court decides it; a record at the "charged" stage says only that police reported laying a charge.

Sources

There is no news source. We do not scrape local news sites without a licence. We do not collect court daily lists: they contain names, cover only two days, and their terms prohibit collection.

Every source fetch is polite: one request every three seconds per site, honouring each site's robots.txt, identifying ourselves with a contact address.

How records are created

1. Each release or document is snapshotted exactly as published, with a cryptographic hash and timestamp. The snapshot is never edited and never published; it exists so a record can be traced to its source and a takedown verified.
2. A language model extracts structured fields (type, date, hundred-block, stage, charges) into a strict schema. It is not allowed to write free text about an incident; the one-line summary you see is composed from the structured fields.
3. Two privacy checks run on every field. The first is a deterministic scan for names, ages and youth descriptors, civic addresses, licence plates, phone numbers and email addresses. The second is a separate model call whose only job is to look for anything that could identify a person or a business, including descriptions the pattern scan cannot catch. Either check firing removes the free-text location detail from the record.
4. The street is resolved against a city-wide street index built from OpenStreetMap; the block is derived from the hundred-block in the source text.
5. Each document is extracted exactly once at a time: a consumer claims it before any model call, so a duplicate delivery from the queue cannot write a second copy, and a re-extraction of the same document (after a prompt fix) withdraws what the earlier run published before it writes, with a corrections-feed entry for each withdrawn record.
6. The record is compared against existing records to link follow-ups (an arrest reported after an incident, a court outcome months later) and to avoid double counting across sources. Follow-up releases are matched to the original by the cited date, street and incident type; a follow-up that cites none of these cannot be linked and appears as a separate record.

Publication rules

There is no human review. Deterministic rules publish or discard every record, and no person edits a record's content. The rules are:

• Source. Only records from police releases, archived police releases, Police Services Board reports, SIU releases, council documents, the intake form and partner feeds may publish. Anything else is discarded.
• Confidence. The model reports a confidence with each extraction. Below 0.5 the record is discarded: the model is telling us it guessed. Between 0.5 and the per-source threshold (0.7 for police, archive and SIU releases; 0.75 for council items; 0.9 for Board statistics) the record publishes and is flagged low_confidence so we can watch it and tune the prompts.
• Privacy. If either privacy check fires, the record publishes without its free-text location detail. The structural fields (type, block, date, stage) still publish.
• Street. If the street in the source cannot be matched to the city's street list, the record publishes without a block and is counted city-wide only.
• Date. If the date is implausible relative to the source's publication date, the record publishes with the publication date at week precision instead.
• Youth. Records involving a young person are coarsened to week precision and published without a block.
• Volume. A document that produces more than five incidents, or a run that produces three times the trailing average, is flagged anomaly_hold for the operator's attention; the records still publish.

"Held back" on this site means discarded: a record that failed the source or confidence rule and was never published. Every rule that fires is logged with what it did, and the count of records held back appears with the summary on the home page and on the block, type and council pages, with the detail on the quality page.

The operator can remove a published record or send it back through the rules, and every such action is written to an audit log with who did it and what changed.

How the thresholds are set

The confidence floor (0.5) and the per-source flag thresholds above are constants in the publish gate. They are not tuned by code; they are read off a measured curve and changed by hand, and the curve is public.

1. A labelled set. Ninety-odd real police releases (daily releases and archived captures back to 2021) are labelled by hand with the exact extraction we expect: which incidents the release contains, their type, date, hundred-block, stage, charges and the optional detail fields, or — for notices, missing-person appeals, traffic matters, sexual offences and intimate-partner violence — that it contains no incident at all. The set is stratified so that every failure mode found in the data audit is represented (multi-incident releases, follow-ups, youth releases, out-of-scope matters, weapons and mischief arrests bundled with breach charges, releases with no usable location) and it includes a handful of releases with instructions planted inside the text, which the model must ignore. No expected output contains a name, an age or a youth descriptor; the test suite refuses any label that would.
2. A live run. The production model and prompt extract every release in the set. Each item the model returns is paired with the labelled incident it corresponds to (same type, date within the stated precision, same block) and marked right when type, date, street, block and stage all agree, partly right when it matched an incident but got one of those fields wrong, and wrong when it corresponds to no labelled incident. Each item also carries the confidence the model reported and the record quality score described below. A replayed run (the offline test suite feeds the labels back as if the model had produced them) scores perfectly by construction and is flagged as carrying no information about thresholds. The raw output of every live run is kept beside the public record (audit action evals.police.predictions), so a run can be re-scored offline when the scorer or the score's weights change, without paying for the model again.
3. Two curves. For each candidate threshold from 0.5 to 0.95, per source kind and pooled: how many items sit at or above it (what the gate would publish), what share of those are right (core precision), what share matched a labelled incident at all (detection precision), and what share of the labelled incidents they recover (recall). The curve is read twice — once on the model's confidence, once on the quality score — and both are written into the eval record with the run and published with the quality data (/v1/cities/ssm/quality, fields evals.thresholds and evals.thresholdsScore). Alongside each curve: reliability buckets (does an item reported at 0.9 turn out right nine times in ten) and a Brier score (the mean squared distance between the number and the outcome; 0.25 is what a constant 0.5 scores, lower is better).
4. The rule. The recommended flag threshold for a source kind is the lowest candidate at which core precision is at least 0.95 with at least ten items above it. The recommended floor is the lowest candidate at which core precision is at least 0.8: below that, an item is discarded rather than flagged. Where no candidate qualifies, or the kind has fewer than ten predictions in the set, the record says so and recommends nothing.
5. Applying it. A recommendation is advice to the operator, alongside the current gate value; nothing edits the gate automatically. When a prompt version changes, the set is re-run and the curve re-read before the version is switched on. Two live runs of the same prompt on the same day differ on about one release in seven (precision 0.84 and 0.81, recall 0.82 and 0.78 on 2026-09-10), so a change smaller than that is noise, not evidence. Numbers from the small archived-capture stratum are treated as indicative only until that stratum grows.

The scoring also reports, per run, a document-level confusion matrix (did the model find an incident where the label says there is one, and stay silent where there is none), per-field accuracy on matched incidents, the same accuracy on each optional detail field, and the hard privacy count, which must be zero.

The record quality score

The model's confidence turned out to say almost nothing: on the live runs of 2026-09-10, 68 of 71 items sat between 0.85 and 0.95 whatever their accuracy (about seven in ten were right), no threshold on it separates right records from wrong ones, and the recommender returns nothing for every kind. So every record now also carries a quality score, computed by the gate from the extraction and the document it came from, with no extra model call. It is stored beside the confidence (column quality_score, internal, never published) and no rule reads it yet: it exists so the thresholds can be set on a number that carries information.

The score is a weighted sum of checks, each in [0, 1], minus two penalties, clamped to [0, 1] (packages/prompts/src/quality.ts; the tests assert it is bounded and monotone in every feature):

Follow-up status is recorded as a feature but carries no weight: on the labelled set follow-ups were right more often than other items. The weights were set by hand on a proxy of the live run — the production model's output under the previous prompt version on the same 93 documents, scored against the same labels (75 predicted items) — because the live runner did not yet keep raw output; they are to be re-read from the first live run under this build.

What the proxy showed (2026-09-10, gpt-5-mini, prompt v3 output, 75 items, 50 right on every core field):

Reliability on the score: items in [0.9, 1] were right 84 % of the time, in [0.8, 0.9) 36 %, below 0.8 about one in six. The features that separate right from wrong on their own: block resolved (74 % right vs. 11 % unresolved), block stated in the text (77 % vs. 15 %), date stated in the text (71 % vs. 0 %), fewer than five charges (73 % vs. 50 %), at most one secondary type (72 % vs. 50 %), one item per release (70 % vs. 44 %), type not "other" (69 % vs. 0 %).

Per kind, on the score: archived captures reach core precision 0.94 (16 of 17, detection 1.00) at 0.95 and 0.83 at 0.85 (29 items, detection 0.97); daily releases reach 0.86 at 0.95 (21 items) and 0.85 at 0.90 (26 items, detection 0.92). No kind reaches 0.95 core precision with ten items, so the rule in step 4 still recommends nothing. What remains above 0.9 is almost entirely one error: the primary type chosen among several applicable keys in a release listing many offences. The v5 candidate remains inactive until a live A/B run passes the published thresholds.

Recommendation (not applied): until a live run under this build confirms the curve, keep the confidence constants as they are and read the score as a second, flagged tier: flag low_confidence below 0.85 on the score for police releases and archived captures (proxy: core precision 0.80 and detection precision 0.92 above it, recall 0.61 of labelled incidents), and discard below 0.5 on the score (only items the relevance rules or every check reject). On the published corpus of 2026-09-10 (225 records, scored against their own documents) that flags 50 records (22 %; 8 of 48 daily releases, 42 of 177 archived) and discards 3; a cut at 0.9 would hold back 81 (36 %) and one at 0.95 would hold back 134 (60 %), which the precision gained does not justify. The held-back records fall short mostly on the multi-offence terms (44 list five or more charges, 37 carry two or more secondary types), on a street the index could not resolve (19), or on a date the text does not state (17).

Two over-exclusions in the deterministic relevance rules surfaced while scoring the labels, each putting a correctly labelled record below the floor: a title naming a bundled traffic charge ("Calgary Driver Charged with Dangerous Operation, Assault with a Weapon" is an assault) and the charge "Mischief – Obstruct, Interrupt or Interfere with the Lawful Use of Property", whose "obstruct" reads as a process charge (one published and three rejected production records carry it). Both are left as they are here and listed for a rules fix.

Council items

Council and Police Services Board items are extracted one agenda item at a time. An item publishes only when it acts on public safety and disorder or its causes: policing and its budget, security measures, encampment and drug-use response, downtown safety programs, enforcement against nuisance properties, shelters and crisis services, and the money for those. The model must state, in one sentence drawn from the item's own text, what the decision does about disorder; without that sentence the item is dropped. A second, deterministic check requires the item's title and summary to name such a response, and a list of named exclusions (parking, wildlife, snow, road repair, public toilets, carpet, year-end financial reports, and similar) drops items even when the model kept them. Roads, zoning, taxes, tariffs, tenders and routine by-law housekeeping are left out entirely. The confidence floor for council items is 0.75. Identifying words in a title or summary are removed before publication.

For council items we publish the item's title and a short paraphrase of what was decided, with a link to the agenda or minutes. A promise with a due date shows as "overdue" once the date has passed with no follow-up recorded. Follow-ups are recognized by matching a later item's title (or by-law number) against the original within the same meeting series, so a follow-up filed under a different title is missed and the promise may stay marked overdue.

Business and resident reports

A report counts on the block the moment it is submitted. Instead of a reviewer, three automated guards apply: the reporter's email address was confirmed by opening a sign-in link; the incident date is within the last three years and not in the future; and no business or resident may file more than ten reports in thirty days. Photos are private evidence seen only by the operator and are never published or linked.

Partner feeds

A partner is an organisation the operator has set up by hand and that sends rows from its own log. Rows publish on delivery, under these rules: the message or upload must come through the partner's own credential (a per-partner feed address and page; email from any other domain than the partner's is refused at delivery); the date is within the last three years and not in the future; the type is one of the ledger's sixteen categories, matched from the partner's own words (a word the ledger cannot place is refused, never guessed); the location resolves through the city's street index to a hundred-block, or to a street alone for an intersection the block map cannot place; duplicates (the partner's own reference, or the same day, place, type and time band) are skipped; at most 2,000 rows per delivery. The address the partner wrote is discarded once the block is known. Every delivery is recorded with what was accepted and, per row, why the rest was not. See the partner program.

Block walks

A walk is recorded once per block per person per day; a second submission the same day corrects the first. A walk may be dated up to 30 days back. The block page shows the latest walk's values only once three different people have walked the block in the last 90 days.

How the numbers are counted

• The funnel is cohort-based. "Of incidents reported in this window, N have since reached arrest, charges, or an outcome." Later events attach to the incident they belong to, so an arrest reported after the window still counts for an incident inside it.
• Police releases are selective. The Sault Ste. Marie Police Service publishes a release for roughly one in a hundred of the incidents the Police Services Board counts, and releases are mostly issued when charges are laid. That is why the share of records at the "charged" stage is high: it reflects what police choose to publish, not how often incidents end in charges. The quality page lines our counts up against the Board's each month.
• Outcomes lag. Court outcomes appear in police releases only occasionally, and courthouse statistics are annual and aggregate. The "outcomes recorded" figure is honest but low; it should not be read as a conviction rate.
• Trend lines are event-based (by the week the event happened).
• Same-source items are never merged. Two releases about the same block are counted as two incidents. Cross-source records are merged when type, block and date agree and a second detail (target or damage) agrees; when only type, block and date agree they are also merged, on the reasoning that undercounting a rare coincidence beats double-counting the common case.
• Incidents whose date is only known to the month are counted in monthly totals, not weekly ones.

Privacy

• Locations are hundred-blocks. Business names never appear, even when a source names them.
• Business reports are shown for a block only once at least three distinct businesses on that block have reported — the count as well as the repair-cost total, because "2 reports" on a block with one shop identifies the shop. Below three the block page shows nothing, and an empty block looks the same as a suppressed one; the number of reports held back this way is on the quality page.
• Partner feeds follow the same rule with their own population: a block shows a partner count only once three distinct partners have rows on it, and a combined first-party count (businesses and partners together) only once three contributors of either kind exist. A partner organisation is never named anywhere: not on the site, not in the API, not in licensed exports. The quality page says how many organisations are sending rows and how many rows a month. A partner's notes and reference numbers are private; the address in a row is resolved to a block and not kept.
• A block's condition (from walks) is shown only once three different people have walked it in 90 days, and shows the latest values; who walked a block, and their notes, are never published.
• Photos a business or resident sends us are private evidence, seen only by the operator: hidden camera data (location, device, timestamps) is removed before storage, they are stored privately on Cloudflare's network, they are never published or linked, and they are deleted with the rest of a reporter's data on request (and after seven years regardless).
• Whenever a privacy check fires, the affected detail is removed before publication (an incident's free-text location, a council item's flagged words); the structural record still publishes.
• Deleting your data removes your private fields, withdraws your reports from the counts, deletes your photos, and replaces your email address with a hash in the audit log.
• If you believe a record identifies you or your business, or is wrong, contact us; the operator takes the record down at once (it disappears from every page and the cached copies are purged), the removal is logged with its reason, licensees are notified through the corrections feed, and a taken-down record is never served again even if its status were later changed.

Local context

Beside the numbers, a block page, a zone page and the delegation kit show a short list of local context notes: plain facts about the ground the numbers sit on, such as "Boat season: the waterfront blocks see visitor traffic June to September", "Streetlights on this block out since May per the city's 311 log", or "Rotaryfest, third week of July: Queen Street closed". They are written by verified locals (the operator, the BIA, partner organisations whose feed we have confirmed) and published by the operator; the model never writes one. Each carries the window it applies to and who said so, as a label (operator, BIA, a verified partner, OpenStreetMap for a seeded place); a partner is never named.

The rule for a note is the site's rule: no names, no addresses, no ages, no contact details, no opinions. The same identifier check that runs on every extracted record runs on every note before it is saved, and a hit rejects it. Notes are editorial context, not incident data: the no-human-review rule above applies to records, which publish or are discarded by deterministic rules; a note is a sentence a local wrote, and the operator reading it before it publishes is the right control. A note never changes a count.

Two more parts of the same layer are configuration. A places gazetteer lists the named places a police release refers to instead of a street (the transit terminal, Station Mall, the Bondar pavilion, the hospital), each on the hundred-block of its public address; when a release names a place and gives no street, the record is placed on that block, and the extraction audit records that the place, not a street, placed it. A recurring events list (each event with its usual week or days, the streets it affects and a source URL from the city's own calendars) is drawn on the zone dashboard's weekly line and listed in the monthly summary as "what was on"; it marks weeks, it does not assert dates, and nothing is scraped. Details: docs/LOCAL_CONTEXT.md.

Reconciliation against official statistics

Every month we line our published counts up against the Police Services Board's figure for the matching category (for example, our break_enter_business against the Board's "Break & Enter ~Business"; our mischief_property and mischief_graffiti together against "Mischief"). The comparison is public at /v1/cities/ssm/reconciliation and is honest by construction: because we only count incidents that appear in a media release, our capture rate is far below 100% for high-volume categories (theft, mischief) and closer for the incidents police choose to publicize (robbery, break and enter into businesses). The Board's figures are the ground truth for volume; our records add the lifecycle, the block, and the source link that the totals do not carry.

Board figures are revised in later reports ("statistical classification revisions cause figures to change perpetually", as each report notes); we keep the most recently published value and record which report it came from. Annually we do the same against Statistics Canada table 35-10-0180-01 (/v1/cities/ssm/reconciliation?against=statcan), which gives the police service's official calendar-year counts back to 1998 under the Uniform Crime Reporting classification; our taxonomy carries the matching UCR codes, verified against that table's own classification codes. StatCan revises prior years with each release; as with the Board figures, we keep the most recently published value and record the series it came from.

What runs unattended

Nothing in the pipeline waits for a person. Fetches run on a schedule; a source that fails is retried, then parked on a dead-letter queue with the failure recorded and an alert opened; a source that stops yielding items (three empty fetches in a row, or a fraction of its usual volume) is flagged as a broken parser and its "last successful" time is not advanced; a source that has never succeeded is flagged after three attempts. When the daily model-spend cap is hit, waiting documents are parked until the cap resets, not retried in a loop. A stalled archive backfill is re-kicked, and one paused by a site error resumes itself after six hours, up to five times. Every alert closes on its own when the condition clears. The nightly job that rebuilds the public numbers writes a heartbeat; when that heartbeat is more than 36 hours old the status endpoint says so (cron.stale) rather than serving quietly stale figures.

Known gaps

• Daily police media releases only begin in July 2026; earlier history depends on archived captures (Internet Archive) and is incomplete.
• Court outcomes are rarely in police releases and lag by months to years.
• Council items are only as complete as the model's reading of each agenda item; an item that acts on disorder without saying so in its own text is left out.
• Security measures reported by businesses are self-reported counts; nothing verifies them.
• Partner rows and block walks are what the partner or walker recorded; nothing verifies them, and the mix of what different organisations count as an incident varies. Their counts are shown separately from the police record and never merged into it.
• The downtown boundary is a first approximation and will be refined.

Attribution

Street index © OpenStreetMap contributors (ODbL). Crime statistics from Statistics Canada, reproduced under the Statistics Canada Open Licence.