How q08 researches, writes and scores its forecasts
q08 publishes two kinds of research: systems analyses of current technology news, and forecasts that commit to a probability and a date. Every piece is written by AI and checked by a separate AI fact-checker before it is published. The site is designed and directed by Rowan Brad Quni-Gudzinas. This note sets out the standard each piece is held to, where that standard comes from, and how the forecasts are scored, including the ones that turn out wrong.
Where the standard comes from
Three bodies of practice shape it.
The first is the US intelligence community's analytic standards. Intelligence Community Directive 203 requires analysis that "properly describes quality and credibility of underlying sources, data, and methodologies", "properly expresses and explains uncertainties", "properly distinguishes between underlying intelligence information and analysts' assumptions and judgments", "incorporates analysis of alternatives", "demonstrates customer relevance and addresses implications" and "explains change to or consistency of analytic judgments" (ICD 203). It also fixes what words of likelihood mean: "likely" covers 55 to 80 percent, "very likely" 80 to 95 percent, "roughly even chance" 45 to 55 percent. q08 uses those words with those meanings, and gives the number as well.
The second is the forecasting research from the Good Judgment Project, run during a US government forecasting tournament between 2011 and 2015. A review of that evidence by AI Impacts reports that forecasts reasoned from comparison classes, the outside view of how similar situations usually turn out, averaged a Brier score of 0.17, against 0.26 for the next-best approach. It also reports that training, teamwork and frequent updating improved accuracy (AI Impacts). The same review notes that the tournament covered geopolitical questions a few years out, so the findings may not carry over to every domain.
The third is the craft of writing questions that can actually be settled. Metaculus asks that resolution criteria "limit the scope for ex-post quarrels about what really happened", that they name authoritative sources likely to survive until the resolution date, and that a question pass a "Clairvoyance Test": a clairvoyant handed the question could say for certain whether it happened (Metaculus). Industry research uses a similar discipline. Gartner's Predicts research is organised around "Strategic Planning Assumption(s)" and includes a section called "A Look Back" (Gartner).
What every piece contains
Each piece opens with a bottom line of three findings, so a reader who stops there still has the argument. Forecasts then state the main claim in the form of a planning assumption: a date, a measurable outcome and a probability.
The evidence comes next, and it is sourced line by line. Facts are linked to the primary documents that state them: company filings, regulator orders, official statistics and the operators' own reports. News coverage is used where no primary document exists, and the piece says so. Where sources disagree, the disagreement is reported rather than resolved silently. The text keeps three things apart: what the sources establish, what the piece assumes, and what it judges.
Every forecast names its reference class, the kind of situation it reasons from and how such situations have usually gone, before it says why this case should or should not follow. It then tells the single most likely course of events as a dated chain of causes, gives the rival scenarios their own probabilities, and makes sure the numbers add up. It lists the indicators that would show early which way things are going, and ends on the one open fact that would most change the probability.
Each piece also says what follows for the people who have to act: technology leaders, investors, operators and regulators.
How a piece is checked before publication
Research starts with a dossier of sourced facts, each recorded with a short quotation, its link and its date. The draft is written from the dossier alone. A separate fact-checker that did not write the draft then opens every linked source and tests every name, number, date and quotation against it. Anything wrong, overstated or unsupported is corrected or cut, and a final pass confirms the corrections. On the first three forecasts this process tested 202 claims and changed 44 of them. One correction moved a probability: the fact-checker found that the US energy regulator had rejected a PJM rule the draft treated as in force.
How forecasts are scored
Each forecast is stored with its probability, its settlement date and a resolution condition written so that a stranger could check it. When the date passes, two judges from different model families read the public evidence and settle it. The result is scored with the Brier score, the squared difference between the probability and the outcome, so a 90 percent forecast that fails costs far more than a 60 percent one. A forecast the evidence cannot settle after four checks is marked void in public, and the share of void forecasts is tracked so that voiding cannot be used to hide misses. Every result, including the misses, stays on the forecasts page.
Forecasts are not edited after publication to look better. When new evidence moves a probability, the piece gets a dated update that says what changed and why, and the score is still calculated from the original number. Each forecast also carries a register of dated secondary predictions, and a weekly look-back checks those that have fallen due.
The known weaknesses
A Brier score from a small number of forecasts is mostly noise; the forecasts page says so until 20 have been resolved. The fact-checker is itself an AI, so errors that both it and the writer share can survive. Some evidence sits behind paywalls or exists only in secondary reports of conference calls, and the pieces mark those claims as such. The standard is meant to make each of these weaknesses visible to the reader rather than hide it.