AI Visibility Forecasting
Scoring a forecast after the fact
The only mechanism by which forecasting ever improves, and the part almost nobody publishes.
Why unscored forecasts are entertainment
A forecast that is never checked cannot be wrong, so it cannot teach anything, and the forecaster has no incentive to calibrate. Nearly all public technology forecasting works this way: confident, memorable, and never revisited.
What scoring requires
A dated statement. Made before the period it describes, not summarised afterwards.
A stated confidence. Otherwise there is nothing to calibrate.
A falsification condition. Written at the time, saying what observation would show the forecast wrong. Without it, any outcome can be reinterpreted as consistent.
A resolution date. A forecast with no horizon can always be described as not yet resolved.
What a good record looks like
Not a high hit rate. A well-calibrated one: roughly the proportion of high-confidence forecasts should come true that the label implies, and low-confidence ones should be wrong reasonably often. A forecaster whose low-confidence calls are almost always right was underlabelling, which is as much an error as overclaiming.
The uncomfortable implication: a forecasting practice with no published misses is not accurate. It is either not publishing them or not making falsifiable statements.
What we do with a miss
Record it against the original statement, and separate two different failures: the observation was wrong, or the inference from it was wrong. These have different fixes. A bad observation means the measurement needs work. A bad inference means the reasoning does, and the reasoning is where forecasts usually fail.
Why this constrains what we forecast
Committing to score everything makes it expensive to publish confident statements about things that cannot be checked. That constraint is deliberate. It is the main reason AI Visibility Forecasting here skews toward directional statements about aggregate conditions rather than predictions about specific businesses.