SuperForecasting

Review

This book surprised me. I expected a probability primer and got something more useful: a structured argument that forecasting skill is trainable, and that the gap between good forecasters and poor ones is less about raw intelligence than about how they approach uncertainty. Tetlock grounds everything in years of actual forecasting tournament data, which makes it unusually rigorous by business book standards. The chapter on team dynamics is where it pays off most for product work. The research on how groups fail at prediction (groupthink, homogeneous perspectives, lack of independent judgment) maps almost directly onto why product teams miss hard calls. Practical, evidence-based, and dense enough to reward close reading.

Key Takeaways

The parts worth keeping

Calibration and measurement

• Forecasting has real limits, but those limits are not a reason to stop. The skill is learnable, and deliberate practice is the strongest predictor of improvement.
• To meaningfully compare forecasters, you need precise terms, stated timelines, numerical probabilities, and volume. Vague predictions cannot be assessed or learned from.
• The Brier score measures forecast quality on two dimensions: calibration (how accurate your overall predictions are) and resolution (how decisive and specific they are). A perfect score is 0. Random guessing averages 0.5. Predicting 100% certainty on something that doesn't happen scores 2.0. Aim for both accuracy and specificity, not just one of the two.

Thinking frameworks

• System 1 is designed for speed, not accuracy. It jumps to conclusions from thin evidence and is insensitive to evidence quality. Slow down and engage System 2 when a forecast actually matters.
• Foxes vs hedgehogs: hedgehogs hold firm beliefs and filter new information to confirm them. Foxes are versatile, willing to update, and pull from multiple sources. In forecasting tournaments, foxes consistently outperform on accuracy, calibration, and resolution.
• Fermi decomposition: break a hard question into smaller knowable components. This exposes your assumptions, brings your reasoning into the open where it can be inspected, and produces more accurate estimates than treating the question as a single opaque thing.
• Inside view vs outside view: start every forecast with the base rate (how often does something like this happen in situations like this?). Then layer in the specifics of the case at hand. The base rate guards against over-indexing on what makes this case feel unique.
• Bayesian updating: your new belief should be your prior belief multiplied by the diagnostic value of new information. Update in proportion to evidence weight, not in large lurching swings. Beliefs are hypotheses to be tested, not positions to be defended.

The superforecaster method

• A common approach that emerged among top forecasters:
  • Unpack the question into components.
  • Distinguish between the known and the unknown; leave no assumption unscrutinized.
  • Adopt the outside view first: treat this as a special case of a wider class of events.
  • Then adopt the inside view: what makes this case genuinely different?
  • Explore similarities and differences with other forecasters' views.
  • Synthesize all perspectives into a single probability estimate.
  • Express that estimate on a fine-grained scale.
  • Update regularly as new information arrives.
• Superforecasters update more often than average forecasters, but each update is smaller (moving from 38% to 41.5%, for example). Train your brain to think in small units of doubt.
• Precision matters. Top forecasters debate the difference between 3% and 4%, or 1% and 0.5%. Granularity in probability expression is a predictor of accuracy.
• When a question carries irreducible uncertainty, stay cautious. Keep estimates in the 35%-65% range and move out tentatively.

Traits of superforecasters

• Superforecasters share a profile: cautious, humble, non-deterministic, actively open-minded, intellectually curious, introspective, and self-critical. They are not wedded to positions, they are capable of stepping back, and they think in small units of doubt.
• Growth mindset matters. Treat forecasting ability as a product of effort, not fixed talent. Failure is data. Postmortems with teammates deserve the same effort as the original forecast.
• What makes superforecasters good is not some native gift. It is the work: research, careful thought, self-criticism, gathering of diverse perspectives, granular probability judgments, and relentless updating.

Team forecasting

• Groupthink: cohesive groups unconsciously develop shared illusions that interfere with critical thinking. Groups that get along too well stop questioning assumptions and stop confronting uncomfortable facts.
• Aggregation only works when people form judgments independently. Shared reasoning before independent judgment anchors everyone to the first voice in the room.
• Perspective diversity beats raw ability. The more varied the team, the more likely that some members hold relevant information others don't have.
• Do team pre-mortems: assume the forecast or plan has already failed and explain why. This gives people cover to express doubts before the group commits.
• Mission command: tell people what needs to be achieved, not how to achieve it. Decentralized decision-making allows for faster adaptation when conditions change on the ground.

Ten Commandments for Superforecasters

1. Triage. Focus on questions where work can pay off. Don't waste time either on easy "clock-like" questions (where simple rules of thumb can get you close) or on impenetrable "cloud-like" questions (where fancy models won't help). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most.
2. Break seemingly intractable problems into tractable sub-problems. Channel the playful but disciplined spirit of Enrico Fermi. Decompose the problem into its knowable and unknowable parts. Flush ignorance into the open. Expose and examine your assumptions. Dare to be wrong by making your best guesses. Better to discover errors quickly than to hide them behind vague verbiage.
3. Strike the right balance between inside and outside views. Nothing is 100% unique. Look for comparison classes even for seemingly unique events. Ask: How often do things of this sort happen in situations of this sort?
4. Strike the right balance between under- and overreacting to new evidence. Belief updating pays off in the long term. Skilful updating requires spotting non-obvious lead indicators about what would have to happen before X could.
5. Look for the clashing causal forces at work in each problem. Acknowledge counter arguments. List in advance the signs that would nudge you toward the other view. Synthesis is an art that requires reconciling irreducibly subjective judgments. Create a nuanced view.
6. Strive to distinguish as many degrees of doubt as the problem permits but no more. Nuance matters. The more degrees of uncertainty you can distinguish, the better a forecaster you are likely to be. In poker you need to know a 55/45 from 45/55.
7. Strike the right balance between under- and overconfidence, between prudence and decisiveness. Long-term accuracy requires getting good scores on both calibration and resolution. Know your track record, and find creative ways to tamp down both types of forecasting errors (misses and false alarms).
8. Look for the errors behind your mistakes but beware of rearview mirror hindsight biases. Don't try to justify or excuse your failures. Own them! Conduct unflinching postmortems. Ask: Where exactly did I go wrong? Don't forget to do postmortems on your successes too.
9. Bring out the best in others and let others bring out the best in you. Master perspective taking (understanding the arguments of the other side), precision questioning (helping others to clarify their arguments so they are not misunderstood), and constructive confrontation (learning to disagree without being disagreeable).
10. Master the error-balancing bicycle. Implementing each commandment requires balancing opposing errors. Learning requires doing, with good feedback that leaves no ambiguity about whether you are succeeding.

Don't treat commandments as commandments. Guidelines are the best we can do in a world where nothing is certain or exactly repeatable. Superforecasting requires constant mindfulness, even when dutifully trying to follow these commandments.