Expert Political Judgement (2005) by Phillip Tetlock is a classic book that anyone with an interest in current affair should read. It is heavy going but the book is well worth the effort. Tetlock himself is a psychologist who was curious as to how accurate people are and how they perceive their own views. The New Yorker has a highly positive review.
Tetlock made a long term survey of experts and their predictions and then compared their predictions to what happened. His survey ran for about 20 years and ended in 2003. Overal what he found is that expert prediction is not good. Dart board ands heuristic algorithms would have fared little worse than experts. Experts also did little better in their own areas than knowledgable dilettantes.
Tetlock did find, however, that there where two broad categories into which experts could be placed. They were either ‘hedgehogs’ who ascribed strongly to a particular ideology and saw things through that lens or foxes who had no coherent overall pattern of prediction but who gathered information from diverse sources to make predictions. The foxes substantially outperformed hedgehogs. The foxes, however, were often not as famous as hedgehogs or as sought after for commentary because of the way their predictions were hedged.
Tetlock goes on to look at criticism of his work, from the subjects of the survey and others. Many of the subjects thought the time period, covering the tumultuous end of Communism, was particularly hard to predict but Tetlock points out that most other 20 years spans also had a raft of ‘events’ occurring.
Foxes did, however, have a weakness in that if a range of scenarios were presented in detail their predictive skill lessened greatly. Hedgehogs were less liable to this tendency. Hedgehogs appear to have instead constructed their own scenarios that made their initial predictions worse.
Tetlock classifies how people see his work in technical language that is dense and hard to read but that is justified. He talks of ‘radical skeptics’ who suggest that such prediction is always doomed to fail. He suggests that their views, while not completely outlandish, are not quite correct.
The book does, however, provide a very welcome reason to ignore much of the day to blather in newspapers and no on blogs on the web on the grounds that Tetlock has firmly shown that political expertise is fairly weak predictively.
The book is an example of very solid scholarly work written up well in order to provide a very readable popular work of great value. It’s absolutely superb and hard to recommend highly enough.