Uncontrolled: The Surprising Payoff of Trial and Error for Business, Politics and Society by Jim Manzi is an excellent book that looks at the use of randomized field trials (RFTs) for public policy, business and society. The book advocates that RFTs be used as much as possible.
The book is divided into three parts, in the first Manzi looks at Science and how it changed to use empirical validation, in the second Social Science is examined and finally in the third Political Action is where Manzi makes recommendations.
In the first section on Science Manzi looks at the origin of the scientific method and implicit and explicit knowledge. Implicit knowledge is defined by Manzi as the knowledge within a system of what works reasonably well even if it is not conscious. He uses genetic algorithms as an example of this kind of knowledge being present in a system. Manzi goes on to suggest that Scientific advancement is similar to evolution in that theories that can be tested and that are proven right survive. He provides the failure of scientific advancement without empirical validation as being important. Finally he goes on to describe RFTs as they are used in science, particularly in medical trials.
In the second part of the book Manzi talks about his experience as a consultant and how consultants constructed models that were not greatly useful but would up looking at RFTs that did provide utility. He also talks about the problems with such trials and how they need a lot of good data and are sometimes not as good as the wisdom of experts. He talks about how RFTs have been made to work in business and provides the example of the Capital One credit card company in the US.
Then he looks at how these trials can be used with crime, welfare, education, economics and political science. He also shows that some of the scientific studies that are widely quoted, such as the study that showed that an over abundance of choice reduced consumption were flawed. Manzi emphasizes that iterative trials should be used to avoid such problems.
The final part of the book on Political Action is not particularly strong. Manzi does provide an interesting look at what he terms the paradox of Liberty and ponders how far liberty should be taken. Finally he provides some recommendations that are to use RFTs more, which is fairly sensible but he also provides some far weaker suggestions of his own including the well worn suggestion to spend more or science and technology which appears to come more from Manzi’s background as a math major at MIT than from the rest of the book.
The strength of the book in the first two sections is terrific though. It is a terrific, fairly short and well written public policy book. Anyone who works in public policy would be well served by reading it. For anyone interested in science and social science it’s also very much worth reading.