The rise of what is broadly called “big data” has given new impetus to those who believe that more and better information will solve our decision-making troubles.
The term big data is generally taken to refer to the digital exhaust from our time spent online: our searches, commercial transactions, social media interaction, movements tracked by GPS devices and so on. The sheer amount of data we generate has spawned hopes that we can take the guesswork and estimation out of decision-making. The number of data points will eliminate the need for statistical sampling. Big data will simply tell us what we need to know and what to do.
And it is cheap.
But big data can be misleading. Viewed out of context, or out of order, data can give us wrong answers, or be used to reinforce our prejudices. They can show correlations among events. But uncovering the causes behind the data are still susceptible to our biases.
Big data and the algorithms that seek to analyze it are certainly valuable, and they are only in the early stages of development. But we are far from the point where we can hand decision-making over to an algorithm. The human component — asking the right questions of the data and making sure they are used to serve our objectives and not the other way around, and deriving the right answers from them — remains crucial to good decisions.
We can easily see the difficulty in outsourcing decisions to the data sets in the struggle to craft climate change policies. No one can say we lack data on the issue. The Intergovernmental Panel on Climate Change has produced several reports, based on comprehensive information gathering and modelling produced by thousands of scientists. Yet all that data do not provide us with iron-clad projections about the precise impact of different variables on the future climate. They can’t, and they never will.
Because many policy decisions about climate lack a structure that would help policy-makers design strategies that are sensitive to the trade-offs between competing objectives, big data doesn’t help as much as it should to improve the decision process. The physical science does not resolve the social science challenges.
Is the boom in natural gas fracking a route to a lower-carbon energy substitute; does it have unintended consequences for the health and environment of local communities? Is reviving nuclear power a viable option, or are the perceived risks too high for a world spooked by Three Mile Island, Chernobyl and Fukushima? Is the massive capital expenditure required by adaptation technologies more acceptable to citizens than the considerable slowdown in economic growth from the shift away from carbon-based fuels?
The decisions that are at the ends of these questions require trade-offs that big data alone can’t make.
If used wisely, big data can help us make better decisions.
But access to big data does not remove the need for good judgment and human insight.
Read more of Joe Arvai’s blogs.