Here’s a book excerpt in the Wall Street Journal on improving your judgment of risk:
Most of us have to estimate probabilities every day. Whether as a trader betting on the price of a stock, a lawyer gauging a witness’s reliability or a doctor pondering the accuracy of a diagnosis, we spend much of our time—consciously or not—guessing about the future based on incomplete information. Unfortunately, decades of research indicate that humans are not very good at this. Most of us, for example, tend to vastly overestimate our chances of winning the lottery, while similarly underestimating the chances that we will get divorced.
Psychologists have tended to assume that such biases are universal and virtually impossible to avoid. But certain groups of people—such as meteorologists and professional gamblers—have managed to overcome these biases and are thus able to estimate probabilities much more accurately than the rest of us. Are they doing something the rest of us can learn? Can we improve our risk intelligence?
Sarah Lichtenstein, an expert in the field of decision science, points to several characteristics of groups that exhibit high intelligence with respect to risk. First, they tend to be comfortable assigning numerical probabilities to possible outcomes. Starting in 1965, for instance, U.S. National Weather Service forecasters have been required to say not just whether or not it will rain the next day, but how likely they think it is in percentage terms. Sure enough, when researchers measured the risk intelligence of American forecasters a decade later, they found that it ranked among the highest ever recorded, according to a study in the Journal of the Royal Statistical Society.
The excerpt says that you can improve your risk analysis abilities by getting immediate feedback. However, if you’re trying to answer the risk of deploying undertested software with the potential for hidden defects or if you’re estimating the chances of a discovered error occurring in the wild, that feedback might not be immediately available if the circumstances don’t occur until six months after the software is in use.
At any rate, it’s an article worth reviewing and maybe it’s worth getting the whole book Risk Intelligence: How to Live with Uncertainty.