We are veering into the wonderful world of ACCURACY when we start talking about situations like this. It is clear that in both the VA case and the ConEd case that they took the number of complaints to be an accurate indicator of the actual number of errors.

It’s an unfortunate aspect of human nature that we will always look for what I call “the happy path” and will latch on to numbers which are supportive of that. In my old job I was more often asked to recheck figures that had a negative impact than ones that supported the “Happy Path Delusion”. Of course, there was that one time when the horrifying impact figure I presented actually was the Happy Path, but that’s another story.

A better view might have been that (applying Poisson distribution) more than one defect reported suggests that there are likely to be more in your data and you should really look a bit closer at the actual level of errors before breathing a sigh of relief.

“The intersting thing for me as an information quality practitioner is that the VA’s initial estimate was based on the number of people who had contacted the agency.”

Reminds me of an issue I read about that I highlighted in my Master’s thesis:

“During a blackout the week of July 17, 2006, ConEd estimated the number of customers without power at 2,500, which was reported to the media. Days later, estimates went up to 100,000 without power. When this larger number became public, customers were furious [i] that the company could not reliably estimate the effect of the blackout, and [ii] that it took so long to restore power. The original figure was based on the number of complaints the company had received, while the updated number was an estimate of the number of actual people without power. There was a belief that ConEd’s lax response was due to assuming the problem was not as widespread as it was. A number from one context was used inappropriately in a different context, leading to a lower level of action that resulted in angry customers.”

Cheers,
Eric