In August of 2011, a post of mine addressed the policy of drug testing welfare recipients. At that time, much was made of the fact that Florida's mandatory drug testing policy produced just a 2% failure rate. From this statistic, Tampa Bay Online concluded:
The initiative may save the state a few dollars anyway, bearing out one of Gov. Rick Scott’s arguments for implementing it. But the low test fail-rate undercuts another of his arguments: that people on welfare are more likely to use drugs.
Irrespective of whether the policy of drug testing welfare recipients makes sense, we just cannot draw these kinds of conclusions from the data. Indeed, the data suffers from a problem known as "selection bias." Simply put, individuals most likely to fail a drug test are not going to take the test. We just can't generalize about the entire population of welfare recipients from this severely self-selected sample.
In the current version of this story, Utah has just published it's version of the same data. Upworthy has the results in a not-so-subtle pie chart. Here are the findings:
Yeah. Wow. Again, it's reasonable to conclude that this policy may be "unfair" or a "poor use of taxpayer dollars" given the low rate of positive results. But unfortunately the author of the post draws the same faulty conclusion as in the Florida case. For us to conclude that welfare recipients (the population of interest) have a lower incidence of drug use compared to the state, we would need a random sample where every individual in the population has an equal, non-zero chance of being selected. We don't have that here.