Debunkers Debunked: Who Fact-Checks the Fact-Checkers?

By Alex Berezow — November 4, 2019 @ American Council on Science and Health

I gave a lecture recently on debunking junk science for a class at the University of Washington called "Calling Bullsh**." As its not-so-subtle title suggests, the class teaches students how to recognize misinformation in our digital age.

One of the topics I touched on was the role that fact-checkers play in our online debates. Certainly, they can and do play a very constructive role. There are so many lies swirling around that professional fact-checking seems to be a necessary public good.

But, who fact-checks the fact-checkers? While that might seem like a silly question, the objectivity of fact-checking websites has already been called into doubt and for good reason. For example, an in-depth analysis by Matt Shapiro at the Paradox Project revealed that PolitiFact (the site that famously uses "pants on fire" as one of its ratings) is biased in its fact-checking.

The analysis makes several different arguments, but one of the more compelling ones is that the articles that debunk Republicans are longer than those that debunk Democrats. Why? Well, it comes down to a bit of chicanery:

"We’ve found that PolitiFact often rates statements that are largely true but come from a GOP sources [sic] as 'mostly false' by focusing on sentence alterations, simple mis-statements, fact-checking the wrong fact, and even taking a statement, rewording it, and fact-checking the re-worded statement instead of the original quoted statement. 

"Doing this takes time and many, many words."

Another popular website, Snopes (which got its start by debunking urban legends), bizarrely decided to debunk satire. Snopes appeared to be aiming the majority of its wrath toward one satire site in particular, the Babylon Bee, which happens to be published by conservative Christians.

Can Something Really Be "Mostly True" or "Half True"?

Another problem with fact-checking is the rating system. Fact-checkers like PolitiFact often conclude that statements are true, mostly true, half-true, mostly false, or false. But is such a classification even possible? In other words, can something really be "mostly true" or "half true"? The Babylon Bee captures this problem in a pretty funny headline:

If a politician said that the lost city of Atlantis has been discovered underneath Puget Sound, everybody would know he was lying. It's entirely absurd. But if a politician makes a false claim within a context that is mostly true, people are much more likely to believe it. In other words, the most effective lies are "mostly true."

Who Fact-Checks the Fact-Checkers?

Ultimately, fact-checking is a much more subjective enterprise than we would like it to be. Truth is real but sometimes difficult to ascertain, particularly when political ideologies and motivated reasoning are involved. To eschew these pitfalls, fact-checkers need to be keenly sensitive to such biases. Otherwise, fact-checkers will be seen as simply another manifestation of "fake news."