Pew Study's Lack Of Comparison

While I was on vacation a couple of weeks ago, the Pew Project for Excellence in Journalism released a groundbreaking new study that attempted to address the most pervasive criticism of nonprofit news organizations -- that their journalism is biased by their business model.

In "Non-Profit News: Assessing a New Landscape in Journalism," Pew asks the right questions -- "Are these sites delivering, as they generally purport to be, independent and disinterested news reporting? Or are some of them more political and ideological in their reporting?" -- and it succeeds in providing us with the first methodical assessment of nonprofits that have launched since 2005. Among other things, it finds that transparency of mission, size of staff and multiplicity of funders are associated with balanced reporting.

But where the study falls short is in benchmarking its findings. It looks at a total of 46 sites that purport to produce objective state and/or national news, and it includes seven sites that operate as for-profit business. It then applies a content analysis to determine which harbor some kind of ideological bias, and it concludes that 44 percent of the sites in its universe are "ideological."

That conclusion led to this unfortunate headline in the Chronicle of Philanthropy: "Ideology Drives Many Nonprofit News Sites, Study Says." What constitutes "many"? We don't know because Pew didn't compare nonprofits to a broader universe of state and national media.

Take a look around, and it's not hard to discern the ideological biases that guide many of our media outlets. Plenty of for-profit outlets skew their reporting to capture the loyalties of a particular slice of the market and deliver it to advertisers. Fox News is the obvious example of how to make a profit by introducing ideological bias into news reporting. But it happens in less obvious ways as well. Online newsletters often reflect the biases of the industries and professions they serve, for example.

I don't dispute the method Pew used to assess ideology, but I think the analysis would have benefited from having a more rigorous comparison with a broader control group. As a result, we are left to ask: Ideological? Comapred to what?

Here's one piece of the analysis that I thought worth cheering. Pew found that news organizations with high levels of transparency and diversity of revenue sources were most balanced in their reporting. From the report:

Sites that offered a mixed or balanced political perspective, on the other hand, tended to have multiple funders, more revenue streams, more transparency and more content with a deeper bench of reporters. The six most transparent sites studied, for instance, were among the most balanced in the news they produced.

That conclusion confirms what a lot of people in the nonprofit sector have assumed for years, and it lends additional credibility to many of the organizations that have worked hardest to play by the rules of good journalism.