How the Liberal Media Myth is Created - Part 5
UPDATE 4/16/05: This is an updated version of my original post which is archived here. As I was doing a review of some other published literature on media bias on 4/16/05, I discovered that Groseclose-Milyo (G-M) had posted an updated version (HTML, PDF) of their original paper as of 2005-01-03. The revised version of their paper corrects some of lacunae in the original version; however, the most fundamental problems with the original paper remain in this new version. [NOTE: The fact that I missed the latest version in my original critique was purely an unintentional oversight. The updated G-M paper does not in any way invalidate my original critique (indeed, one of the fixes they made shows that one part of my critique was right on target). I have updated my critique here to refer to their revised paper.]
This is a continuation of a series on how the "liberal media" myth is created. Previous installments covered myth-creation using "tone" of media coverage (Part 1), "catch-phrases" like 'right-wing extremist' v. 'left-wing extremist' (Part 2), "newspaper headlines" (Part 3) and "topics" covered (Part 4). This part highlights an unusual, indirect approach that uses "think-tank" citations.
The focus of this post is a paper titled "A Measure of Media Bias" (HTML, PDF) by Tim Groseclose and Jeff Milyo. I found this paper via Language Log (there has been some back and forth at Language Log between critic Geoffrey Nunberg and the paper's authors), where it was also noted that:
Groseclose and Milyo's study has been approvingly cited by Bruce Bartlett in National Review, by Linda Seebach in the Rocky Mountain News, and by Harvard economist Robert J. Barro in Business Week, not to mention conservative bloggers like Instapundit, Andrew Sullivan, and Matt Drudge, among a number of others, who trumpet its "objectivity."
A single blog post, once again, is insufficient to provide a detailed critique of the paper. So, I'll refer readers who are more curious to my detailed critique over at ICM - Sec. 2.9. Here, I'll reproduce my summary (with links to details) showing why this paper's conclusions are wrong.
The Groseclose-Milyo (G-M) paper (HTML, PDF) attempts to assess media bias using an approach wherein adjusted ADA (Americans for Democratic Action) scores (0-to-100) are used to assess legislator ideology (archconservative-to-archliberal), and separately, the think-tank citations of the legislators are compared to the think-tank citations of the media outlet to then derive the media outlet's "bias". Based on their methodology (presented and discussed in this paper), they claim that:
Our results show a strong liberal bias.
I examined the paper from three perspectives:
1. Is the methodology used for assessing the ideology of think-tanks correct and reliable?
2. Is the methodology used for assessing the ideology of the media correct and reliable?
3. Is the definition of media bias used by the authors correct and reliable?
The answers to each of those questions is NO.
The methodology used by the authors for assessing think-tank ideology (i.e., based on the average adjusted ADA score of the legislators citing the think-tank) is deeply flawed because it omits public or private disagreements that legislators have with the same think-tank and it does not account for the fact that legislators may agree with a think-tank but not state it publicly for various reasons (e.g., they are unaware of the think-tank; they are aware of the think-tank but the latter may not be known well enough to cite, it may be a "controversial" think-tank, there may be no need to cite a think-tank, etc.). This can effectively skew their results in the wrong direction, to an unknown degree. For example, the fact that their methodology found the ACLU to be "conservative" was a result of the former flaw. To address this, they say:
The reason the ACLU has such a low score is that it opposed the McCain-Feingold Campaign Finance bill, and conservatives in Congress cited this often. In fact, slightly more than one-eight of all ACLU citations in Congress were due to one person alone, Mitch McConnell (R.-Kt.), perhaps the chief critic of McCain-Feingold. If we omit McConnell’s citations, the ACLU’s average score increases to 55.9. Because of this anomaly, in the Appendix we report the results when we repeat all of our analyses but omit the ACLU data.
Unfortunately, omitting McConnell's citations or the ACLU data point is the wrong approach to fix this problem. The way to fix this is by actually ADDING all those instances in which Republicans actually disagreed with ACLU, not incorrectly and artificially remove situations where *they agreed with ACLU* in order to get an average score that seems more in sync with a *separately established* reality. In other words, if we already knew ACLU is "liberal" and need to know that to "adjust the data", then what is the value or point of this study?
Additionally, a legislator may cite a think tank not because he or she mostly agrees with the think tank but because that think tank's view is closer to his or her view than any other think-tank the legislator is aware of or cares to cite. It is very unlikely that legislators who cite a think tank agree with everything the think tank says or stands for. For example, some legislators may cite it because their position is in agreement with, say, only one or two or three of the think tank's positions and they may cite it for that reason, repeatedly (like in the ACLU case). The bottom line is that their think-tank ideology ratings are unreliable and incorrect, as I show in detail in at ICM Sec. 2.9.
The methodology used by the authors for assessing media ideology is completely untenable. There are three principal reasons for this:
(a) The approach G-M use establishes media ideology indirectly, by using the media's think-tank citations and comparing those to think-tank citations by legislators in order to find the legislator whose citations are the closest match. Thus, if a legislator is liberal and the media's think-tank citations match that of the liberal legislator, they would declare the media to be liberal. Momentarily setting aside the fact that this definition of media bias is itself incorrect, their claim would make sense only if it can be independently proven that the think-tanks cited by the liberal legislator are actually liberal. Their study does not prove this at all, considering that their methodology to establish think-tank ideology is itself deficient. Thus, at a fundamental level, their entire conclusion on media bias breaks down. (NOTE: It is not at all implausible that left-leaning legislators may cite more centrist think-tanks in public than progressive/liberal ones, especially considering how the liberal advocacy groups and think-tanks are tarred negatively by the GOP in the illiberal conservative media).
(b) The use of weighted-average ADA scores (for the House and the Senate) is slightly more meaningful than the Median (which they used in the original version of their paper), but even this is completely deficient and incorrect because the ideological center is set not using an independent, objective measure of ideology but based on the (political) positions of the people in Congress at a given point in time. Thus, their model simultaneously assumes that ADA scores can provide an absolute picture of a legislator's ideology but that media and think-tank ideology should be determined not using the same absolute reference but a relative, moving reference that is highly dependent on who's the majority in Congress and how they think or vote. This is not an acceptable model, for, if the minority party becomes the majority party in the next election, the derived ideology of think-tanks or the media could change significantly even though their actual positions underwent ZERO change.
Put another way, if the Republican majority suddenly decides to become 100% conservative, guess what happens. The weighted-mean ADA score would drop, even if the Democrats in Congress DID NOT change at all, and even if the media outlets that are considered "liberal", by the G-M definition, remain STATIC (i.e., no change in their think-tank citation ratios and that of the corresponding "liberals" in Congress). In this case, even though the media's ideology has NOT changed at all, it's adjusted ADA score(s) will artificially look more liberal compared to the lower weighted-mean ADA score. (BONUS FOR LEFTIES: This is right in line with one of the long-time Republican strategies of declaring the media (and Democrats) to be too "liberal" by moving the country to the Right). This is not a partisan issue though. The opposite could occur when we are talking about media outlets that are considered "conservative" because they match the citations of conservative Republicans and if the Democrats decide to become 100% liberal.
(c) The final, and perhaps most serious, problem with their analysis is their attempt to derive a conclusion of media bias using this study - because their definition of media bias, is in itself, completely flawed. Their confident conclusion that they have demonstrated "liberal" media bias is wrong because the study does not examine whether the media's news reporting is accurate. Their assumption that "seldom do journalists make dishonest statements" is also fatally incorrect. The focus on think-tank citations completely ignores what the media communicates to viewers or readers when it is NOT citing think-tanks, which is a big chunk of the time. The irony of the authors' citing serial liar Brent Bozell's claim that there is "rarely a conscious attempt to distort the news" is incredibly ironic! Their claim that "the citations that they gather from experts are also very rarely dishonest or inaccurate" also suggests that they are very un-skeptical when it comes to absorbing news.
When controlled for other factors, the more fundamental determinant of bias in news reporting is accuracy -- not whom the news reports cite. To the extent that news reporting could become inaccurate by citing certain think-tanks over others, one may have a case that think-tank citations could influence the accuracy of the reports. But, G-M have fallen into the trap of assuming that the part is the whole. Think-tank citations are merely one part of the whole - which is the media's accuracy in news reporting.