Email
Manufacturing 'Crisis': How Polling on the Border Exaggerates Extreme Opinions
| From | FAIR <[email protected]> |
| Subject | Manufacturing 'Crisis': How Polling on the Border Exaggerates Extreme Opinions |
| Date | August 25, 2022 7:14 PM |
Links have been removed from this email. Learn more in the FAQ.
Links have been removed from this email. Learn more in the FAQ.
[link removed]
FAIR
View article on FAIR's website ([link removed])
Manufacturing 'Crisis': How Polling on the Border Exaggerates Extreme Opinions David W. Moore ([link removed])
Last week, two media polls announced results that seemed almost willful efforts to portray the public as extremists.
The Economist/YouGov poll (8/18/22 ([link removed]) ) announced that “Most Americans see the US/Mexico border situation as a crisis,” with 59% of Americans accepting that characterization, with just 22% who do not. The same day, NPR/Ipsos (8/18/22 ([link removed]) ) reported that “a majority of Americans see an ‘invasion’ at the southern border.”
Both of these polls reinforce the larger narrative ([link removed]) being promoted by Fox News and Republicans that the increased number of migrants being stopped at the border represents a serious threat to the United States. And Biden and the Democrats are to blame. Yet the poll results are based on faulty polling questions that seem designed to produce extreme results.
** Unbalanced questions
------------------------------------------------------------
Economist: Most Americans see the U.S.-Mexico border situation as a crisis
Economist (8/18/22 ([link removed]) )
The question posed in the Economist/YouGov poll ([link removed]) was simple: “Do you think the current situation at the US/Mexico border is a crisis?”
For any experienced pollster, that wording screams of malfeasance. First of all, it is “unbalanced” because it provides no negative counter.
In his 1951 book, The Art of Asking Questions (Princeton University Press), Stanley Payne showed how much an unbalanced question can distort results. In one poll, half the sample of respondents were asked whether in general, manufacturers could avoid laying off workers during slack periods. The other half of respondents were read the same question, but with the added line, "or do you think the layoffs are unavoidable?"
With the unbalanced question, respondents agreed by a margin of almost three-to-one that companies could avoid the layoffs (63% said they could, 22% disagreed). With the balanced question, the margins changed in the opposition direction: 35% said the companies could avoid layoffs, 41% said the layoffs were unavoidable.
That represented a 47-point swing in opinion (41 points in favor to 6 points against), simply by changing the question wording to make it balanced.
And pollsters have known about the problem of unbalanced questions for the past seven decades.
** Ensuring an extreme response
------------------------------------------------------------
Economist: More than half of Americans say the current situation at the U.S.-Mexico border is a crisis
The Economist (8/18/22 ([link removed]) ) showed that you can get most people to describe the border situation as a "crisis"—if you don't offer them any other way to describe it.
But the Economist/YouGov question is also so vague, we really don’t know what people might mean when they agree there’s a crisis. As opposed to what? Do the reporters want to know if people think there is a crisis as opposed to just “serious problems”? If so, they need to provide both options and ask which one the respondents think is more applicable. But they didn’t do that.
They could have asked if people thought there were “serious problems” at the southern border. But how exciting would the resulting headline be? “A majority of Americans think there are serious problems at the US/Mexican border!!”
No. Not worth a screaming headline. Better to ask if there’s a “crisis.” By not providing a counter factual option (such as, “there are problems but no crisis”), the pollsters all but ensured a majority choosing the extreme response.
When I was the managing editor of the Gallup Poll in the early 1990s, we wanted to know how seriously the public viewed the healthcare issue in the country. In 1994, we asked three questions to address that concern, with the results below:
* In your opinion, is there a crisis in healthcare in this country today, or not?
It was an ostensibly balanced question, because it added the “or not” phrase. But it, like the Economist/YouGov question, provided no context. The results showed 84% saying yes to the “crisis.”
* Which of these statements do you agree with more: 1) the country has a healthcare crisis, or 2) the country has healthcare problems but no health care crisis?
Here 53% chose “crisis.”
* Which of these statements do you think best describes the US healthcare system today: 1) the healthcare system is in a state of crisis, 2) it has major problems, 3) it has minor problems, or 4) it does not have any problems?
Just 17% chose “crisis.”
The number of Americans believing the health system was in a crisis went from 84% to 53% to just 17% as respondents were provided with alternative ways of looking at the issue.
The people at the Economist/YouGov poll aren’t necessarily aware of the Gallup results just cited, but they should be aware more generally of the problems with unbalanced and vague questions as outlined decades ago, and certainly included these days in any basic lessons on poll question design.
What is the best way to measure the public’s opinion about the problems at the southern border? The four-part Gallup question about healthcare is a good model for the border issue, because it provides for a fuller understanding of the varieties of views that Americans might hold. Getting at the nuances of public opinion may not provide dramatic headlines, but it’s a more honest way of reporting what Americans are actually thinking.
** NPR manipulation
------------------------------------------------------------
NPR: A majority of Americans see an 'invasion' at the southern border, NPR poll finds
NPR (8/18/22 ([link removed]) )
The NPR/Ipsos poll ([link removed]) is perhaps worse than the one just analyzed. It reeks of manipulation. The question is written in a true/false format, which itself is problematic. And it clearly favors the “true” option.
The poll question: “To what extent, if any, do you believe the following are true? — The US is experiencing an invasion at the southern border.” The answers provided were: “Completely true, Somewhat true, Completely false, Don’t know.”
True/false questions are inherently biased in favor of true, because of a phenomenon called “response acquiescence.” As described in this carefully researched Wikipedia article ([link removed]) :
Acquiescence bias, also known as agreement bias, is a category of response bias common to survey research in which respondents have a tendency to select a positive response option or indicate a positive connotation disproportionately more frequently. Respondents do so without considering the content of the question or their 'true' preference. Acquiescence is sometimes referred to as "yea-saying" and is the tendency of a respondent to agree with a statement when in doubt....
Acquiescence bias can introduce systematic errors that affect the validity of research by confounding attitudes and behaviors with the general tendency to agree, which can result in misguided inference. Research suggests that the proportion of respondents who carry out this behavior is between 10% and 20%.
In addition, the NPR/Ipsos question is unbalanced, providing two responses for “true” (completely, and somewhat), while only one response for “false” (completely). It’s not at all clear what “somewhat true” means, but if that option is given, a balanced approach would also offer the option of “somewhat false.” But for the “false” option, the question included only the “completely” category.
And then the media pollsters combined ([link removed]) the “completely true” percentage (28%) with the “somewhat true” responses (25%) to produce a “majority” of Americans saying there is an “invasion.”
** Designed to support a view
------------------------------------------------------------
NPR: Majority of Americans say there is an “invasion” at the southern border
NPR (8/18/22 ([link removed]) ) counted people who said it was "somewhat true" to say that there was an "invasion" at the border as agreeing that there was an invasion—although their answer implies that it's also somewhat false to call it an invasion.
Why did the pollsters ask the question in this biased way? It appears to me as though they wanted to show that “extreme rhetoric” about the border issue has become widespread.
According to the NPR report (italics added):
Republican leaders are increasingly framing the situation as an "invasion. ([link removed]) " Immigrant advocates say the word has a long history in white nationalist circles, and warn that such extreme rhetoric could provoke more violence against immigrants.
Still, the polling shows that the word "invasion" has been embraced by a wide range of Americans to describe what's happening at the border.
Note I said the pollsters wanted to “show” that extreme rhetoric has become widespread, not that they wanted to find out if it was actually the case. They seem to have had their minds made up before the poll was conducted, and designed a questionnaire that would support their view.
Had they wanted to investigate whether most people embraced “invasion” as a way to describe what’s happening at the border, the pollsters could have avoided the simplistic and biased true/false format, and the unbalanced question construction that favored “true,” and instead asked a more objective question.
One such series of question could have been worded this way:
How much would you say you know about what’s happening at the US/Mexico border these days: a great deal, a moderate amount, not much or nothing at all?
From what you’ve read or heard, which do you think better describes what’s happening at the southern border: 1) There is an invasion of immigrants into this country; or 2) There is not an invasion of immigrants into this country, but rather an unusually large number of immigrants seeking legal asylum – or 3) are you unsure?
The first question would allow an analysis to see whether people’s perceived knowledge of what’s happening is correlated with people’s use of “invasion” to describe the border events. The “unsure” option is to let respondents know that it’s okay to admit they don’t know.
In 1994, when Gallup’s polling partners were informed that the percentage of people saying the healthcare system was in “crisis” was only 17% (with the new wording format), the on-air pundit at CNN said he didn’t want to use that question. He wanted a simple result—crisis or no crisis. It was easier, he said, to describe on air. It mattered not that the question forced respondents into extremist positions, even though few people felt that way.
That seems to be the problem with the two polls analyzed here. Dramatic results showing an extremist public are far more “newsworthy” than what the public is really thinking.
Read more ([link removed])
Share this post: <a rel="nofollow" target="_blank" href="[link removed]" title="Twitter"><img border="0" height="15" width="15" src="[link removed]" title="Twitter" alt="Twitter" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Facebook"><img border="0" height="15" width="15" src="[link removed]" title="Facebook" alt="Facebook" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Pinterest"><img border="0" height="15" width="15" src="[link removed]" title="Pinterest" alt="Pinterest" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="LinkedIn"><img border="0" height="15" width="15" src="[link removed]" title="LinkedIn" alt="LinkedIn" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Google Plus"><img border="0" height="15" width="15" src="[link removed]" title="Google Plus" alt="Google Plus" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Instapaper"><img border="0" height="15" width="15" src="[link removed]" title="Instapaper" alt="Instapaper" class="mc-share"></a>
© 2021 Fairness & Accuracy in Reporting. All rights reserved.
You are receiving this email because you signed up for email alerts from
Fairness & Accuracy in Reporting
Our mailing address is:
FAIRNESS & ACCURACY IN REPORTING
124 W. 30th Street, Suite 201
New York, NY 10001
FAIR's Website ([link removed])
FAIR counts on your support to do this work — please donate today ([link removed]) .
Follow us on Twitter ([link removed]) | Friend us on Facebook ([link removed])
change your preferences ([link removed])
Email Marketing Powered by Mailchimp
[link removed]
unsubscribe ([link removed]) .
FAIR
View article on FAIR's website ([link removed])
Manufacturing 'Crisis': How Polling on the Border Exaggerates Extreme Opinions David W. Moore ([link removed])
Last week, two media polls announced results that seemed almost willful efforts to portray the public as extremists.
The Economist/YouGov poll (8/18/22 ([link removed]) ) announced that “Most Americans see the US/Mexico border situation as a crisis,” with 59% of Americans accepting that characterization, with just 22% who do not. The same day, NPR/Ipsos (8/18/22 ([link removed]) ) reported that “a majority of Americans see an ‘invasion’ at the southern border.”
Both of these polls reinforce the larger narrative ([link removed]) being promoted by Fox News and Republicans that the increased number of migrants being stopped at the border represents a serious threat to the United States. And Biden and the Democrats are to blame. Yet the poll results are based on faulty polling questions that seem designed to produce extreme results.
** Unbalanced questions
------------------------------------------------------------
Economist: Most Americans see the U.S.-Mexico border situation as a crisis
Economist (8/18/22 ([link removed]) )
The question posed in the Economist/YouGov poll ([link removed]) was simple: “Do you think the current situation at the US/Mexico border is a crisis?”
For any experienced pollster, that wording screams of malfeasance. First of all, it is “unbalanced” because it provides no negative counter.
In his 1951 book, The Art of Asking Questions (Princeton University Press), Stanley Payne showed how much an unbalanced question can distort results. In one poll, half the sample of respondents were asked whether in general, manufacturers could avoid laying off workers during slack periods. The other half of respondents were read the same question, but with the added line, "or do you think the layoffs are unavoidable?"
With the unbalanced question, respondents agreed by a margin of almost three-to-one that companies could avoid the layoffs (63% said they could, 22% disagreed). With the balanced question, the margins changed in the opposition direction: 35% said the companies could avoid layoffs, 41% said the layoffs were unavoidable.
That represented a 47-point swing in opinion (41 points in favor to 6 points against), simply by changing the question wording to make it balanced.
And pollsters have known about the problem of unbalanced questions for the past seven decades.
** Ensuring an extreme response
------------------------------------------------------------
Economist: More than half of Americans say the current situation at the U.S.-Mexico border is a crisis
The Economist (8/18/22 ([link removed]) ) showed that you can get most people to describe the border situation as a "crisis"—if you don't offer them any other way to describe it.
But the Economist/YouGov question is also so vague, we really don’t know what people might mean when they agree there’s a crisis. As opposed to what? Do the reporters want to know if people think there is a crisis as opposed to just “serious problems”? If so, they need to provide both options and ask which one the respondents think is more applicable. But they didn’t do that.
They could have asked if people thought there were “serious problems” at the southern border. But how exciting would the resulting headline be? “A majority of Americans think there are serious problems at the US/Mexican border!!”
No. Not worth a screaming headline. Better to ask if there’s a “crisis.” By not providing a counter factual option (such as, “there are problems but no crisis”), the pollsters all but ensured a majority choosing the extreme response.
When I was the managing editor of the Gallup Poll in the early 1990s, we wanted to know how seriously the public viewed the healthcare issue in the country. In 1994, we asked three questions to address that concern, with the results below:
* In your opinion, is there a crisis in healthcare in this country today, or not?
It was an ostensibly balanced question, because it added the “or not” phrase. But it, like the Economist/YouGov question, provided no context. The results showed 84% saying yes to the “crisis.”
* Which of these statements do you agree with more: 1) the country has a healthcare crisis, or 2) the country has healthcare problems but no health care crisis?
Here 53% chose “crisis.”
* Which of these statements do you think best describes the US healthcare system today: 1) the healthcare system is in a state of crisis, 2) it has major problems, 3) it has minor problems, or 4) it does not have any problems?
Just 17% chose “crisis.”
The number of Americans believing the health system was in a crisis went from 84% to 53% to just 17% as respondents were provided with alternative ways of looking at the issue.
The people at the Economist/YouGov poll aren’t necessarily aware of the Gallup results just cited, but they should be aware more generally of the problems with unbalanced and vague questions as outlined decades ago, and certainly included these days in any basic lessons on poll question design.
What is the best way to measure the public’s opinion about the problems at the southern border? The four-part Gallup question about healthcare is a good model for the border issue, because it provides for a fuller understanding of the varieties of views that Americans might hold. Getting at the nuances of public opinion may not provide dramatic headlines, but it’s a more honest way of reporting what Americans are actually thinking.
** NPR manipulation
------------------------------------------------------------
NPR: A majority of Americans see an 'invasion' at the southern border, NPR poll finds
NPR (8/18/22 ([link removed]) )
The NPR/Ipsos poll ([link removed]) is perhaps worse than the one just analyzed. It reeks of manipulation. The question is written in a true/false format, which itself is problematic. And it clearly favors the “true” option.
The poll question: “To what extent, if any, do you believe the following are true? — The US is experiencing an invasion at the southern border.” The answers provided were: “Completely true, Somewhat true, Completely false, Don’t know.”
True/false questions are inherently biased in favor of true, because of a phenomenon called “response acquiescence.” As described in this carefully researched Wikipedia article ([link removed]) :
Acquiescence bias, also known as agreement bias, is a category of response bias common to survey research in which respondents have a tendency to select a positive response option or indicate a positive connotation disproportionately more frequently. Respondents do so without considering the content of the question or their 'true' preference. Acquiescence is sometimes referred to as "yea-saying" and is the tendency of a respondent to agree with a statement when in doubt....
Acquiescence bias can introduce systematic errors that affect the validity of research by confounding attitudes and behaviors with the general tendency to agree, which can result in misguided inference. Research suggests that the proportion of respondents who carry out this behavior is between 10% and 20%.
In addition, the NPR/Ipsos question is unbalanced, providing two responses for “true” (completely, and somewhat), while only one response for “false” (completely). It’s not at all clear what “somewhat true” means, but if that option is given, a balanced approach would also offer the option of “somewhat false.” But for the “false” option, the question included only the “completely” category.
And then the media pollsters combined ([link removed]) the “completely true” percentage (28%) with the “somewhat true” responses (25%) to produce a “majority” of Americans saying there is an “invasion.”
** Designed to support a view
------------------------------------------------------------
NPR: Majority of Americans say there is an “invasion” at the southern border
NPR (8/18/22 ([link removed]) ) counted people who said it was "somewhat true" to say that there was an "invasion" at the border as agreeing that there was an invasion—although their answer implies that it's also somewhat false to call it an invasion.
Why did the pollsters ask the question in this biased way? It appears to me as though they wanted to show that “extreme rhetoric” about the border issue has become widespread.
According to the NPR report (italics added):
Republican leaders are increasingly framing the situation as an "invasion. ([link removed]) " Immigrant advocates say the word has a long history in white nationalist circles, and warn that such extreme rhetoric could provoke more violence against immigrants.
Still, the polling shows that the word "invasion" has been embraced by a wide range of Americans to describe what's happening at the border.
Note I said the pollsters wanted to “show” that extreme rhetoric has become widespread, not that they wanted to find out if it was actually the case. They seem to have had their minds made up before the poll was conducted, and designed a questionnaire that would support their view.
Had they wanted to investigate whether most people embraced “invasion” as a way to describe what’s happening at the border, the pollsters could have avoided the simplistic and biased true/false format, and the unbalanced question construction that favored “true,” and instead asked a more objective question.
One such series of question could have been worded this way:
How much would you say you know about what’s happening at the US/Mexico border these days: a great deal, a moderate amount, not much or nothing at all?
From what you’ve read or heard, which do you think better describes what’s happening at the southern border: 1) There is an invasion of immigrants into this country; or 2) There is not an invasion of immigrants into this country, but rather an unusually large number of immigrants seeking legal asylum – or 3) are you unsure?
The first question would allow an analysis to see whether people’s perceived knowledge of what’s happening is correlated with people’s use of “invasion” to describe the border events. The “unsure” option is to let respondents know that it’s okay to admit they don’t know.
In 1994, when Gallup’s polling partners were informed that the percentage of people saying the healthcare system was in “crisis” was only 17% (with the new wording format), the on-air pundit at CNN said he didn’t want to use that question. He wanted a simple result—crisis or no crisis. It was easier, he said, to describe on air. It mattered not that the question forced respondents into extremist positions, even though few people felt that way.
That seems to be the problem with the two polls analyzed here. Dramatic results showing an extremist public are far more “newsworthy” than what the public is really thinking.
Read more ([link removed])
Share this post: <a rel="nofollow" target="_blank" href="[link removed]" title="Twitter"><img border="0" height="15" width="15" src="[link removed]" title="Twitter" alt="Twitter" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Facebook"><img border="0" height="15" width="15" src="[link removed]" title="Facebook" alt="Facebook" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Pinterest"><img border="0" height="15" width="15" src="[link removed]" title="Pinterest" alt="Pinterest" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="LinkedIn"><img border="0" height="15" width="15" src="[link removed]" title="LinkedIn" alt="LinkedIn" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Google Plus"><img border="0" height="15" width="15" src="[link removed]" title="Google Plus" alt="Google Plus" class="mc-share"></a>
<a rel="nofollow" target="_blank" href="[link removed]" title="Instapaper"><img border="0" height="15" width="15" src="[link removed]" title="Instapaper" alt="Instapaper" class="mc-share"></a>
© 2021 Fairness & Accuracy in Reporting. All rights reserved.
You are receiving this email because you signed up for email alerts from
Fairness & Accuracy in Reporting
Our mailing address is:
FAIRNESS & ACCURACY IN REPORTING
124 W. 30th Street, Suite 201
New York, NY 10001
FAIR's Website ([link removed])
FAIR counts on your support to do this work — please donate today ([link removed]) .
Follow us on Twitter ([link removed]) | Friend us on Facebook ([link removed])
change your preferences ([link removed])
Email Marketing Powered by Mailchimp
[link removed]
unsubscribe ([link removed]) .
Message Analysis
- Sender: Fairness & Accuracy In Reporting (FAIR)
- Political Party: n/a
- Country: United States
- State/Locality: n/a
- Office: n/a
-
Email Providers:
- MailChimp