All Survey Questions Are Not Equal

By. Neil Williamson, President

Surveys are interesting and valuable tools for gauging public opinion.  They are limited however because as they seek to provide important contextual information to create an informed answer, they often slip, perhaps unintentionally, into a “push” poll, predetermining the outcome.

The recent  Jefferson Area Community Survey conducted by the Center for Survey Research (CSR) at University of Virginia’s Weldon Cooper Center offers some questions that in their wording or position (fixed, not randomized) in the questionnaire might prove questionable.

In a telephone interview, CSR Director Dr. Thomas Guterbock indicated each of the survey sponsors paid CSR a fee per question to be asked.  Thomas Jefferson Health District, Charlottesville Gas, WTJU (Public Radio) and the Jefferson Area Board on Aging along with Charlottesville Tomorrow funded the survey.

It is important to note, Charlottesville Tomorrow should be applauded for paying for what looks like the majority of the survey because it does provide many interesting data discussion points.  We do take issue with several of the questions.

First and foremost The Free Enterprise Forum appreciates that the survey repeated the question we posed in our 2004 Transportation Survey:

Do you believe a U.S. Route 29 Bypass around Charlottesville is needed, or not.

bypass Survey HeadlineAs survey questions go, this is a clear black and white question.  Over two-thirds (69.3%) of respondents said that such a bypass was needed.  This was even greater than the approval rating in our 2004 Survey.

The next question, however, fell into the aforementioned “contextual quandary”

As you may know, a Western Bypass of U.S. Route 29 has been approved. Some people have recommended alternative transportation investments as being both more effective in reducing traffic congestion and costing less than the bypass. Do you favor or oppose our elected officials evaluating these options as an alternative to the Western Bypass? Would you say you . . . .
1 Strongly favor
2 Somewhat favor
3 Somewhat oppose
4 Strongly oppose
8 UNABLE TO RATE/DON’T KNOW
9 REFUSED [emphasis added-nw]

Should more cost effective alternatives be considered? 

Who could be opposed to this? 

When the Free Enterprise Forum asked CSR Director Dr. Guterbock directly if he felt this question was fair, he said he believed the question was a fair “hypothetical” question.  We respectfully disagree.

This question provides as a given that such alternatives exist, and infers that funding could simply be shifted from one project to another.  The Free Enterprise Forum contends neither is the case.  In such a “What if” question It is NOT surprising that over 66% somewhat favor or strongly favor evaluation of alternatives.  What is surprising is that the rate is not higher considering the phraseology of the question.

There is a theory in survey research regardingCT Survey Graphic question ordering.  If you examine the order the questions regarding Albemarle County’s growth management strategies the survey starts with what I call an apple pie question:

“How important is the rural countryside, Albemarle County’s farms, fields, and forests, to your quality of life?

This question was asked in Charlottesville Tomorrow’s previous survey. While we can appreciate the desire for question integrity by repeating  the same questions that were asked in previous surveys, we find ourselves repeating the same criticism we raised in the earlier surapple pie sweetpeaskitchenvey.

Words like rural countryside, farms fields and forest have a positive, warm and fuzzy connotation and tend to elicit a positive result.  Considering there is no cost in the question, who doesn’t like apple pie.

After getting the majority of respondents to say, absent any cost, they value the rural countryside, the survey rolls into a series of questions about the Western Bypass and then comes back to a rural areas question:

Do you favor or oppose having the County Board of Supervisors change Albemarle’s designated growth areas to create new locations for business on land currently zoned as rural countryside?

Not only does this question not include any cost issues, it could be inferred that the entire rural area may be in jeopardy.  There really is no “right” way to ask this question without shading the answer.

I believe the answers would have been different if asked in the same tenor as the Western Bypass Alternatives Question above: 

PROPOSED SURVEY QUESTION: The Existing Development Area Boundaries were established in 1980.  Much has changed since then.  Some have indicated that Albemarle County is losing career ladder jobs, should supervisors consider alternatives to the current growth area boundaries that will promote economic development and allow new jobs to come to Albemarle?

If the question was asked, what do you think the response to such a question might be?

Do you think such a question is fair?

Compare this question to the one listed for the Western Bypass Alternatives.  Now do you think either is a fair way to gauge public opinion?

After examining the Jefferson Area Community Survey, we find we have more questions than answers.

Respectfully submitted,

Neil Williamson

—————————————————————

20070731williamson Neil Williamson is the President of The Free Enterprise Forum, a privately funded public policy organization covering the City of Charlottesville as well as Albemarle, Greene, Fluvanna, Louisa and  Nelson County.  For more information visit the website www.freeenterpriseforum.org

Photo Credits: graphics: Charlottesville Tomorrow, Photo www.sweetpeaskitchen.com 

 

Advertisements

2 responses

  1. There is a phenomenally adumbral standard set up and being perpetrated here. Such is so scamy and so tactless, one would have the expectancy of Richard Dawson to dart out from a stage backdrop and shout “survey says.” Where and when do these research firms and survey organizations ever lay off and give the public some breathing space? Well at least, some enterprising entrepenuer (hopefully from around here) is earning a sweet livelihood off it. How sorely miserable for the rest of us.

    With appreciativeness for behalf of an unaware public, the Free Enterprise Forum has pinpointed the poison in the well with this. “Not only does this [question] not include any cost issues, it could be inferred that the entire rural area maybe in jeopardy. There is no ‘right’ way to ask this [question] without shading [the answer.]” So what are we left to believe, that some painstaking manipulation of a sort has NOT been undertaken? Au contraire, au contraire!

    Anyone from Charlottesville has seen a similar tact, taken with the parade of “annual citizens budget surveys” from the local government’s municipal website and mail-outs. These often touch and do seldomly skirt about “cost issues.” Does it make these surveys all the more useful or on the level? Hardly but not absolute. How about actually disseminating instead of concocting information with such tools?

    Ah but the people could cry for unencumbered access by less of these tactical played predictors. Sadly there’s no certainty any of the noted surveys will be henceforth presented all neutrally or in diminished volume. What more likely would a deaf ear be turned upon, a single individual or what some blind survey is behind?

    How ironic if it were not all simultaneously far fetched and far reaching. Free Enterprise Forum played a “good for goose is too for gander” turnaround, and bespattered back its own proposed survey shot to aforementioned entities. To answer the Forum’s deadpan seriously tacked on question: no, it is not fair and it is also quite disparaging.

    Still the last time any of us looked, a poll is a poll. What poll isn’t some protected form of free speech? Perhaps we should treat these like getting a referendum on the ballot? Not impossible but so very improbable. Of course, there is the exact opposite extreme. Charlottesville could emulate Orlando Florida, and let the survey instigators be fruitful and vastly multiply. Like Easter Island, it doesn’t guarantee new employers don’t stay and that established residents won’t move away.

  2. I had a brilliant statistician work for me at my last employment. During the hiring interview she handed me a list of ten important things I should know about her. Among them she had graduated first in her class for staistics from Rutgers and she had won multiple awards at her position for zero audit mistakes over her time at the IRS.

    When I finished reading I said, Why wasn’t this included with the rest of her paperwork when she sent me her resume? Because it is all true and all a lie at the same time she explained. She was the only graduate that year in statistics so she was also last in her class. She had been awarded about a dozen commendations for never making a mistake on the audit because, though it was her departments responsibility to do the particular audit on the certificate, she had never done one, so she was flawless by default.

    She had that rare gift of being able to write survey questions that are untainted regardless by her personal prejudices. Most of our surveys we various technical or IT departments and were very valuable. Yet she is correct in my judgment with her claims that any survey we, or any group, sent to the general populace in regards to local issues, schools, quality of life, so forth, is inherently worthless for a multitude of reasons. Foremost to get correct answers there is often hundreds of words, fact, figures and supporting data that would need to be read in order to even attempt to answer the question.*So, all surveys targeted at the general populace are really only gauges of ‘feelings’ and nothing more. She also noted studies that concluded that the average general populace survey respondent read the first 11 words and then looked at the choices , answered and moved on.*

    So take all surveys of this type this with a grain of salt regardless of how well the questions are written.

    * A question about the Bypass or the Meadow Creek Parkway, Dams versus dredging, or even something a simple as making changes to traffic patterns would require volumes of reading to provide an informed answer.

    ** As an example, the reader determines this is an attempt to figure out how you feel, neutral, pro-Obama or Anti-Obama by word 11 of the question, then they skip to the choices and pick an answer based on how they feel about the President, not the balance of the question.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: