What’s wrong with this question?
August 10, 2006
A flurry of questionnaires have inundated Aspen recently, each earning its share of criticism for poorly worded questions. The survey war began after the city of Aspen held two community meetings with survey-style, instant-feedback questions that some local residents found to be biased. Those anonymous locals subsequently commissioned a telephone survey of local opinions.But that survey, developed by The Connections Group, a Seattle-based firm that advises political campaigns, companies and individuals on media strategies and public relations, was widely criticized, too – first for asking leading questions and then because telephone interviewers told a number of participants that the “Aspen Planning Commission” had sponsored the survey. The city had no part in that survey.Soon after, a discussion of a survey conducted by historic preservationist Les Holst and The White Shirts, a group of community activist that wants to slow the pace of development in Aspen, was printed in The Aspen Times. Some respondents to that survey suggested Aspen might have a reputation for being pretentious, which prompted some people to question how the survey was crafted.David Becher, an analyst with the Boulder-based market research and strategic consulting firm RRC Associates, says it’s important to think about the wording of survey questions to ensure useful results.”Wording can influence the result you get,” he said, so when crafting a survey, “it probably helps to be aware of the kind of pitfalls that come from a poorly worded question. …”There are subtleties in wording and question order that can affect the results,” he added, and certain types of questions can “inadvertently lead respondents to answer in a particular way.”RRC Associates specializes in the ski and snowboard industry, and the Aspen Chamber Resort Association contracts the firm to conduct its annual survey. The company does not do political market research, Becher said.
Writing good questions is an art, as evidenced by the large body of information offering guidance on how to do just that.Questions can be either open-ended, in which the respondent writes in a response, or closed-ended, in which the respondent chooses among a set of answers.The website for the company StatPac, specialists in survey-style market research, outlines some of the qualities of a good question.Closed-ended questions should “accommodate all possible answers,” the company states. “Asking a question that does not accommodate all possible responses can confuse and frustrate the respondent.”For example, asking the question, “What brand of computer do you own?” but offering only two choices, IBM PC or Apple, can be frustrating for someone who owns a different brand or who doesn’t own a computer at all.Some participants at the city’s meetings were frustrated by this problem. In one instance, participants were asked to complete the following statement about trees in Aspen:Aspen has …• Too many trees• A perfect amount of trees• Not enough trees• Trees in the wrong places• The trees on the mall are too tall• I’ve never even thought about it beforeUpon seeing the choices, one woman commented that the city has the wrong type of trees, but that was not an option.
“Double barrel” questions are those that pose more than one question at once, making it difficult to interpret responses.One of the multiple-choice questions in the city’s instant-feedback community forum asked respondents to choose among several answers to finish the statement, “Regarding this technology and this type of meeting …” One of the response choices was, “I can’t wait to do this again – start on transportation as soon as possible.”A total of 84 percent of respondents chose that response, but as the meeting’s facilitator noted, it’s not clear if they liked the technology or if they were ready to tackle transportation issues – or possibly both.
A good question “does not presuppose a certain state of affairs,” according to the StatPac website. Asking the question, “Are you satisfied with your current auto insurance,” for example, assumes the respondent has auto insurance.The Connections Group asked respondents to evaluate the following statement about the city’s land-use codes: “The proposed code would not allow an owner to build a duplex on the property and live in one half of it.” Possible responses ranged from “strongly in favor” to “strongly oppose.”Critics of that survey have said this question and several similar ones assume the council has proposed changes to the land-use code. Council members are examining the code, but they have not proposed any specific changes.What can you really conclude from results?Holst and The White Shirts conducted their survey during the Food & Wine Magazine Classic in June. The group passed out questionnaires asking people to answer six open-ended questions and one multiple-choice question about their likes and dislikes about Aspen.The website for the College of Education at San Diego State University states that the “drawback to open-ended questions is that the responses are more difficult to catalogue and interpret” than closed-ended questions.Holst and The White Shirts did not summarize responses. Rather, they bound all the returned surveys and gave copies to the City Council with a short preface, which states that “the questions weren’t leading” and that the sponsors “have not edited or discarded” any of the surveys.Councilman Jack Johnson read all the responses but said he “couldn’t discern trends from it because it’s just answers,” he said. Nonetheless, the remarks about Aspen’s attitude did stand out.”I thought it was interesting how many people talked about pretentiousness,” he said.He also noted that most respondents said they preferred older, small lodges or historic hotel rooms.Although Johnson found the responses interesting, he said it didn’t offer him much new information and that he “couldn’t draw any conclusions from it.””I don’t find any inconsistencies between what I learned at the city’s meetings and Les’ survey,” he said. “Is this valuable? Yeah. We get an idea what these particular people at Food and Wine thought.”
An entry for “polls and surveys” in The Associated Press Stylebook, a manual for newspaper style, states that who paid for a survey is required information when it comes to polls, and it cautions readers to be “wary of polls paid for by candidates or interest groups. … Any reporting of such polls must highlight the poll’s sponsor, so that readers can be aware of the potential for bias from such sponsorship.”The Connections Group survey has been criticized for failing to include the names of the sponsors.Shortly after that group conducted the survey, company president and CEO Cathy Allen said her firm was hired by a “sizable list of property owners and homeowners,” all Aspen residents, to research local views on development. But, she said, they wish to remain anonymous.The Associated Press Stylebook includes a long entry on polls and surveys, outlining the rationale for including information such as who conducted the poll and who paid for it, how many people were interviewed and who they were, among others. For each of three recent polls some of the people involved in drafting the questions have stated that their questions are unbiased. But the AP’s entry suggests that it is up to the reader to determine the value of the information in a poll, and certain information is necessary “for an intelligent evaluation of the results.”Abigail Eagye’s e-mail address is firstname.lastname@example.org
• StatPac: http://www.statpac.com• College of Education at San Diego State University: http://edweb.sdsu.edu• The Associated Press Stylebook: See entry for “polls and surveys.” The AP Stylebook is available online for a fee, or search Google for “Associated Press Stylebook polls and surveys” to reach a link to a pdf version of that entry.Survey snapshotsCity of Aspen CORE Beliefs meetings at the Hotel Jerome ballroom.Conducted by: City of AspenPaid for by: City of AspenNumber of people interviewed (estimate): More than 400Who participated (estimates): 63 percent Aspen residents; 18 percent Pitkin County residents (not in Aspen); 11 percent Eagle and Garfield County residents; 6 percent second-home owners; 2 percent “other” How were participants selected? Open to the publicHow was the poll conducted? Powerpoint slides with questions; participants responded simultaneously and anonymously with remote-control paddles; results tabulated instantlyWhen was the poll taken? July 19Sampling error margins for the poll: Not availableQuestions asked: Available at http://www.aspenpitkin.com; link to “Core Beliefs Information” then “Core Beliefs: Instant Voting results” for questions and compiled responsesAnonymous telephone surveyConducted by: The Connections Group, a Seattle-based firm that advises political campaigns, companies and individuals on media strategies and public relations, was commissioned to perform the survey. Inquire Research, a phone room, was hired to call area residents and ask the questions; CRG Research, a firm that conducts “multi-tiered market research for businesses, campaigns and the government,” was also involved, but its role is not clear.Paid for by: UnknownNumber of people interviewed: 249Who participated: Surveyors asked the following questions• Do you own property in Aspen? Yes: 66.3% No: 33.7%• Is your property your home: 91.5%; other residential property 12.1%; commercial property 4.2%; no answer 0.6% (Some respondents gave more than one answer, so total percentage is more than 100%)• Is your home employee housing or is it a free-market dwelling? Owns free market home 42.6%; rents free market home 15.7%; owns ADU 22.5%; rents ADU 15.7% (surveyors do not define ADU); owns unknown 1.2%; rents unknown 2.4%• Annual household income category: Less than $75,000 41.7%$75,000-$150,000 39.0%$151,000-$300,000 9.6% $301,000-$500,000 3.2%$500,000-$800,000 2.8%Over $800,000 3.7%How were participants selected? UnknownHow was the poll conducted? Telephone surveyWhen was the poll taken? July 22-25Sampling error margins for the poll: Not availableThe White Shirts surveyConducted by: Local historic preservationist Les Holst and an informal group known as “The White Shirts”Paid for by: The sameNumber of people interviewed (estimate): More than 110 Who participated (estimates): Majority of participants were from out of town; some locals How were participants selected? Unknown How was the poll conducted? Paper surveys handed out; responses written in When was the poll taken? During the Food & Wine Magazine Classic in Aspen, June 16-18 Sampling error margins for the poll: Not available Questions asked:• Name• Hometown• How many times have you visited Aspen?• What is it that makes Aspen special?• If you were describing Aspen to a friend, what words would you use?• What do you least like about Aspen?• As a visitor to Aspen, do you feel that the historic mining character makes it a special place?• Should it be preserved?• What lodging do you prefer? Older small lodges, historic hotels, newer luxury condos, Ritz-type contemporary lodging or timeshare hotel?Note: The White Shirts compiled all completed surveys, bound them and gave copies to the Aspen City Council with the following preface:”This first section is comprised of questionnaires filled out by out of town participants to the Food and Wine Classic. Our questions weren’t leading, but covered some areas we thought important. The demographics of the participants are appropriate as they come here for a reason, and know what they are looking for.”We have not edited or discarded any of these items, and there are a few filled in by Aspenites at the end, as they wanted to be included. We believe that if you read all of them you might begin to realize that the people you should be catering to are not the local realtors and contractors, but the tourists who really drive the economy of this town. If you are doing any national information hunting, we believe that you will find that the comparable towns to us who have a NO demolition policy, and strong Historic Preservation, have more vibrant economies and a much more sustainable future.”A few locals wanted to be heard, so we put them at the end of the out of town survey.”Survey done June 2006”