Monday, September 17, 2012

On survey design, courtesy and opting-out: an open letter to StPetePolls.org, a.k.a. Fextel, Inc.

Image via this site

The following is an open letter to StPetePolls.org, a group run by parent company Fextel, Inc. that administers phone-based surveys to gauge the opinions of the residents of St. Petersburg, Florida, on various current and local issues of interest. I regularly receive phone calls from this organization (about every month or month and a half) to participate in surveys, even though I no longer live in St. Petersburg. A phone-based option to opt out of the call list has never been provided. I also regularly take issue with their approach to survey design, including their lack of providing sufficient/appropriate response options as well as their failure to operationalize, or define, certain words and phrases in their questions. Given that the data collected by this group could potentially be used by City officials for policy-related decision-making, the survey designers should work to ensure its reliability and validity. Failing to provide the appropriate number and type of response options or to define words or phrases that might be unclear to respondents causes me to question the data that StPetePolls.org has been collecting.


Hello,

My name is Amy Santee. I have been receiving survey calls from your organization for months now without any phone-based option to opt-out from them. I would like you to remove my phone number from your list, mostly because I no longer live in St. Petersburg, but also because the issues typically discussed in your surveys are not relevant to me, nor do I know enough about them to give my feedback.

I noticed on your website that you call people who are registered to vote in Florida for your surveys. You should probably check your list for accuracy, because I am not a registered Florida voter, and haven't been for about a year. When you allow this, it opens up the possibility of allowing non-residents and non-FL voters to participate, ultimately calling into question the validity of your data. You should also consider providing people who are not interested in taking your surveys with a phone-based option to opt-out, rather than just the option to email you. That is just common courtesy. As far as I can tell, the only way to remove oneself from your call-list is to contact you via this email address. This is extremely inconvenient and borders on harassment, especially if the individual does not want to take the entire survey just to get your contact information (website address) at the end. In order for me to find out that I had to email this address, I had to take the entire survey, and I don't even live in St. Pete, so all of my answers are invalid! I just kept hitting #2. Normally I would not do such a thing because I wouldn't want to mess with data validity. Think of the implications for the quality of your data if there are others in this situation!

Finally, as a consumer research analyst with a background in research design, I suggest that you reconsider the wording of some of your questions and the options you provide to survey respondents. In my opinion, some of the questions you have asked in past surveys have provided only some of the response options they should have. For example, on this latest survey, when asking about the redevelopment of the St. Pete Pier, you gave only two options for respondent preferences: 1.) build the new Lens pier design, or 2.) have no pier at all. What if the respondent wanted a third option or neither of these? There should usually be "other/none of the above", "not sure", and "no opinion" options in surveys. That is survey design common sense, and something you should know as a company focused on administering call center phone surveys.

Wording and operationalization are also very important. For another example, if you're asking someone if they support the "current police chase policy" (as you did in your August 2012 general survey), you either need to define what "current police chase policy" is or give the respondent the option to say they are not sure, because that may very well be the case. Citizens run the gamut in terms of how knowledgeable they are about current and local events. When you don't do this, you're forcing the respondent to either pick a choice that doesn't fit his or her true feelings about the issue, or to hang up the phone because they cannot continue the survey, which lowers your overall completion rates. Also, these issues are not as simple as asking questions and providing just two options to choose from; there are obviously many different perceptions on these important issues. You clearly can't cover them all either, but that doesn't mean the obvious ones should be neglected.

In this same survey, you asked if the respondent "supports the Republican National Convention kick-off party that will be hosted in St. Pete" at Tropicana Field in August. This question could be interpreted in multiple ways. Are you asking if the person supports the Republican Party, or if they support that the kick-off party is taking place in St. Pete, or if they support it taking place at Tropicana Field? I think you get my point.

I think it is great that you are taking on the task of researching these important topics and providing the perspectives of residents of St. Pete. However, as both a researcher and a past resident of St. Pete, I care too much about the sharing of misinformed research findings to not say something about this. I also really would like my name and number removed from your list. Please let me know that you received this message and that my name and number have been removed. If you would like to discuss any of the points I raised here, please feel free to get in touch via email or phone. You have my number.

Thank you,
Amy Santee


Update: Not long after emailing StPetePolls.org, I received this response:


Hello,

Thank you for the feedback. We have added your phone number(ending in "1377") to our do not call list so you will not be receiving any more calls from us.

We are using the August 2012 voter list from Pinellas County, and I have confirmed that you were still listed as an active voter. We should be receiving the September 2012 voter list this week, so we will have to take a look and see if you are still included in that list. We have had several people successfully remove their numbers from our list(and many other organizations' calling lists) by simply contacting the supervisor of elections and updating their voter records to remove their phone number.

For our polling we decided to use shorter questions without a long explanation because we receive a much higher response rate than when we give an in-depth explanation before each question. Most people won't stay on the line for a human operator or automated call if the call is over a minute in length.

As for the Pier question, we have asked about it several times, sometimes with a "something else" option and sometimes with a "refurbishment" option. We received a suggestion to offer only "the Lens or nothing", and we thought it would be interesting to see what kind of response we received to those limited choices.

Thanks again for your feedback,

info@stpetepolls.org 

Here are my thoughts on their response.

  • It seems that the Pinellas County voter registration list may not be quite up-to-date, which is obviously not the problem of StPetePolls.org. I hope they are keeping track of the number of folks with incorrect voting information and sharing this with the County. I also still disagree that people should either have to email them or call the county simply to have their names/numbers removed from the list. A phone-based option to opt-out should be provided for both convenience and courtesy.
  • I understand the point of having shorter questions. Yes - shorter and fewer questions yield higher response/completion rates. However, leaving out crucial information and assuming your respondents are privy to that information is also a big risk, the potential bias of which should be shared with those consuming the survey findings. As I mentioned in my initial email, if this is the route they're going to take, then respondents should be provided with "other/none of the above", "not sure" and "no opinion" options.
  • Asking people for their opinions about the Pier without including alternative options that are known to be popular with residents, just for the "interestingness" factor, doesn't make any sense at all. The data are not going to mean anything because those other choices are being ignored. This is a huge issue if the information is going to be shared with policymakers or other City officials. It's is comparable to a pizza chain asking people if they want only Hawaiian-style pizza or no pizza at all. You can't do anything with it. You also can't capture month-to-month trends for this question since you asked it in a different way than before.

Again, I applaud the efforts of StPetePolls.org in trying to capture the opinions of the residents of my hometown on these important issues. Unfortunately, I don't feel that the data are conclusive or should be used to guide anything policy related since there are too many potential issues with the validity of the data. At the very least, the risks and potential biases of this research should be shared with anyone with access to the results.

Here is the second and final response I received from StPetePolls.org:


Hello,

Our St Pete Polls are always evolving, we are looking into the best way to offer an opt-out method over the phone, but since this is something that almost no other automated phone survey organization offers, we will probably need to test a few different options before we roll out a solution across the board.

We have done surveys before with in-depth explanations before the questions, and the response rate is usually less than 1/4 of the response rate of the short question surveys, and the demographics are much less representative as well. So we have to call more people, and we get a higher margin of error with a less representational sample.

On the subject of "unsure/undecided" options, most of our paid polls have this, and almost all of our election polls have this as an option. We haven't offered this on our St Pete Polls very often because we wanted to force a decision in most cases. Again, we are always evolving, and we have received other recommendations on this, so we are looking into adding more options for some questions on future surveys. The Pier question has another side to it as well, many city officials have stated that renovation is not an option, and with a very tight budget for the Lens pier design and the millions already spent on it, our limited survey question of "Lens or nothing" is a very real possibility.

Thanks again,

info@stpetepolls.org  

No comments:

Post a Comment