Saturday, July 28, 2012

Survey no-nos: Holiday Inn Express

I'm no expert in survey design, but I know a thing or two about the basics. It's pretty researcher-nerdy of me, but I actually enjoy filling out surveys for other companies, not only because I like to share my opinion but because I take interest in seeing how other companies put theirs together (it's a great opportunity to get ideas for questions for future research projects).

I especially love it when I stumble upon what I like to call "survey no-nos" - questions, wording or anything else that should typically be avoided in questionnaire design (e.g., leading questions, no opportunities for open-ended responses, etc.) Here are a few gems I encountered while taking a customer experience survey for a recent stay at a Holiday Inn Express.


My immediate reaction to this page of scale questions was that the scale is going in the wrong direction. Rather than ranking each item from 10 to 1, it should be read across the page, left to right, from 1 to 10. If you're administering the survey to people who were educated to read from left to right, they're likely going to be confused that the scale is flipped. They may not even notice it. This may cause issues with your data if people don't actually read the scale at the top and just assume it's in the order they're used to.


On this next page, respondents are again asked to rank each item. My issue here is with the fourth item down: "Holiday Inn Express is a brand for me." What does that even mean? What are you asking? It looks like they're trying to gauge whether or not people think of a Holiday Inn Express as a brand. If so, how do you define brand? The word "brand" has many different emotional, psychological and cultural implications, and could be interpreted to mean different things. Suggestion: don't use wording that is open to any sort of interpretation, unless you operationalize those words for the respondent.

---

I don't have a screen shot of them, but there were two other questions in this survey that bugged me. One asked the respondent to quantify the number of nights they have stayed in a hotel within the InterContinental family of hotels (of which Holiday Inn Express is a member) within the past 12 months. Then it asked the respondent to answer the same question again, this time for non-InterContinental hotels. There are a few things wrong with this. First, how is anyone going to remember how many nights they've stayed in any hotels in the specified period of time - one year?! Self-reported questions about behavior or anything requiring the respondent to remember such things probably won't yield very accurate data. Second, will people recall the names of each and every hotel chain they stayed at during that time? Third, this question is just too much work. If I weren't so into taking surveys and sharing my opinion, and were I not a believer in Research Karma, I would just enter some random estimate off the top of my head and move on. Finally, if you're not paying people to take a survey, don't ask them to do more work than can be accomplished in a few minutes and a few clicks.

Lastly, there was an open-ended question that allowed respondents to provide more details if they reported having an issue at the hotel. However, the response box only allowed for 999 characters. When I filled mine in, I used up most of the space to discuss a rather serious occurrence that took place during my stay. There were a couple other issues I thought the hotel management would like to know about (lack of electrical outlets near the bed, slow internet, clogged toilet). Alas, I had run out of characters and didn't have the room. Respondents should be provided with as much space as they need to elaborate if they so choose. Some people probably don't need much space, I don't see any benefit in limiting them except to save time on the analysis end. I'd stay away from doing this to maximize the opportunity to collect good data if that's what people are willing to provide.

4 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. As a fellow survey geek, I offer the following:

    On the order - yes, I suspect the reason is that the sought after response "10" is the first thing that you see, and therefore, most likely to respond to. In a recent survey by the Faculty Senate at the University of Memphis where Administrators were being ranked on effectiveness, the least favorable rating was on the left and the most favorable on the right. This arrangement is consistent the presumed sought after response given the routine Faculty Senate rants on the ineffectiveness of the University Administrators.

    The 999 character thing is a real issue to me as well. By the time one spits out names, dates, place and such, one is left with very little room for detail. I take this to mean they are not really interested in my concerns or opinions short of "Holiday Inn, Peroria IL, May 22, 2012 - dead body left in bed of room - had to sleep in chair" type of comments.

    I rank Holiday Inn with Sears, and several other corps I have dealt with over the years - they are really only looking for egregious problems and trends. I suspect that the routine data are compiled and sent out to each hotel with notes like "Based on our national data people ranked 8 on likely to stay again at HI - at your place it was only a 4 - shape up"

    I am a firm believer in reviews. I do tripadvisor.com regularly. I particularly appreciate the times when I have given either a negative or positive review and the venue actually responds in the comment sections of tripadvisor. At least they project an image of responsiveness.

    If you want a real negative treat, try Starbucks online customer service!

    ReplyDelete
  3. Dr. Connolly - I absolutely agree with your comment about purposefully ordering the responses for a scale question in the "wrong" direction to potentially positively influence the data/findings. It's hard to say how many people actually pay attention when they're taking a survey, and I can see how that would benefit survey designers with personal or business-related agendas. Thanks for bringing that up.

    I, too, am a fan of tripadvisor.com, but only recently signed up for it. I've been doing a lot of traveling for work and otherwise, and frequently use it for both writing and reading reviews (especially when I'm looking for a decent hotel). I've seen many reviews with responses from hotel managers and such, but some of them seem pretty formulaic. Others seem intentionally choosy about what they respond to. For example, I left a mostly negative review of a Downtown Chicago Holiday Inn Express; their approach to my response was to thank me for the one or two positive things I mentioned at the end.

    Thanks for checking out my blog!

    ReplyDelete