Saturday, September 22, 2012

But someone was already here: teaching social studies in public schools

Colón llega a América by Gergio Deluci via Wikipedia
"Social studies is the neglected subject in school these days. Not enough time in the day." - My dad, a second grade teacher in Pinellas County, FL
This statement is a very true and very sad one. Thanks to budget cuts, and, according to my dad, more time scheduled for standardized test-related subjects during the school day (two hours per day just for reading, one for math), social studies (including history, government, economics, map skills, etc.) as a subject has been forced to take a seat on the back burner in public schools. Not that math and reading are unimportant by any means, and there are other subjects who've gotten the shaft even worse, like the arts and music (my dad says that his students only get 40 minutes per week total for these two!). I would argue, however, that social studies are just as crucial for kids growing up on today's globalized earth. Studies in this subject area can provide great benefits to critical thinking skills and a broader, more well-informed understanding of the greater world.

When I was a kid, the social studies curricula I learned at school was, for the most part, probably pretty standard fare for a public school system. But I learned about some of the information I think it takes to be a decent citizen of the world: basic things about the history of my home state (Florida), Native Americans of the region, fundamental concepts around macro- and microeconomics, U.S. geography, American government, U.S. history (including timeline stops at the Civil War, Prohibition, the Civil Rights movement, Vietnam, etc.), and World History (with a disproportionate focus on Europe and the so-called "Age of Enlightenment").


I give much credit to my social studies teachers, who actually gave a darn about what they were teaching and about kids learning things rather than just memorizing names and dates (there was a bit of that). For the most part, they made it interesting, which I appreciated because I have always loved the subject area, especially world history. I recall spending a good amount of time understanding the events, people, human experiences and histories of various social studies subject areas; when I was in middle and high school, I remember talking about the implications of historical lessons for the present day, and using problem solving skills to understand situations we might encounter in the real world. Content-wise, I learned a lot. I recall extended units on the Holocaust in more than one of my classes, units on Medieval European history, class discussions about the conflict in Bosnia and Herzegovina in the early 1990s, and debates about the OJ Simpson trial (the latter was not a major world event, but it was a national one with great political and cultural implications characterized by a divided public opinion that I observed even amongst my fellow 4th grade students).


Indeed, I learned quite a bit, and am thankful for the education that gave me the skills and worldview I have today. However, it wasn't perfect, especially when it came to presenting "alternative" viewpoints. For example, when we did units on World History, there was always a big emphasis on the European explorers and pilgrims, who we were taught to thank for their "discovery" of America and all the "great" stuff they did for our young country (like destroying the lives of the people who already lived here). The stories of Columbus and the other great European explorers were always presented in a very positive light. We were taught to think of them as folk-heroes because they made long and arduous journeys from their homelands to America, planting the seeds upon their arrival of the independence and patriotism that make up our nation's cultural and historical narrative. We never learned about any of the stuff you will find in, say, Howard Zinn's acclaimed book, A People's History of the United States (e.g. how the Europeans spread disease, stole land and resources, enslaved the natives and forced them to adopt Catholicism, propagated a greed-driven cultural genocide, etc.) Without a doubt, this book should be on every high school kid's summer reading list, and should be re-examined again (maybe even twice) as required reading in college. 


As kids, we celebrated Columbus Day without a second thought, grateful for the day off from school. With our paper headdresses and hand-traced turkeys, we took part in plays for our families about the first Thanksgiving between the pilgrims and the Indians. We reenacted that famous scene in which the Indians graciously shared their foodstuffs with the starving pilgrims, and everyone lived happily ever after (except for the Indians, but we didn't really focus on that too much in our studies). All the while, we were being enculturated with the values of sharing and caring through our reenactment of this celebrated day.


The First Thanksgiving, 1621, by J.L.G. Ferris via Encyclopaedia Britannica

I imagine that what students learn about the world and its history has a lot to do with where they go to school (geographically) and who teaches them (the academic training and background of teachers, the political or religious slant of the school, etc.) I say this because for some reason, the subject of Christopher Columbus, came up in a conversation I had the other day with my boyfriend about a song he learned about the famous explorer when he was in third grade. He went to school in Oakland, California, at a teachers college that was known for its liberal/alternative leanings. I was surprised when he described the song to me, because the story it tells is so different than what I learned throughout my K-12 education. Let's have a look at the lyrics:

1492

In fourteen hundred and ninety-two

Columbus sailed the ocean blue.
It was a courageous thing to do,
But someone was already here.

The Inuit, the Cherokee, the Aztec and Menominee,
Onondaga and the Cree…
Columbus sailed across the sea
But someone was already here.


Columbus knew the world was round

So he looked for the East while westward bound
But he didn’t find what he thought he found
For someone was already here.

It isn’t like ‘twas empty space
The Caribs met him face to face
Could anyone discover the place
When someone was already here?


So tell me who discovered what-

Thought he was in a different spot
Columbus was lost, the Caribs were not
They were already here.
There are at least two great things about this little jingle. First, it actually questions the grand narrative most commonly presented by teachers across the nation about the discovery of America (or at least that's how it was when I was in school). Instead of focusing on how awesome Columbus was for discovering new lands in the name of the greatness of Europe, it makes a point of emphasizing that there were actually already people living here. What a novel idea. And second, it wasn't just one homogenous group of people called "Native Americans" or "Indians", but a whole bunch of different people (seven are mentioned here). As a side note, the year my boyfriend learned this song (1992) happened to be the same year that next door neighbor Berkeley renamed Columbus Day "Indigenous Peoples' Day".

I often think about how different our society would be if students were afforded more time to learn about the past and present as history. Instead of spending inordinate amounts of time being drilled with math formulas and vocabulary words, what if they were taught to value the stories and cultures of other people, and to think critically about world events? This might sound radical to some, but what if in learning about these things, the information they were taught didn't just represent one side of the story? There are probably many students out there who benefit from such an educational perspective, but it doesn't seem to be a major component of teaching/learning in public schools today. 

And it doesn't have to start in high school. A lot of these topics might be more appropriate for middle school and high school than elementary school, but I feel that the latter is a crucial time during which children are really impressionable and during which it makes sense to teach them about some of the alternative viewpoints on history that aren't usually highlighted by run-of-the-mill public school classes. Kids can learn from the lessons of the past and the present from a much earlier age. I'm not suggesting we teach kindergarteners about the horrible cruelties of the European explorers in detail, but surely they would benefit from something more high-level, like talking about why it might be wrong to impose one's beliefs on someone else, and the importance of respecting people who are different from oneself.

Monday, September 17, 2012

On survey design, courtesy and opting-out: an open letter to StPetePolls.org, a.k.a. Fextel, Inc.

Image via this site

The following is an open letter to StPetePolls.org, a group run by parent company Fextel, Inc. that administers phone-based surveys to gauge the opinions of the residents of St. Petersburg, Florida, on various current and local issues of interest. I regularly receive phone calls from this organization (about every month or month and a half) to participate in surveys, even though I no longer live in St. Petersburg. A phone-based option to opt out of the call list has never been provided. I also regularly take issue with their approach to survey design, including their lack of providing sufficient/appropriate response options as well as their failure to operationalize, or define, certain words and phrases in their questions. Given that the data collected by this group could potentially be used by City officials for policy-related decision-making, the survey designers should work to ensure its reliability and validity. Failing to provide the appropriate number and type of response options or to define words or phrases that might be unclear to respondents causes me to question the data that StPetePolls.org has been collecting.


Hello,

My name is Amy Santee. I have been receiving survey calls from your organization for months now without any phone-based option to opt-out from them. I would like you to remove my phone number from your list, mostly because I no longer live in St. Petersburg, but also because the issues typically discussed in your surveys are not relevant to me, nor do I know enough about them to give my feedback.

I noticed on your website that you call people who are registered to vote in Florida for your surveys. You should probably check your list for accuracy, because I am not a registered Florida voter, and haven't been for about a year. When you allow this, it opens up the possibility of allowing non-residents and non-FL voters to participate, ultimately calling into question the validity of your data. You should also consider providing people who are not interested in taking your surveys with a phone-based option to opt-out, rather than just the option to email you. That is just common courtesy. As far as I can tell, the only way to remove oneself from your call-list is to contact you via this email address. This is extremely inconvenient and borders on harassment, especially if the individual does not want to take the entire survey just to get your contact information (website address) at the end. In order for me to find out that I had to email this address, I had to take the entire survey, and I don't even live in St. Pete, so all of my answers are invalid! I just kept hitting #2. Normally I would not do such a thing because I wouldn't want to mess with data validity. Think of the implications for the quality of your data if there are others in this situation!

Finally, as a consumer research analyst with a background in research design, I suggest that you reconsider the wording of some of your questions and the options you provide to survey respondents. In my opinion, some of the questions you have asked in past surveys have provided only some of the response options they should have. For example, on this latest survey, when asking about the redevelopment of the St. Pete Pier, you gave only two options for respondent preferences: 1.) build the new Lens pier design, or 2.) have no pier at all. What if the respondent wanted a third option or neither of these? There should usually be "other/none of the above", "not sure", and "no opinion" options in surveys. That is survey design common sense, and something you should know as a company focused on administering call center phone surveys.

Wording and operationalization are also very important. For another example, if you're asking someone if they support the "current police chase policy" (as you did in your August 2012 general survey), you either need to define what "current police chase policy" is or give the respondent the option to say they are not sure, because that may very well be the case. Citizens run the gamut in terms of how knowledgeable they are about current and local events. When you don't do this, you're forcing the respondent to either pick a choice that doesn't fit his or her true feelings about the issue, or to hang up the phone because they cannot continue the survey, which lowers your overall completion rates. Also, these issues are not as simple as asking questions and providing just two options to choose from; there are obviously many different perceptions on these important issues. You clearly can't cover them all either, but that doesn't mean the obvious ones should be neglected.

In this same survey, you asked if the respondent "supports the Republican National Convention kick-off party that will be hosted in St. Pete" at Tropicana Field in August. This question could be interpreted in multiple ways. Are you asking if the person supports the Republican Party, or if they support that the kick-off party is taking place in St. Pete, or if they support it taking place at Tropicana Field? I think you get my point.

I think it is great that you are taking on the task of researching these important topics and providing the perspectives of residents of St. Pete. However, as both a researcher and a past resident of St. Pete, I care too much about the sharing of misinformed research findings to not say something about this. I also really would like my name and number removed from your list. Please let me know that you received this message and that my name and number have been removed. If you would like to discuss any of the points I raised here, please feel free to get in touch via email or phone. You have my number.

Thank you,
Amy Santee


Update: Not long after emailing StPetePolls.org, I received this response:


Hello,

Thank you for the feedback. We have added your phone number(ending in "1377") to our do not call list so you will not be receiving any more calls from us.

We are using the August 2012 voter list from Pinellas County, and I have confirmed that you were still listed as an active voter. We should be receiving the September 2012 voter list this week, so we will have to take a look and see if you are still included in that list. We have had several people successfully remove their numbers from our list(and many other organizations' calling lists) by simply contacting the supervisor of elections and updating their voter records to remove their phone number.

For our polling we decided to use shorter questions without a long explanation because we receive a much higher response rate than when we give an in-depth explanation before each question. Most people won't stay on the line for a human operator or automated call if the call is over a minute in length.

As for the Pier question, we have asked about it several times, sometimes with a "something else" option and sometimes with a "refurbishment" option. We received a suggestion to offer only "the Lens or nothing", and we thought it would be interesting to see what kind of response we received to those limited choices.

Thanks again for your feedback,

info@stpetepolls.org 

Here are my thoughts on their response.

  • It seems that the Pinellas County voter registration list may not be quite up-to-date, which is obviously not the problem of StPetePolls.org. I hope they are keeping track of the number of folks with incorrect voting information and sharing this with the County. I also still disagree that people should either have to email them or call the county simply to have their names/numbers removed from the list. A phone-based option to opt-out should be provided for both convenience and courtesy.
  • I understand the point of having shorter questions. Yes - shorter and fewer questions yield higher response/completion rates. However, leaving out crucial information and assuming your respondents are privy to that information is also a big risk, the potential bias of which should be shared with those consuming the survey findings. As I mentioned in my initial email, if this is the route they're going to take, then respondents should be provided with "other/none of the above", "not sure" and "no opinion" options.
  • Asking people for their opinions about the Pier without including alternative options that are known to be popular with residents, just for the "interestingness" factor, doesn't make any sense at all. The data are not going to mean anything because those other choices are being ignored. This is a huge issue if the information is going to be shared with policymakers or other City officials. It's is comparable to a pizza chain asking people if they want only Hawaiian-style pizza or no pizza at all. You can't do anything with it. You also can't capture month-to-month trends for this question since you asked it in a different way than before.

Again, I applaud the efforts of StPetePolls.org in trying to capture the opinions of the residents of my hometown on these important issues. Unfortunately, I don't feel that the data are conclusive or should be used to guide anything policy related since there are too many potential issues with the validity of the data. At the very least, the risks and potential biases of this research should be shared with anyone with access to the results.

Here is the second and final response I received from StPetePolls.org:


Hello,

Our St Pete Polls are always evolving, we are looking into the best way to offer an opt-out method over the phone, but since this is something that almost no other automated phone survey organization offers, we will probably need to test a few different options before we roll out a solution across the board.

We have done surveys before with in-depth explanations before the questions, and the response rate is usually less than 1/4 of the response rate of the short question surveys, and the demographics are much less representative as well. So we have to call more people, and we get a higher margin of error with a less representational sample.

On the subject of "unsure/undecided" options, most of our paid polls have this, and almost all of our election polls have this as an option. We haven't offered this on our St Pete Polls very often because we wanted to force a decision in most cases. Again, we are always evolving, and we have received other recommendations on this, so we are looking into adding more options for some questions on future surveys. The Pier question has another side to it as well, many city officials have stated that renovation is not an option, and with a very tight budget for the Lens pier design and the millions already spent on it, our limited survey question of "Lens or nothing" is a very real possibility.

Thanks again,

info@stpetepolls.org  

Wednesday, September 12, 2012

On research incentives

Incentivizing people to participate in research is really important. Time is money, and not just for those of us working in business. Consumers have busy lives and we should thank them for taking time out of their day to participate in our surveys, focus groups, interviews, contextual observations, usability testing, and any other human-centered research for which we somehow manage to convince people to be our guinea pigs.

Anyone in the business knows that doing this well pays off in long-term ROI. But I feel like researchers more often compensate people for the more extended methodologies ($50-$100+/hr. on average for things like focus groups, interviews, etc.), and just expect that because it takes considerably less time and effort, consumers will take a web- or phone-based survey for free. Oh, and don't forget the company's admirable goal of "improving the customer experience"! Well, that's great and all, but I doubt customers see it as their job to improve the customer experience by spending 20 minutes answering survey questions. Yes, the long-term benefit is that their responses help companies to improve their products and services, but what's in it for them in the short-term? I bet they appreciate immediate gratification as much as we researchers do when conducting the research itself (high response rates are such things of beauty!)

Gap is one company that clearly gets this. For its current post-website visit customer experience survey, Gap offers participants the opportunity to select a charity to which it will make a donation (up to $12,000 per year; it doesn't specify how much is donated per participant). It can even be said that there is a double incentive here, since not only will they make a donation to a charity of the individual's choosing, but will also enter the person into a drawing for a chance to win a $200 AMEX gift card. Finally, as the image below shows, even those who are unqualified to take the complete survey (as I was when I attempted it earlier this week) are offered the same exact incentives. It would be nice if other corporate-based research followed in this company's footsteps.

Click image to enlarge

By the way, does anyone else feel that when a company uses a drawing entry as an incentive, or anything else with the word "chance" in it, it's almost like offering nothing? The sample size (500? 1,000? 5,000?) as desired by the researcher can really affect ones odds of winning the prize and potentially even their odds of taking the survey in the first place.