Patrick Madden

Apple’s iPad Finally Makes Survey Research Cool

2010-08-11T13:13:00.001-05:00

It took Steve Jobs to do it, but the iPad has finally given the survey research industry a tool that makes the average person interested in doing surveys. USA Today (the place where I get all my news, usually in picture form) reports on companies that have started using iPads to conduct face-to-face interviews at shopping malls with glorious results. According to the article:

People "are attracted by the cool factor," says Jude Olinger, CEO of the Olinger Group, a marketing research firm that conducted surveys at 130 shopping malls for the past two months using 200 iPads. "People who haven't seen iPads are fascinated."

At many of the centers, he says, response was so good that survey takers collected the required information in about three weeks instead of the four they'd anticipated.

If true, a 25% reduction in data collection time is actually a significant amount. The time and money saved not having to pay interviewers an extra week probably comes close to paying for the cost of buying all those iPads to begin with. The article mentions “clipboard-wielding researchers” and “pencil-and-paper surveys” but given these companies’ early adoption of iPads, it’s more likely that they were already using laptops or tablet PCs to conduct similar surveys. A company that is still doing pencil and paper surveys in the year 2010 is probably not going to immediately jump on the Steve Jobs bandwagon.

But the long-term rub is that while iPads are a great way to get people’s attention right now, the novelty will soon wear off. If I tried to get you to do a survey on my cool, new PalmPilot, I doubt you’d be that interested. In the same way, as iPads become more ubiquitous, the cool factor is going to go out the window along with their ability to get people do want to do surveys on them.

That being said, iPads do offer a number of advantages over old-school pencil and paper surveys, the most important of which is being able to compile, verify, analyze and report on data in real time. We’ve been able to do the same thing with PDA’s, laptops, and tablet PCs for a while now, but the iPad offers a unique combination of usability, portability and (most importantly) battery life. Much like Ron Burgundy, having 10 hours of battery life when you are conducting face-to-face interviews in a mall all day long is kind of a big deal.

History tells us that other manufacturers will eventually catch up to what Apple is doing, but until then, all hail the iPad as the survey research tool of the future. At least until people get bored with it.

Fake Your Data like a Machine

2010-07-14T16:06:00.002-05:00

You may have heard the news about a little polling company called Research 2000, which has gotten into a little hot water recently for supposedly faking polling data during the 2008 election. The Daily Kos uncovered the story based on some tips from a few statistical wizards who spotted some abnormalities in the data.

To break it down, in addition to the Daily Kos, Research 2000 provided polling services for a large number of local television and newspaper affiliates. Well, “provided” in the sense that they will likely not be providing said research services much longer. Research 2000 president Del Ali quickly and unsurprisingly shot back against the charges of faking data, writing in a statement:

Every charge against my company and myself are pure lies, plain and simple and the motives as to why Kos is doing it will be revealed in the legal process and not before that. I will share one little minor reason that Kos is doing this and it pertains to the fact they owe us a significant sum of monies that is in the six figure category and payment was on June 15, 2010.

Of course, the fact that he won’t publicly release his data (likely because it doesn’t exist) does hurt his credibility a little.

But this brings us to the larger and more important issue – when you fake your survey data, don’t do it like a human being. See, the world has both a randomness and an order to it that our feeble minds can’t quite grasp. And when we try to randomize things, we do it in a much too orderly fashion.

The thing that brought down Research 2000 is that their data was much too “clean.” It didn’t have the error associated with it that one would expect from a random survey. For example, when you fake your data, don’t make all the breakdowns either even OR odd. It’s best to mix it up a little.

Bad fake data when ALL the male/female comparisons are both either even or odd

Also, when you fake your data, you most likely want to make sure it is normally distributed, because that’s how the world operates. See, this Gallop poll demonstrates what a normal distribution looks like. It's what's refered to the Bell curve.

This Research 2000 poll on the other hand, demonstrates that humans don’t like the number 0 when faking data.

There were a number of other problems with the data that have been well documented on the Daily Kos website. It all adds up to a damning set of evidence indicating that Research 2000 faked some serious-ass data. Hundreds of thousands of dollars worth of data. And didn't do it particularly well.

The fact that the company didn’t have a mailing address and operated out of a Kinko’s probably should have been enough of an indication that something wasn’t right.

About Me

2007-03-29T08:03:00.007-05:00

Contact Information

Patrick Madden
75 Washington Ave, Suite 206
Portland, ME 04101
pmadden@marketdecisions.com

Who is this Guy?

I was born in the city of Fall River, Massachusetts around the time that Steve Jobs and Steve Wozniak released the Apple II+ (now with 48K of RAM)! Fall River is probably best known as the hometown of Lizzie Borden, or if you’re not so gruesome, home to the largest collection of WWII battleships in the world.

I did not stay there long. Instead my family bounced around the New England area, trying out different states in the region. Growing up, I loved sports (particularly the Red Sox and Celtics) and video games. Sounds like just about every boy more or less. But I also remember creating my own computer spreadsheets when I was about 13 years old to track the results of my fake basketball league (where I was the star player of course). That, I assume, is a little unusual.

Education

MBA, University of Southern Maine, Business Administration (2003)
B.S., University of Southern Maine, Business Administration (2001)

When my professional baseball career flamed out at the age of 17 (I had a tough time hitting that high school curveball), it was time to come to the harsh realization that I was not going to be the second coming of Wade Boggs.

Since I was a solidly-above average high school student (but not in a showy kind of way) I did have my pick of non-elite colleges as long as I was willing to pay for them. I ended up picking my college based on it’s proximity to mountains that I could ski, which looking back, is probably not the best way to choose a college. I would not recommend that method to my kid.

However, it turned out ok, and I even decided to stay there an extra couple of years to get my MBA. To this point in my life, the best thing about my MBA is that it allowed me to stay in school for an extra two years.

Survey Research

Senior Research Analyst, Market Decisions, LLC (2003-present)
Research Assistant, Center for Business and Economic Research, University of Southern Maine (2000-2003)

Being a solidly above average student (but not in a showy kind of way) and in the graduate program at USM afforded me the opportunity to become a graduate and research assistant at CBER. Less than a decade removed from the experience and I can’t even tell you what CBER actually did, but I remember that it 1) gave me my first introduction to survey research and primary data that was collected specifically for things WE wanted to know, and 2) gave a data geek like me access to cool toys like SAS.

Of course, being a graduate student living off your credit cards and getting paid like an indentured servant can only last so long and eventually I was forced to get a real job, as they say. It feels like an eternity ago, but 2002-03 was also a bad time for job hunting, as we were coming off the tech/internet “boom” of the late 90’s. It took a while, but after many, many failed job interviews, a company called Market Decisions was desperate enough to hire a young guy with little experience and teach him the survey research ropes.

Fantasy Sports

Give Me The Rock
Creating my own fake basketball leagues on computer spreadsheets naturally led to fantasy sports, which led me to creating my own spreadsheets for fantasy sports, which led me to the world of blogging for fantasy basketball. I created my own now-defunct blog back in 2004, but joined forces with Nels at Give Me The Rock in 2005 to form the MBA power-blogging duo at the longest running fantasy basketball blog in the world. Everyone needs their niche.

House Projects

2005 was a petty busy year for me. I got promoted, got married and bought a house. A nice, old house that looked like it was last updated in 1985 and need a lot of TLC. A combination of ambition, cheapness, and ignorance led me to take on many home projects myself. Most have been successful and most have gone over budget and way over time. But the results, while painful at times, have been decent.

Skills

You mean like nunchuck skills, bowhunting skills, computer hacking skills?

It’s too bad that Jared Hess and Jon Heder turned out to be one trick ponies, because that was an awesome movie.

Introduction to Internet Surveys

2010-07-12T19:29:00.000-05:00

The internet represents the fastest growing and most exciting way to conduct survey research. If you’ve been online for more than a couple minutes, there is a 100% chance that you’ve either signed up to participate in an online survey or have at least been invited to take an online survey (blame those annoying pop-up surveys).

Internet surveys have grown exponentially compared to other modes of surveys because they offer distinct advantages: they are 1) quick and easy to create and implement, and 2) usually don’t cost a lot of money. The cost of an internet survey ranges from almost nothing (if you have the sample and program the survey yourself) to moderately expensive but still cheaper than a telephone, mail or in-person survey (if you have to buy sample and outsource the development of the online survey).

Internet surveys are especially good in situations where the respondents are known to you and have an interest in the subject, such as employee, membership, or customer surveys. They can be used for hard to find respondents as online sampling firms have established “panels” of respondents whose characteristics are in a database and have pre-emptively agreed to participate in surveys.

Of course, like all research approaches, Internet surveys have their weaknesses. Response rates to internet surveys are typically very low because they are usually very easy to ignore, which means that 1) you’ll need a lot of sample to complete even a small number of surveys, and 2) there is more chance that your survey results are not representative of the larger population you are targeting. In addition, there is the ‘professional survey taker’ phenomenon, where people sign up for many online surveys for the sole purpose of making money. Just check out their rules.

Internet surveys certainly have an important place in the market research toolbox (or is it a shed) – but be aware that they have some tradeoffs compared to other modes of data collection. In addition, online surveys have become much more pervasive in the past few years (how many websites now have surveys that popup when you log onto them?) meaning that they are becoming more of an annoyance as well as more likely to be ignored.

Quantitative Analytical Techniques

2010-07-09T04:43:00.001-05:00

Statistics is a branch of mathematics that deals with the collection, analysis, interpretation, and presentation of masses of numerical data. When a measurement is calculated for an entire population, say the average age, it’s called a parameter. When we look across a sample and calculate a measurement, also the average age, we call it a statistic. Since people make entire careers out of the study of statistics, the point of this post is to present a birds-eye overview and brief description of common terms you’ll hear in conversations about quantitative analysis.

When discussing statistics, researchers usually talk about the data in terms of “variables.” A variable is a characteristic that may assume more than one set of values (age, income, birth place can all have more than one value). A variable can either be nominal, ordinal, interval, or ratio in its scale. Nominal variables are also referred to as categorical variables because they represent categories of responses. The color of a car would be represented by a categorical value (for example, black, red, or silver). Categorical variables have no set order, meaning that a black car is not necessarily any better than a silver car.

The level of satisfaction with one’s car on a 1 to 10 scale is an example an ordinal variable (where a 10 is a better score than a 5 and a 5 is better than a 1). Ordinal variables have a clear, set order, but they still represent categories of responses. Interval and ratio variables are numerical variables whose numbers have direct meaning. The age of a car would be ratio variable because it can be measured precisely and at equal intervals (in hours, years, or decades).

Variables can also be discrete or continuous. Continuous variables, such as time, have an infinite number of possible values, while discrete variables, such as a satisfaction scale, have a finite (in this case 10) number of possible values.

Descriptive statistics are simple portrayals of what the variables show. They are summaries of the frequency of the different values (like percentages); the central tendency (mean, median or mode); and the dispersion (like the range and the standard deviation). Cross tabs (short for tabulations) are popular for displaying the joint distribution of two or more variables. They are usually presented in a matrix called a contingency table. In a cross tab table, each cell gives the number of respondents that gave a particular combination of responses.

Measures of association summarize the relationship between two variables (correlation and regression, for instance). Two variables are associated when information about one can help us predict information about the other. A variety of techniques to measure association are available, each better suited to different classes of variables. When analyzing data, most statisticians use multivariate analysis where the effects of many variables are considered.

Tests of statistical significance are used to determine how sure we can feel about the associations found in the data -- Could it just be chance? Can we infer that the result can be generalized to the study population? Confidence intervals, chi square tests and t-tests are the most common statistics used to indicate the probability of saying that there is a difference between two groups when actually there is none (level of significance).

Measures of association can be used in very sophisticated ways. Conjoint analysis can be used to determine trade-offs customers are willing to make among product or service attributes. In addition to understanding current preferences, this technique allows modeling of the impact of the introduction of new factors on preferences.

Discrete choice analysis models selection of a product or concept with many attributes from a set of products or concepts. In essence, it models how people make decisions in the real world. For example, one could test products with varying combinations of features to assess which consumers prefer. As with conjoint analysis, discrete choice analysis allows modeling of the impact of the introduction of a new product or concept on factors such as market share.

Cluster analysis identifies population segments using groups of variables. This provides information to better understand and communicate with customers, or help you understand your place in the market place. In general, whenever one needs to classify a mass of information into manageable and meaningful results, cluster analysis is a technique of great usefulness.

Discriminant analysis is used to define which variables best differentiate between predefined groups. The key difference is that discriminant analysis relies on previously defined groups whereas cluster analysis uses the data to discover these groups.

Factor analysis finds the underlying construct behind answers to a series of questions. In other words, factor analysis is designed to classify variables. For clients, it simplifies the interpretation of answers to many questions to a few “factors” that seem to drive answers to all questions. It can be used to determine the key factors that drive aspects like satisfaction, image or customer retention. In addition, factor analysis is used when designing surveys. Often complex concepts (like “leadership”) need to be turned into a group of concrete questions in order to query meaningfully.

Regression Analysis (linear, non-linear and logistic) is widely used for forecasting. It compares the effects of one or more variables on another. The objective of regression analysis is to understand the relationship between several independent or predictor variables on a dependent or criterion variable. This allows forecasting or estimation of the change in a dependent variable based on the change in an independent variable.

***

Conducting Focus Groups

2010-05-01T09:11:00.008-05:00

Focus groups have become the go-to industry data collection technique for soliciting and collecting qualitative information. They involve bringing together small groups of people who usually share a commonality (such as people who share a common experience or work in the same industry) to explore open-ended questions. In focus groups, participants discuss a topic or series of topics and are led by a trained moderator who works from a written guide. Focus group research is not designed to produce data for statistical analysis, such as correlations, prediction, projections, estimation.

What are they good for then? Focus groups are often used to better understand how and why people think the way they do on just about any topic. They can be used to develop research hypotheses and inform the development of quantitative research (and sometimes when your topic is not well understood, they are a required start to the survey development process).

Types of Groups

Your run-of-the-mill focus group consists of a group of people sitting around a table and being led in discussion by a moderator. The groups are usually video taped and last from 1-2 hours. Respondents travel to a central location are paid for their time.

A modification to the traditional focus group is the online focus group, in which a moderator leads a similar type of discussion among people over the Internet for 60-90 minutes on average. Special software packages can be used that allow people to answer questions and interact with each other in real time. Often online focus groups can be done more cheaply and faster than in person groups because they eliminate travel, take care of the transcribing/notes taking, and respondents can be paid less of an incentive than in in-person groups. On the downside, you have less control over the quality of an online group since many participants are recruited from those online survey panel lists.

Wait, I Have to Pay People?

Focus group participants are usually paid for their time. How much? Well, the least amount possible while still getting people to show up. If you are doing a group with your customers at a conference, then you can probably get them to do it for free (or at most for some free drinks). For groups of your average person, we often start incentives at $50 per person (or sometimes slightly less if the client is particularly cheap). The incentive increases as the participants become harder to get. By the time you are recruiting really hard to get people like doctors or executives, incentives can easily start to exceed $150 per person.

How Many People Should Be in a Group?

Despite many studies on the subject, there is no golden number of focus group participants (and in fact, the optimal number probably depends on the type and subject of the group). The general consensus within the industry is that optimal number of focus group participants is 6 – 12 (is that general enough for you?)

We often recommend that groups consist of 8 - 12 people. That is enough people for a diversity of opinions, yet everyone has enough time to speak and no one can really avoid participation. With more than 12 people, speaking time becomes constrained and reluctant participants become discouraged from participating. Smaller groups of 5 - 6 are often used for business focus groups or for very technical topics. Mini groups allow each participant more time to speak without the pressure of speaking in front of a larger group. All focus groups, when well managed, have the advantage of using peers to both prompt open participation and to allow snowballing of ideas and points of view.

How Many Groups Should I Do?

The answer is probably not as many as you think. It’s recommended that you do at least two groups to ensure that you are getting consistent information and that there was not something funky going on with one of the groups. Reasons researchers do more than two focus groups include:

1. When you have multiple groups of people with different characteristics (if you wanted to test an idea among teachers, students, and parents, you may do 1-2 focus groups among each of those three targets).
2. When you expect that differences will exist by geographic area
3. If you want to explore a new concept or idea because the first set of groups led to a discovery or a debunking
4. When you’ve got money to burn (or a multi-billion dollar decision riding on the results)

Other than that, it is important to remember that focus groups do not provide data that is representative of a larger population, so there are diminishing returns when you do a ton of focus groups unless large difference exist among the characteristics of the groups being conducted.

How Many Participants Should I Recruit?

I was shocked and appalled to find out that some people no-show a group after confirming that they will be there. But things happen sometimes; we’ve had participants get into car accidents on the way over to our office, get lost and end up in a different town, and show up in the wrong place on the wrong day.

It’s a good idea to recruit at least 2 more people than you need to participate (if you’re group is 8, then recruit 10) expecting 8 people to show up. It’s better to have to turn someone away (with their incentive of course) than to run an 8 person group with the four people who showed up.

What Questions to Ask: Development of a Moderator’s Guide

Before the group begins, a moderator’s guide should be developed. This guide contains the questions that the moderator will use to lead the group. With 6+ people speaking, plus introductions, explanations, etc., a well run focus group will only have time to discuss about 5-6 questions. These questions should be as open ended as possible (since the point is to start a discussion) and move some general concepts to the more specific information you want to know. The worst way to run a group is to try to ask a thousand yes/no type questions. You might as well save some money and do a survey.

The Johari Window

The diagram of the Johari Window (modified from its original form) is a handy way to appreciate the benefits of focus group research. With in-person focus groups, a researcher is able to use an array of tools to access information in all four windows. The art and science of focus groups is not just determining what is on the top of the minds of respondents but what is “beneath the surface” (what they really do and how they think and why this is so). There are a number of scientifically grounded, rigorous techniques for soliciting information in focus groups (such as storytelling). Properly employed, these techniques obtain unusually insightful and deeper information from respondents.

Interpreting and Analyzing the Results of Groups

Once the focus groups have been completed, the fun part begins: condensing hours of recorded and transcribed materials into a usable report or set of research findings. This is usually a multi-step process:

1. During a group, participants can be asked to fill out materials or conduct written exercises that can be used later in the development of reporting materials
2. Immediately after the group (or sometimes even during it), the moderator should write-up a brief summary of the groups along with their immediate interpretations
3. Shortly thereafter, one lucky person should sit down and watch, listen or read transcriptions of the groups and start to put together notes in a question by question format that will use used to develop the bulk of the report. These notes can be supplemented by quotes from the groups to illuminate the findings.
4. Finally, the moderator or another person who attended the groups should pull together the notes into a set of cohesive of conclusions and/or recommendations.

Focus groups are often precursors of sample surveys. The insights and understandings developed from observing patterns in focus group data are often developed into a set of hypotheses (preconceived ideas or theories) about the larger population. These hypotheses can then be confirmed or denied using structured questions and sample surveys. Similarly, focus groups are also used after sample survey results indicate the need for deeper understanding of particular findings.

While focus groups are not representative of the larger population of people in a statistical sense, if the population you would like to make generalizations about is highly specific (say, males who are between the ages of 30 and 50, who graduated from college with a master of fine arts degree, who live with their parents and have in-home painting studios) then you could probably make generalizations from a thoughtfully selected focus group.

Qualitative vs. Quantitative Research

2010-04-30T15:07:00.005-05:00

Discussing research methods can be confusing because what is often called a research method may actually be a data collection technique (such as a focus group) or it may be an analytical technique (such as a correlation study, where data is analyzed to discover relationships between things). Even more confusing are the way the terms “qualitative” and “quantitative” tend to be used in the survey research industry.

What is Qualitative Research?

Qualitative research is used to understand how and why people think the way they do using data collection techniques such as observation, storytelling and in-depth discussion. Qualitative research does not include numerical analysis of responses, at least in any way that can be generalized beyond the group of people you are studying. For example, if I was thinking about starting a service called “Man Maids” where men could be hired to perform chores around the house, I might first sit down with some woman and collect qualitative information about what they think about that service. Using the results of this discussion, I could form hypotheses about the service and its ability to succeed, but would not get any actual numerical information, such as the percentage of women in my area who would use that service.

Qualitative research is a wide field of study that attempt to examine and interpret people’s attitudes, perceptions and beliefs using techniques that probe or explore. In reality, a lot of times the term qualitative research is used when people really mean “focus groups” because it is easy to visualize a bunch of people sitting in a room talking about nonsense. However, the term also applies to other research approaches (such as in-depth interviews, content analysis, and ethnography)

What is Quantitative Research?

Quantitative investigation is analysis designed to determine the amounts or proportions of the components of something being studied. For example, while my discussions with women would allow me to develop hypotheses about my potential service man maid service, I would need to conduct quantitative research to test those hypotheses and measure the level of interest in my potential service area.

Like qualitative research, quantitative survey research attempts to investigate human activity. Unlike it, however, quantitative research does this by measuring how many times certain behaviors take place (buying a product, hiring man maids), or how many times people respond to a question in a specific way. We can then make reasonable estimates, projections and predictions of how a population of people may behave.

***

In the course of any particular research study, a combination of qualitative and quantitative methods can be employed. And, different combinations of data collection and analysis techniques, also referred to as “research methods,” will be utilized.

Coming up will be more detailed descriptions of frequently used research methods. Of course, the type of data you will be collecting (qualitative vs. quantitative) will have an effect on the data collection and analysis techniques you use and the types of interpretation that can be done. Focus groups, for instance, are by design representative only of the participants in the groups. They provide information that can be used to develop hypotheses. These hypotheses would need to be tested using some type of quantitative research in order to confirm whether they are true. Random quantitative surveys are conducted to be representative of a population, but they only if the survey questions, sampling and data collection processes are conducted in a scientific way.

Qualitative Analytical Techniques

2010-07-05T11:36:00.001-05:00

Qualitative Research employs special techniques that allow researchers to observe the ways respondents analyze and synthesize information. When attempting to understand people’s perceptions, beliefs, and behaviors, it is important to solicit information in a way that is meaningful to them. Rigorous sampling and advanced quantitative analysis will not remove bias introduced by a researcher who is subconsciously imposing his/her viewpoint on the study population. The following are a few methods that can be used to collect qualitative data.

Observational techniques are those where a researcher simply observes human behavior or actions first hand. This technique is useful to gain a full understanding of the context in which the behavior is talking place and also when people are unwilling or unable to verbalize the topic being evaluated.

Collecting nonverbal data requires such fineness of observational detail that special training is required to use the terminology and notational systems. Examples of nonverbal study include: Kinesics (observing detail of bodily movement) and Proxemics (social symbolic uses of space). A research study that includes analysis of nonverbal data is usually videotaped for repeated viewing (like focus group tapes).

Listing, Selecting and Sorting – Asking respondents to generate and/or sort lists, and make selections, is a way to explore their taxonomic systems (the way they organize information).

Projective Techniques are based on the understanding that people naturally project their beneath the-surface perceptions, beliefs and personal themes in their verbal responses and behavioral styles. There are many different tests and games to help probe beneath the surface that have been devised and tested over time. Familiar examples include: Sentence Completion Tests, Thematic Apperception Tests, Personalization, and Role Playing.

Interviews or Discussion – Are good at collecting information that can be verbalized easily. Common settings include focus groups and in-depth interviews.

Self-Administered vs. Interviewer-Administered Surveys

2010-07-02T06:02:00.001-05:00

One of the important factors to think about when designing or conducting a survey is who will be filling it out? A self-administered survey is really any survey where the respondent, rather than an interviewer, fills out the questionnaire. Examples of this include:

Intercepting a customer in a store – but having them fill out the questionnaire
Employee feedback surveys where they fill out surveys anonymously
Administering a survey to a group such as students in schools
Consumer product testing on the internet

And the list goes on and on. They can be conducted via a number of different modes, such as online, mail, IVR, or in-person. Self-administration simply means that the respondent (rather than an interviewer) takes on the burden of filling out a survey. This has more consequences than you probably think; some good and some bad.

On the plus side, since interviewer time is not required to complete a self-administered survey, they are usually cheaper to conduct. Surveys can also be completed more quickly when respondents fill out their own surveys rather than waiting for a limited number of interviewers. In addition, respondents are more likely to be truthful and provide information to sensitive topics when they can fill out a survey themselves and are assured of its confidentiality. If you have a survey on a sensitive topic, self-administration is likely the way to go.

On the negative side, self-administration reduces response rates, increases drop-offs and will likely increase non-response bias (because of the lower response rates). Respondents will find it easier to ignore these types of surveys and you will have to send out more surveys to get the same amount of completes compared to an interviewer-administrated survey.

The lack of an interviewer or researcher during a self-administered survey means that questions are more likely to be misinterpreted or an inappropriate answered given. If a respondent does not understand something, they won’t be able to get an explanation during the survey. Therefore, it is important that self-administered surveys are designed clearly and are easy for respondents to understand right from the start. Interviews can provide feedback on a poorly designed survey or difficult to understand questions while the survey is being conducted, but you probably won’t know that respondents can’t understand a self-administered survey until you analyze the data and see that it’s junk.

No probing can be done by an interviewer, so self-administered surveys provide less depth for the information collected. This is fine if you are conducting a simple satisfaction or feedback survey, but might cause a problem if you wanted more detailed information from customers, employees, etc.

Despite the drawbacks, the cost and time advantages to a self-administered survey make them a commonly used method, especially when you have a simple survey to conduct.

Writing Questions for Surveys

2010-05-22T08:46:00.010-05:00

Developing questions is the aspect of research that non-researchers normally feel the most comfortable with. However, it is something that is deceptively difficult to do well and is the place where things are most likely to go wrong in the survey process. People make entire careers out of the study of questionnaire development and there are many books and scholarly articles devoted to the topic. I am clearly not one of these scholars, but I’ve at least seen enough to know all the things I don’t know about developing questions for surveys.

So, what don’t I know?

The goal of a survey question is to gather reliable and consistent information on something you want to measure, such as the number of times people have been to the doctor in the past year. Sounds easy enough. But how people define the word ‘doctor’ is very different, especially as you cross cultures. Are we talking about primary care physicians only? Does a trip to the emergency room count? How about a visit to a public clinic? And what do we mean by year? Do we mean the previous 12 months from this very moment, or the last calendar year of 2009? Subtle differences, but to the extent that survey questions can be interpreted differently by respondents, it means that our question is vague and will produce inconsistent data.

Types of Questions

There are two basic types of questions that make up the average survey: closed-ended and open-end questions.

Closed-ended questions are those that require yes/no or a multiple choice answer, such as ‘do you love puppies?’ Open-ended questions cannot be answered in a simple yes/no response and require a longer and more varied response. Something like: ‘Please tell me the reasons why you hate puppies?’

Generally speaking, closed-ended questions are simple to ask, analyze and interpret. However, they do require you to phrase questions into something that can be answered using multiple choice. That is not always possible or practical. Open-ended questions can capture more detailed information and are useful when you don’t know what types of answers the respondents are going to provide, but they do require more coding and analysis time after the survey is completed.

Biased or Leading Questions

Ok, that question about puppies above is actually a bad survey question, because it is called a leading question. By using the word ‘love’ and phrasing the question in that way, I am pushing respondents to answer a certain way (in this example, to say they love of puppies). If they answer ‘no,’ then they look like a bad person.

Leading questions and biased surveys are something that I commonly run into, especially with the market research crowd (I rarely do political polling since it makes me feel dirty). These surveys seem to be designed to confirm a point of view. Political push polls are the most common example of these types of surveys, but there are other subtle ways that questions can lead respondents – such as loaded questions, biased phrasing of question wording or unbalanced response scales.

The unbalanced scale:

Please tell me your opinion of puppies, using the following scale of 1-5:

1. I love them more than life itself
2. I love them a lot
3. I love them
4. I like them

The results of my survey show that 100% of people like puppies.

Additionally, the order that you put your questions in can affect how respondents answer them, that is, one question can bias a later question. If you ask a respondent to rate a series of factors on importance, and then the respondent is asked an open-ended question on what is important, their comments will largely be limited to the factors already mentioned. Ask the open-ended question first, and a greater diversity of factors are likely to be mentioned.

Check out StatPac for a good list of more subtle ways to bias survey questions.

Ask Actionable Questions

That question about puppies above is bad for another reason too: It’s not going to give me any information that is actionable. So what if everyone loves puppies? What can I do with that data?

A mistake that people often make is asking questions simply to ask them because they think they should or because they have “space” in their survey. Make sure that every question in a survey serves a purpose and that you will use the information that comes out of it. The best way to do this is to clearly define the goals of your survey. What are your hypotheses? What are the things you want your survey to answer? Make survey every survey question builds towards those goals.

The Lowest Common Denominator

Maybe that sounds a little mean, but it is important to assure that your survey questions can be understood all respondents, not just those with advanced degrees. At minimum, questions need to be put in plain English (think 6th-8th grade reading level), and organizational jargon needs to be stripped away. If you conduct research on short term disability policies, your internal definition of STD is probably a little different than the average person. It’s usually a good idea to let someone outside your industry or area of expertise read your survey to make sure it can be understood by the average person.

For very sophisticated research, cognitive testing can be done to be sure that each question is understood by the respondent as intended. Most studies don’t have a budget to allow this, so you'll need to rely on past research about what respondents have found confusing in the past. And if there is no past research, well, my best advice is to keep it simple.

Response Scales

There is the issue of what are the appropriate question response scales to use (another topic people have written entire books about). Scales can be either even or odd. Even scales allow no middle choice. Research suggests that an uneven scale is better because it allows a respondent to indicate that they are in the middle and could go either way.

Scales can also be anchored or unanchored. An anchored scale has words attached to it, such as "very satisfied" or "somewhat satisfied", and they are preferred because they add clarity on the meaning of the divisions on scales. Alternatively, an unanchored scale might be worded, "on a scale of 1-10 where 10 is very satisfied..." More divisions allow finer distinctions in responses but they also make wording impractical.

Should you label every point in your scale? Generally, yes if possible, as labeling every point increases understanding of your scale and leads to more consistent responses.

How many distinct points should you have in your scale? Five will usually do for most questions, but if you believe that your data is going to be skewed either negatively or positively, then a seven or ten point scale will produce a wider range of answers.

The Final Word

Now that you’ve crafted all your survey questions and given them appropriate response scales, you get to worry about the order to put them in. The order of questions in a survey can be important. If you ask a question early in a survey, the responses will be different than the same question asked later, after a respondent had time to think about a topic. Later responses are more likely to provide more depth than earlier ones

Thinking about the research objectives and the specific informational needs, an experienced researcher will craft questions in different ways to completely surround a topic. For example, you might ask the respondent to name something top of mind, and then ask about familiarity with similar items, and then ask for rankings of these items on particular characteristics. You might also include an open-ended question on why one something is different than another.

***

So survey development is easy, right? For more information on designing questions for surveys, I'd reccommend the following books:

Introduction to Intercepts, In-person Interviews and CAPI

2010-06-28T08:14:00.000-05:00

Intercepts, In-person Interviews and CAPI (Computer Assisted Personal Interviews) are conducted by interviewers going to locations where respondents are apt to be found, and requesting their participation. They are commonly used to gather data from respondents that would be difficult to find by any other method. Common examples include customers of specific restaurants and stores, or a businessperson in a trade show. Intercepts are often conducted in malls where it is easy to find potential respondents and easy to visually screen potential respondents by age or other characteristics.

CAPI or manually collected intercept surveys can often be fielded quickly. It is relatively easy to gather national data using the large number of available field service locations. Since intercepts are in person, almost anything can be tested: visual communications, video, and even food taste and texture. However, people are generally less patient in person than by other modes of surveying, so intercept surveys are generally shorter than other types of surveys and the amount of information you collect is less.

For more complex information gathering, it is common to screen a respondent then invite him/her to an adjacent facility for more precise interviewing. Intercepts can be completed on either paper forms that are data entered at a later time or with laptop/handheld computers.

Go to the Respondents

If you want to know what customers think about a specific location, you can randomly select a sample and screen people based on if they have visited that location. But a much easier way to conduct the research is to go to the location and survey respondents there. This is especially true when you are dealing with difficult to find respondents or a small number of people.

The Problems with in-person Interviews

When surveys are conducted out in the field, it is more difficult to monitor and control the quality of interviewing. So it is important to verify the quality of the data that is collected. Costs are also a consideration. The cost of project management, on-site interviewing and respondent incentives are generally higher than for other methods, making this approach relatively costly.

Finally, this approach can be intrusive since respondents are interrupted and asked to participate. Generally, people don’t like to be solicited in person while they go about their business. So, this type of data collection must be done carefully, especially if your company’s name is associated with the survey.

***

If would like to conduct in-person interviews, but don’t have the capacity to send interviews around to different locations, there are plenty of companies around the country who specialize in this type of data collection. Every major (and probably minor) city in the US will have a facility who can collect this type of data for your company. Many have multiple locations in different cities if you need some sort of national representation.

Home Project 9: Finishing Up the Kitchen

2008-09-14T04:30:00.002-05:00

We are finished in the kitchen except for one minor thing, actually getting the refrigerator to fit into the space that we have for it. From our kitchen floor to the bottom of the cabinets is 64.5 inches (which is ridiculously small by today’s mcmansion standards). After much pain and suffering, we found a fridge that was exactly (and I mean exactly) to those specifications.

However, when the fridge arrived, we found out that our definition of 64.5 inches was apparently different than the manufacturer’s definition of 64.5 inches, as back of the fridge was about ½ inch too tall to slide underneath the cabinets. Maybe that’s how all fridge measurements work, I don’t know. That was the first fridge that I’ve had to order. And based on the experience, I wouldn’t mind if it was my last.

At that point we could have 1) destroyed the cabinet about the fridge, 2) returned the fridge, or 3) carefully cut ½ inch from the bottom of the cabinet without destroying the entire thing.

Time to cut some cabinets...

I’d advise not being a douche like me and actually taking the dishes out of your cabinet before you cut into it. At the very least, it will prevent a lot of annoying rattling.

Fortunately for us, our cabinets have a piece of wood below that hangs about an inch below the cabinet door (referred to as a ‘rail’ if my cabinetry vernacular is correct). So, the good news is that we could cut into this wood overhang without having to cut into the door or otherwise destroy the cabinet. Using my trusty sawzall, I cut the wobbliest straight line in the history of straight lines. A pre-schooler would have been embarrassed by this line I cut. But while I got no points for style, it worked and we were able to get just enough additional clearance to slide the fridge into its spot.

Like a glove

So, with that that, the kitchen is basically done: new appliances, new sink and countertops, new lighting, and everything repainted to look new(er). Here is what the kitchen looks like now:

And here is where we started:

Introduction to Mail Surveys

2010-06-08T20:15:00.001-05:00

Mail surveys can take a wide variety of forms; although they commonly consist of a paper survey that is completed by a respondent and mailed back. You will see these with various levels of quality and professionalism with product warranties, for customer service feedback, and in your mail. They are still popular because they can be very simple to construct and can often provide data at low cost.

Mail surveys are a good choice if the respondent is someone you know and/or they have an interest in the survey results, such as an existing customer or employee, or member of an organization. We also recommend the use of mail surveys where it is important that every member of the sample get the impression that their voice is important. A mail survey can also be used as a relationship building device, demonstrating that you are interest someone’s feedback

On the positive side, it can be easy to collect a large amount of data from mail surveys (although this depends on the exact methodology) - simply mail out a ton of surveys. They are not intrusive (as respondents are not interrupted at an inconvenient time) and they can respond when they want. And because mail is an archaic technology, you can reach nearly everyone you’ll want to survey with a mail or paper survey.

The biggest disadvantage of this type of survey is time, especially if you are mailing it both there and back. While surveys on the phone and internet can be completed in days, it usually takes two weeks at minimum to complete a mail survey (you’re looking at a week in post office transit alone). For this reason, mail surveys are often used to collect data on an ongoing basis for customer information like for product warranties, rather than for a quick snapshot type survey. You probably wouldn't want to do your political poll by mail, for example.

Design of the Surveys

Unlike surveys conducted in person and by telephone, mail surveys don’t have an interviewer who can provide clarification or answer questions, so it is important to carefully consider the design of your survey and to troubleshoot problems that can lead to reduced question comprehension and errors in responses.

Avoid compressing and compacting questions. Too many questions on a page create confusion and can result in response errors. Survey testing shows that respondents will readily fill out a survey several pages in length provided they feel the survey is important.

Obtaining an Adequate Response Rate

Unless the survey is conducted among a highly engaged group (say employees of a company), we find that multiple mailings are almost always necessary to assure an adequate response. Our experience has shown that a first mailing of the survey, a reminder postcard, and a second mailing of the survey to non-respondents is the optimum approach and will result in a response rate that is 10-25% higher than a single mailing.

The Final Word

There is nothing sexy about conduct research via paper surveys or through the mail. But there is a reason they are still around and why you see them everywhere if you look carefully. They are a relatively inexpensive way to collect information at the respondents’ convenience, especially when you don’t have access to email addresses or not everyone you are surveying has easy access to the internet.

The Ministry of the Treasury of the Republic of America Has Sent You an Email

2008-08-24T11:24:00.000-05:00

The Ministry of the Treasury of the Republic of America needs your help with a large transfer of funds. 800 billion dollars US. Your help is urgently needed.

Home Project 8: Where Best Buy Loses a Refrigerator

2008-09-04T14:37:00.000-05:00

With the kitchen nearly complete - no really, almost done – really, I mean it - the only thing left to upgrade were the kitchen appliances. Since I had the gall to force my wife to purchase a house without a dishwasher and little space to put in a dishwasher (other than the possible exception of the bathroom) the only appliances we needed to upgrade were the stove and refrigerator, which was perfect for a cheap bastard like myself.

But, there was one small problem. Literally, a small problem. As I may have mentioned, our house was built by a band of traveling French-Canadian dwarfs, and as a result, they left a space exactly 64.5 inches high between the floor and the bottom of the cabinets above the fridge.

Specifically, the space between the floor and the cabinets where the fridge would go was 64.5 inches high. For comparison, the average refrigerator sold in the US is about 70 inches tall and the smallest full sized refrigerator that stores typically sell is in the range of 66-67 inches. There are plenty of small sized (16 cubic feet or less) fridges that are under 64.5 inches tall, which would be fine for us if we didn’t actually cook, buy groceries or actually want to use it for anything other than a beer fridge.

The old fridge, hiding its shame behind pictures

That did not leave us many options other than a kegerator or possibly a giant block of ice. But after a good amount of searching, we found a 21 cubic foot fridge that was exactly 64.5 inches tall for sale on the Best Buy on their website. Not surprisingly, it was a special order product since most people aren’t in the market for ridiculously-sized refrigerators.

I ordered the fridge through the Best Buy website and received word that it would take a few weeks to ship. That was fine by me since it was an unusual order. My credit card got charged immediately, of course, but a few weeks passed and I did not hear anything further from Best Buy. I called their online customer service number and got someone in India (ugh). There was exactly one thing the Indian customer service rep could tell me: my online order was processed.

Thanks captain obvious. My credit card statement already told me that one. Unfortunately, the rep didn’t know where the stove was or when I could expect it to be delivered, or even how to go about getting that information. You’ve probably heard the phrase “it was like talking to a brick wall” but you don’t really know what that’s like until you speak with a Best Buy customer service rep in India. I asked to speak with the rep’s supervisor and the only additional thing she could tell me was to contact my local Best Buy store for additional help with the problem.

That advice sounded strange on its face, but I took the supervisor for her word and ventured to the local brick and mortar Best Buy. There I spoke with the customer service department and quickly discovered that, yes, the customer rep from the phone had indeed given me stupid advice. The local Best Buy store didn’t know anything about my Internet order and to resolve my problem I needed to call the customer service number provided on my fridge invoice. Now it was official: I’m getting the runaround.

So, I decided to write a rather scathing email to Best Buy using their online help thing. To their credit, I received a follow-up call a few days later from (I’m guessing) the manager of the local distribution center, telling me that they had my fridge and setting up a time when it could be delivered.

A day before the delivery date, I got another call from the distribution center, saying there was some kind of mix-up and they didn’t actually have the fridge I had ordered. It was going to be another few weeks before they could get it.

“Great, cancel my order”

A couple months had passed now since I originally placed the online order for the fridge and the only thing we had to show for it was a lot of wasted time. The stove we had purchased at the same time from Best Buy had already been delivered weeks ago without any problems.

So, fridge-less, my wife and I went down to our local Sears store and – while looking through their internal catalog – found a fridge similar to the one we had tried to purchase at Best Buy. We ordered the fridge and it came in a couple weeks later without any problem. Well, actually there was one problem. The door handle was on the wrong side. But fortunately, Sears just sent a replacement door with the handle on the correct side and had one of their technicians switch out the door for us.

So, the result: A couple low-end stainless steel appliances, a little over 1k overall. The stove was ordered and delivered in less than one month, but Best Buy screwed the pooch on the fridge, gave us the runaround until we canceled the order and found a similar model at Sears. Overall timeframe to get the fridge: about 3 months.

The next project: Actually getting the fridge to fit (it’s far from over).

***

You might be asking yourself, why didn’t we just knock down the cabinets above the old fridge, creating enough space to install a normal-sized refrigerator? Yes, hindsight IS 20/20. If I had known all the trouble we would end up going through to get this damn fridge, I would have definitely done that. At the time, however, we were just being strung along one day at a time.

Introduction to Telephone Surveys

2010-05-31T20:54:00.001-05:00

Despite all the potential and hype of the internet, telephone surveys are still one of the most common methods of survey research. Since being introduced in 1936, telephone survey methodology has established rigorous protocols and standards, which dictate the proper way to conduct these surveys. If these standards are carefully followed, administering a telephone survey to a random sample of a population provides data that can be generalized to the whole population.

Telephone surveys are still a common used method for public opinion polling and other research into the perceptions, attitudes or behaviors of the general public. It is also the method employed by the US government to monitor the health of the American population (the CDC’s Behavioral Risk Factor Surveillance Survey). With telephone surveys, researchers can control the location of the respondent, and “live” interviewing provides the option to probe and clarify responses. Telephone surveys are also common means of conducting business-to-business research. A “live” interviewer can readily confirm that the business respondent is properly qualified (or ask to speak to a respondent that is) and can persuade a reluctant participant to complete a survey. Other passive methods such as mail or internet surveys are not as effective at gaining participation from this group.

People have predicting the death of telephone surveys for the greater part of the decade. Response rates for telephone surveys have decreased dramatically over the past 25 years as phone technology (like caller ID) has allowed people to screen their calls. In addition, the National Do Not Call List has changed many people’s perceptions about receiving unsolicited calls on the telephone. Add in the fact that the number of households without a traditional landline has reached about a quarter of the population in the U.S., and it’s pretty clear why some people feel like the telephone survey is a dinosaur and the internet is a giant meteor.

But I doubt the telephone survey will disappear anytime soon. While the addition of cell-only households has increased the cost and complexity of telephone surveys, cell-only samples can be added as a component to a traditional phone survey. And while response rates are on the decline for phone surveys, internet survey response rates are downright atrocious in most cases.

So phone surveys still serve their purpose as a cost competitive way to do randomly sampled surveys of the general population and surveys where a list of email addresses is not easily obtained.

Pros of Telephone Survey

Can yield information that is representative of a whole population with known probabilities of sample selection
Allows probing and clarifying by interviewers to improve the content of the information gathered
Nearly all people can be reached by telephone (especially if you include cell phones in your sampling frame).
Despite their decline, response rates are usually higher for phone surveys than other types of surveys
Interviewers can convince reluctant participants to complete a survey

Cons of Telephone Survey

Typically more costly than mail or internet surveys
Response rates and landline coverage on the decline
Complex ideas or new concepts are difficult to explore over the phone
Visual materials cannot be studied (one approach is to mail/email these prior to an interview)
Respondents may answer questions on the phone to another person in a way that is socially desirable rather than truthful (especially with sensitive questions).
Accuracy of data is subject to the quality of the interviewer (unless you are surveying using IVR

Sampling for Quantitative Surveys

2010-05-15T09:44:00.003-05:00

Sampling is the process of selecting units (these units are often, but not always, individual people) from a population of interest in a way that allows us to study a smaller group, but generalize our findings back to a larger population.

Of course there are times when you don’t care about generalizing back to a larger group. If you are evaluating how satisfied your employees are, there is no need to select a sample of employees (unless you happen to be GE or something). Instead, it is easier to survey all your employees. This is called a census.

However, unless the group of people you are interesting in surveying is relatively small (in the thousands or less), then it makes financial sense to select a sample of your group, conduct a survey with those people, and generalize those results back to the larger group. In most market and social research, we are interested in generalizing to specific groups. The group you wish to generalize to is referred to as the study population and your sample will come from it.

Once you have identified your study population, you have to get a list of all the members that are accessible, and this list becomes your sampling frame. Finally, you actually draw your sample (using one of the many sampling procedures). The sample is the group of people whom you select to be in your study.

Note that the sample is not the group of people who are actually complete the study. You may not be able to find all of the people you actually sample, or some could drop out over the course of the study. Unless you can get every single person in your sample to respond, the group that actually completes your study will be a portion of the sample (the people who do not complete the survey are often referred to as non-respondents or dropouts).

Sampling is a complex, multi-step process, with lots of opportunities to go wrong. There is the possibility of introducing bias when going through the process of identifying a sample. For instance, you may be able to clearly identify the population of interest, but you probably will not have access to all of them. There are opportunities for error when drawing the sample from the sampling frame. And, of those in the sample, some probably will not fully participate (drop-outs and non-responses).

Sometimes, these problems can be corrected by using weighting. Though statistically complicated, weighting allows the researcher to adjust research results based on non – response, sampling, data collection processes and population characteristics.

Types of Sampling Approaches

Broadly speaking, there are two types of sampling: probability sampling and non-probability sampling. A probability sampling method is sampling that uses random selection, a process that assures that the different units in the study population have equal and known probabilities of being chosen for the sample. A probability sample is required when the objective is to generalize results to a population rather than just those who responded to the survey.

There are a number of sampling methods in this category and each is appropriate in different circumstances.

Simple random sampling (like drawing names from a hat) is easy to accomplish and explain to others, “Everyone has the same chance to be selected”. It is a fair way to select a sample so you can generalize the results from the sample back to the population. It is not the most statistically efficient method of sampling because by the luck of the draw, you may not get good representation of population subgroups.

Stratified random sampling remedies this problem by dividing the study population into homogeneous subgroups (strata) and then taking a simple random sample in each subgroup. Stratified sampling has a couple advantages over simple random sampling. First, it assures representation of subgroups of the population, particularly subgroups based on geography (for example, counties or cities). In fact, if you want to be able to talk about subgroups, this may be the only way to go. Second, if the strata are homogeneous, then this method usually is more precise statistically. The reason is that the variability within groups is usually lower than the variability in the whole population.

Probability proportional to size sampling uses a size measurement (such as the number of employees in a company or the number of students in a school) to assist in the sampling process. Using this method, each item (in this case a business or a school) is assigned a sampling probability in proportion to its size. So a company with twice as many employees as another would have twice the probability to be sampled. This method can increase the representativeness of the sample by focusing it on the larger members of your sampling frame, where more business is conducted.

Cluster or area random sampling solves for administrative efficiency problems when sampling a population that is spread out geographically. The steps in cluster sampling are: divide the population into clusters (sometimes along geographic boundaries), randomly sample clusters, then measure all units within sampled clusters.

Multi-stage sampling is when sampling methods are combined in useful ways to address sampling needs in the most efficient and effective manner possible. Most real marketing and social research uses complex sampling strategies that combine aspects of the ones described above.

Non-probability sampling is sampling that does not involve random selection. Non-probability samples may or may not represent the population well, and there is no sure way to tell. In general, researchers prefer probabilistic or random sampling methods and consider them to be more accurate and rigorous. However, in market and social research, there may be circumstances where it is not feasible, practical or theoretically sensible to do random sampling. For example, it may be impossible to obtain or compile a complete list of the group you want to survey, making it impossible to pull a random sample.

There are two broad types of non-probability sampling: accidental and purposive.

Convenience sampling is another name for accidental sampling. This type of sampling uses whoever is readily available or convenient to the researcher.

Purposive sampling is sampling with a purpose in mind and usually targets specific, predefined groups. Intercepts are generally this type of sampling. A researcher goes to a location and then observes the people passing by to see who appears to be in the target category. Once identified, the researcher will stop him/her to request participation. If the person agrees, the first thing the researcher must do is verify that the respondent meets the criteria for being in the sample.

Purposive sampling can be very useful for situations where you need to reach a targeted sample quickly and where sampling for proportionality is not a concern. With a purposive sample, you are likely to get the opinions of your target population, but you are also likely to over represent subgroups in your population that are more readily accessible.

There are a number of purposive sampling techniques, such as: modal instance sampling (targeting the “typical case”); heterogeneity sampling (diversity sampling, aiming to solicit the full range of possibilities); expert sampling (targeting people with known expertise in some area); quota sampling (targeting people according to some fixed quota); and snowball sampling (identifying someone who meets the criteria and then asking him/her to recommend others). Despite their limited ability to generate population generalizations, each of these methods may make sense under certain circumstances.

In-Depth/Qualitative Interviews

2010-05-13T19:11:00.003-05:00

In-Depth or Qualitative Interviews is a data collection method where individuals are interviewed in a one-on-one setting. These individuals typically 1) have some highly informed perspectives on the subject of study, 2) are interviewed separately instead of in a group and 3) are most often interviewed in person or on the telephone.

Like focus groups, qualitative interviewing is not for collecting information that is representative of the larger population, but rather understanding how and why people think the way they do about a topic.

When conducting qualitative interviews, questions are predominately open-ended rather than closed ended, and are best used for exploring issues or understanding the thinking behind attitudes, perceptions and behaviors. As compared with focus groups, one-on-one interviews eliminate any bias that might be introduced from one respondent to another. They also allow more time for each respondent to talk since there is no competition for airtime. And, they are efficient since it is easier to keep one individual on track than it is a group of people.

Why do one-on-one interviews rather than focus groups?

One-on-one interviews are often used in lieu of focus groups where it is not practical to gather people in one place. For example, it may not be possible to gather a group of doctors together for a focus group. Rather, you have to let the respondents be interviewed at a time and place of their choosing.

This is done by recruiting participants to a central location (at a time that works for them), meeting respondents at a location that is appropriate for them (or a location they are likely to be found), or in some cases, interviewed over the phone. As with focus groups, they are especially effective when things must be seen or touched, etc. in order to be evaluated. Testing products and communications materials are often done with this method.

When conducted by an interviewer that is skilled in asking open-ended questions and able to build rapport quickly, they can provide information that is difficult to obtain by any other means.

The Most Important Part of a Qualitative Interview: The Interviewer

Unlike your average survey interviewer, those who conduct qualitative interviews have to perform many of the same functions as a focus group moderator. Interviewers must be able to quickly build rapport with the respondent, engage them in a discussion, and still ask the desired questions.

Obviously, this requires a more highly skilled (and compensated) interviewer. Qualitative interviews are highly dependent on the skill and training of the interviewer to develop a conversation and to avoid biasing the answers. A skillful interviewer can skillfully probe respondents while in a discussion to get incredibly in-depth and revealing information.

On the flip side, qualitative interviews are time consuming to conduct and strenuous for the interviewers. In addition, analysis and reporting are very time-consuming. Interviews are often recorded and transcribed and copious notes are taken by interviewer. The result is an analysis and reporting process that looks very much like the one used for focus groups, which is time consuming but can produce very detailed and informative information.

Money, Money, Money

Of course, all this one-on-one time gets expensive. Instead of gathering a group of people together in a room for 2 hours, qualitative interviews require that an interviewer sit down with individual respondents for 20, 30, even 60 minutes at a time to conduct a survey. Like focus groups, respondents are often paid an incentive for their time. And analysis and reporting of qualitative interviews requires someone to comb through hours of recordings, transcriptions or notes. Be prepared for sticker shock when you get a quote for qualitative interviews.

Executive Interviews

Executive Interviews are simply one-on-one interviews conducted with senior level respondents, such as business owners. This is the best way to interview senior decision-makers and important stakeholders.

Since these respondents are generally very busy, it is nearly impossible to bring them together in a focus group and often they will not spend the time on a paper or telephone survey. Visiting them on their terms and at a time that is convenient to them is often the best way to get their input. All of the advantages and disadvantages of one-on-one interviews apply. In addition, it is critical that interviewers not just be experienced and effectively trained, but they must be able to relate to senior level executives as peers. This is especially important since the interviewer will in some ways be representing the client. Since these interviewers must be senior themselves, and since they must travel to conduct the interviews, fees reflect these additional costs.

Online Focus Groups

2010-05-08T06:42:00.000-05:00

Online focus groups are an interesting alternative to in person groups. There are two basic forms: real time Internet groups and ongoing chat rooms/bulletin boards. Compared with Internet chat rooms, a real time Internet group is more like it’s in person counterpart. Respondents log onto an Internet site and type their responses to a question posed by the moderator and to comments made by participants. Participants can see the comments of all participants and the moderator. The moderator guides the discussion by probing and moving to new topics. In addition, the moderator can communicate individually with respondents if the need arises – to encourage participation, for example.

Real time groups are generally shorter than in person groups – 60 - 90 minutes as respondents fatigue more quickly. They are limited to the same amount of participants (6-10) but can actually move along at a slower pace than traditional in person groups because people can generally talk faster than they can type. As a result, you will collect less information in a group online than you will in a traditional focus group.

Internet groups can be conducted more quickly than in person groups – as travel time is eliminated, there is less geographic concern when recruiting participants and a greater range of times are available to the researcher since participation is generally at home. They are a great way to conduct groups with the technically savvy, with hard to find respondents, or with younger respondents who are more comfortable communicating online.

Ongoing chat rooms are much like other common chat rooms. These begin with a moderator posing questions and respondents answering, as well as responding to one another’s comments at their leisure. Such groups can go on over the course of a week or more, with the moderator adding questions or redirecting discussion as needed.

One advantage of either approach is that a transcript of the dialog is available immediately after the session. Also, respondents can be less inhibited than with face-to-face groups, and therefore express deeper feelings.

If you are looking to check out online focus group software for yourself, Itracks is a common used one.

How Many People Do I Need to Survey?

2010-04-22T21:14:00.008-05:00

One of the first questions that usual comes up when you are planning to conduct a survey is, how many surveys do I need to do to get statistically valid results? If you’ll allow me to channel Bill Clinton for a second, the answer really depends on what your definition of ‘statistically valid’ is. While there is no universal rule for what constitutes statistical validity or how many surveys you need to do to get ‘valid results,’ typically survey researchers will recommend a baseline of 400 completed interviews. Why 400? Well, it’s a nice round number that will provide an overall margin of error of +/-5% at 95% confidence*. People like round numbers.

In reality, the number of people that you should survey depends on a number of factors related to your target population and the way you’d like to analyze your survey results. Here are the questions you need to answer before deciding how big of a sample you need for your survey.

1. What is the size of the target population: Who is the group of people that you are surveying and how large is this group (e.g., customers, your employees, men age 18-24 still living at home with their parents)? The fewer people in your group (i.e., your target population), the fewer surveys you’ll usually need to do. Why? If you survey 5% or more of the population, then you will need to adjust your sampling error by something called a finite population correction factor.

The math behind this adjustment isn’t important, but the table below shows how many surveys you’d have to do to achieve a margin of error of +/-5% at 95% confidence at different population sizes:

Sample size needed to achieve a margin of error the same as 400 surveys at different population sizes

Population Size	Sample Size
1,000,000	400
8000	381
4000	364
2000	334
1000	286
500	223

2. Will you be segmenting your results: 400 completed surveys will give you a margin or error of +/-5% OVERALL, but what if you want to break out your analysis by demographic factors, such as region, age, gender or income? If the results of your survey rely heavily on specific group segments, then it is a good idea to make sure you do enough surveys within each segment of the population to get generalizable results by those groups.**

Wait, how many surveys do you have to do to generalizable results by segment? Well, it’s a bit like the chicken and the egg situation; it depends on what you will be using the data for. A rule of thumb researchers usually use is to conduct at least 100 surveys per segment. Again, 100 surveys is a nice round number which produces a nice round margin of error of +/-10% at 95% confidence. That, and people tend to freak out when you tell them they need to complete 400 surveys per segment.

3. What is the pain to you or your organization if the survey results are incorrect: Let’s be honest, not all decisions carry the same importance (and yes there is such a thing as a stupid question). So as the financial, strategic, or social importance of your decision rises, so does the need to ensure that your survey results are accurate.

The go-to gold standard margin of error of +/-5% at 95% confidence means that if 50% of respondents indicated something in a survey, then 19 out of 20 times (your 95% confidence level), between 45% and 55% of the overall population would provide the same response.

Unfortunately, there are still a lot of things that can go wrong even if the survey data is collected correctly. If the difference between 45% of your customers saying something and 55% saying something is a make or break deal for your organization, then you need to do more surveys and shoot for a smaller margin of error. If the chance of getting an erroneous survey measurement 1 time out of 20 is still too risky for the type of decision you will be making based on the data, then maybe you should up your confidence level to 99%.

The reverse is true as well. If you are going to be measuring the same questions multiple times in different surveys, then you can do fewer surveys and still protect yourself from the one bad survey result. Sometimes you are simply looking to see if customers have any interest in a product, service or concept. In that case, doing 100 surveys would like be enough as it would provide a margin or error of +/-10% at 95% confidence.

The following table displays how many surveys need to be done to achieve a specific margin of error at different confidence levels:

Sample Size needed to achieve a specific margin of error at different confidence levels

Margin of error	80%	90%	95%	99%
+-1%	4,109	6,765	9,604	16,589
+-2%	1,027	1,691	2,401	4,147
+-3%	457	752	1,067	1,843
+-4%	257	423	600	1,037
+-5%	164	271	384	664
+-10%	41	68	96	166
+-15%	18	30	43	74
+-20%	10	17	24	41

If the table is not specific enough for your needs, you can always calculate the number of surveys you need to do yourself using this general formula in Excel:

“=(CI*CI)*(p*(1-p))/(MOE*MOE)”

where

CI = required confidence level (where the standard score of 1.96 is used for a 95% confidence interval)
p = estimated proportion of response (use .5 for the most conservative result)
m = margin of error (5%, or 0.05, for example)

Ok, maybe it’s not super easy. If math isn’t your thing, then you can use an online sample size calculator to identify what sample size you need based on the population size, margin of error and confidence level. Just make sure to think about the answer those questions listed above first.

---

* Technically, 384 surveys produces an overall margin of error of almost exactly +/-5.0%. However, as I mentioned, researchers like nice round numbers and will therefore usually round that number up to 400.

** The need to segment or split out your survey results inevitability brings us down the road to quotas and stratified sampling, but that is a conversation for another day.

The Survey Research Process: How It Should Work

2010-04-15T15:18:00.003-05:00

When you break it down to its most basic element, the goal of any survey research project is to provide useful and accurate data which can then be used to inform a decision or answer a question.

Despite data and hypotheses, a research project should really begin as a creative process, conducted among multiple individuals, each with different backgrounds and sharing different expertise. If we could simplify reality, every research project would progress smoothly through a step-by-step process. These steps are the stages that follow.

1. Identify and articulate the research hypothesis, that is, what you want to know. You’re the expert here, and by partnering with an experienced research firm, the “unknown” can be turned into a research question or objective.

2. Identify the most appropriate research strategy or approach. There is always more than one way to approach a research objective. Different approaches have unique uses, as well as different timeframes and associated costs. An experienced research firm will listen to you and suggest options to address your needs.

3. Develop the data collection tool or instrument. Survey research entails systematically gathering observations or data. You may want to count how many times individuals chose one product over another or how many times individuals answer questions with a certain response. No matter what you are observing, the data must be collected in a manner that allows it to be analyzed and synthesized. This phase provides an opportunity to revisit and confirm that you are asking appropriate questions to answer your research objectives.

4. Collect the data. Data can be collected a number of ways, such as in person interviews, focus groups and different modes of surveys. The method selected will be closely connected to both the research approach and the types of people you are surveying. Ultimately, the quality of research is dependent on the quality of the data collected.

5. Process and analyze the data. Once collected, data is usually entered into research software, cleaned and coded. Then, the data can be analyzed, or separated into its component parts and relationships.

6. Interpret the data and produce a report. After statistical analysis, the data is synthesized and then summarized into a report, including a summary of the key findings, a set of more detailed findings that allow you to independently evaluate the data yourself and a complete section on methodology (how the research was completed).

7. Present findings. Experienced researchers appreciate the opportunity to present results and discuss the meaning of those results. By combining the recommendations of researchers with your own experience, you will get the most value from the research. These discussions are also great ways to head off misinterpretation of the data.

Why is this Survey Company Harassing Me and How Do I Stop It?

2010-04-09T14:47:00.001-05:00

Being harassed daily on the telephone by people who want you to take a survey? Not interested in taking it and want to get them off your back? Well, there are a couple things you can do to stop these calls.

Wait, I’m on the National Do Not Call List, You Can’t Call Me! Much to your chagrin, yes you still can get called for a whole bunch of reasons even if you are on the do not call list. Established in 2003, the Do Not Call Implementation Act made it illegal for telemarketers to call consumers with whom they did not have a prior business relationship. There are a number of exemptions under this rule, however, including political calls and calls to conduct surveys. Why? Because these types of calls do not involve sales. Well, at least in theory they don’t.

So, generally it’s a waste of time to try to convince the telephone interviewer that you shouldn’t be called because you’re on the do not call list. Also, I’d avoid threatening to sue the interviewer or the survey company, threatening to report the company to the attorney general, or threatening physical violence on the interviewer or their children. They’ve heard all of that before.

So, what you can do to stop being called for surveys

1. Screen Calls – The preferred method for most people of avoiding telemarkers, surveys and “friends” is to simply screen your calls, which is made easier nowadays thanks to caller ID. If the daily and repeated calls don’t bother you and you can live with phone ringing, then more power to you.

2. Call Blocking – Another option is to block or screen unwanted calls, typically with the help of services provided by your phone company. Note that these are not fool proof methods and usually require an additional fee, but they should effectively block many unwanted calls.

3. Just Refuse - The goal of a telephone interviewer is to complete a survey obviously, however, a secondary goal is simply to identify whether a telephone number contains an eligible and willing respondent or not. When a telephone interviewer calls a number and gets a voice mail, answering machine, or no answer at all, the number typically gets flagged as an unresolved case. Industry standards say that a telephone number can be called up to 10 times in an attempt to “resolve” it, which is the reason for the repeated calls every day. So, if you don’t want to do a survey, the easiest thing to do is answer the phone and just say no. Bam, you have just resolved your number and (provided the organization is legitimate) it should be taken out of their database.

4. Never Call Me Back - most survey organizations keep their own personal do not call lists, so if you are the type of person who never does surveys, ask to be put on the organizations do call list. It won’t stop you from receiving calls from other companies, but it’s a start. Many of the surveys within a state will come from small group of organizations who do a lot of survey work in that state, so getting on those lists will help.

The U.S. Census and Race

2010-04-05T14:46:00.008-05:00

As a survey researcher, I’ve had a mild-to-lukewarm interest in following the goings-on of the 2010 U.S. Census, which is currently in full swing as I’m sure you know from their delightful television commercials. You’d think everything about the process of collecting basic household information from every American currently alive would be a tedious and utterly boring process...

And for the most part you’d be correct. However, in the Census Bureau’s attempt to count 300 million people, they have certainly managed to offend more than a few. First, a few weeks ago there was the outrage about the Census including the word “Negro” along with “Black or African American” on the survey form. Although use of the word Negro was once accepted practice (before the automobile was invented), it rapidly fell out of favor thanks to the Civil Rights’ movement in the 1960’s and is almost laughably antiquated in the year 2010.

To many African Americans, including the word Negro on the Census form stirs up memories of segregation and racism. In a press release, the Census has said that it included the word “Negro” because many older African Americans still identify themselves as such and noted that "In the 2000 Census, more than 50,000 persons chose to write down explicitly that they identified themselves as 'Negro.'"

However, despite that official line, Census Bureau Director Robert Groves recently apologized to a caller on C-SPAN's Washington Journal for including the word on the census form, saying:

"The intent of every word on the race and ethnicity questions is to be as inclusive as possible so that all of us could see a word here that rings a bell for us ... it was not to be offensive and again I apologize on that. My speculation is that in 2020 that word will disappear and there are gonna be other words that are gonna change."

Ok, so although they are very sorry, we’ve heard directly from the Census Bureau that they included the word Negro because that is how a small group of Americans still identify themselves and that the form was designed to be inclusive so that every person could fill it out accurately.

Unfortunately, that leads to the issue of how the census counts individuals of Hispanic and Latino decent. According to our government, “Hispanic origins are not races” and as such, Hispanic is not included as an option under the racial category (it is included as an option under a separate ethnicity category). This has understandably upset some Hispanics and has led to much confusion, miscounting and non-response among the Hispanic population. In 2000, for example, 42% of Hispanic and Latino respondents marked down “some other race” instead of White or Black. So, the Census Bureau decides to leave on the word “Negro” because of 50,000 responses, but decides against including Hispanic as a race category even though 14 million Hispanics responded otherwise.

According to the all-knowing Wikipedia:

"The term race or racial group usually refers to the categorization of humans into populations or ancestral groups on the basis of various sets of heritable characteristics. The physical features commonly seen as indicating race are salient visual traits such as skin color, cranial or facial features and hair texture."

So, according to that definition, race is typically defined as skin color (among other physical features). Since many Hispanics and latinos consider themselves to be “brown” or at the very least "not white," why is it that Hispanic not its own race? Well, like the word Negro, it all goes back to a simpler time. Before 1930, the U.S. Government considered all people from central and South America white, mostly because immigration from these counties into the U.S. was not what it is today.

In fact, when you look back at the first U.S. Census form from 1790, you see exactly where the question about race grew out from. The 1790 census asked that households be split into five distinct groups

1) free white males over the age of 16
2) free white males under the age of 16
3) free white females
4) other free persons, and
5) slaves

In 1820, the category “free colored persons” was added to “whites” and “slaves”. In 1850, the category of “slaves” was finally dropped and the first true racial category was added asking whether a person was “white, black, or mulatto”). The category of “Indian (Native American)” was added in 1870. For the 1930 census, with Mexican migration into the U.S. on the rise, the term “Mexican” was added as a race category. However, according to the article, Racial Questions Rock the Census:

"During the 1940 count, however, the Bureau reversed course. The executive branch, eager to secure alliances as it entered World War II, and subject to intense lobbying by Mexican Americans and the Mexican government, decided to classify Mexican Americans as 'white.' As of 1950, Mexican Americans and Puerto Ricans were classified as 'white,' unless they 'appeared' to be 'definitely ... Negro, Indian, or some other race.'"

The ethnicity question was created for the 1980 Census, which for the first time as residents to categorize themselves both according to race and ethnicity. But why was Hispanic included as a separate enthicity question rather than included as a race? Well, blame the Mexicans.

"Part of the reason we have the 'Hispanic' ethnicity question is because Mexican Americans fought for it. They thought it would be advantageous to be categorized as white' and, so, they fought for an ethnicity category instead of a race category"

So, like most of the confusing things our government does, the answer to why comes down to politics. I think it is a safe bet to make that as the Hispanic population continues to grow in the U.S. and the stigma with being a non-white person in this country decreases, that we see a very different race question on the census form in 2020.

***

As an aside, I have worked on a few surveys (not the Census) which nonetheless used the census’ version of the race and ethnicity questions. By far, the most common “other” response for the race question is Hispanic/Latino, but my absolute favorite other response of all time was “brown with a whole lot of sugar.”

My proposal for the 2020 census race question

1) White
2) Black or African American
3) American Indian or Alaska Native
4) Brown with a whole lot of sugar

So, What Do You Do for Work?

2010-04-08T14:20:00.000-05:00

I harbor a secret and somewhat shameful resentment for people with easy to describe jobs like policeman, accountant or stripper. When you say that you’re a cop, people get it. No further explanation is required. On the other hand, I currently work as a research analyst for a company that conducts various types of survey research projects for public and private organizations. I’ve found that a lot of people don’t quite get that.

Here is a reenactment of every conversation I’ve ever had to a stranger about my job.

Stranger: So, what do you do for work?

Me: I work for a survey research company.

Stranger: Survey research? Like what kind of surveys?

Me: Well, we kind of do everything. Telephone surveys, mail surveys, internet surveys, focus groups.

Stranger: Oh, so you’re the guy who calls me up on the phone when I’m trying to eat dinner.

Me: Eh, well, not me personally...

***

So, yeah, I’m the guy who calls you up and bugs you while you’re trying to eat dinner, or sleep, or harasses you when you don’t want to stay on the phone for 25 minutes to answer a bunch of stupid questions about your bank . And in my next post, I’ll let you know a couple things you can do get out of these calls if you don’t want to do them (no, the Do Not Call List is not one of them).

Healthcare Reform

2010-03-25T14:31:00.002-05:00

via The Monkey Cage