We’ve built a survey, distributed it to potential respondents, and are now ready to start analyzing the results.

Although the possibilities for data gathering through surveys are limited only by your imagination, most questionnaires are made up of just a few basic question types. Understanding what they are and what can be done with their respective output is crucial for getting the most value from the raw data.

1. Multiple choice

A multiple choice question in which respondents are limited to making only one selection is used when there is either only one possible choice - what month were you born? - or when you want to understand the most relevant selection from a set of given options - what is your favorite type of ice cream?.

Since respondents are forced to select only one response, be sure to include all possible options. When that is not practical due to length, simply include other - please specify as the final choice. In this example, we added I do not know in case the respondent doesn’t have enough information about hiring from the previous or current year.

The most common way to summarize these select one questions is to share the proportion of responses from each unique option relative to the total number of respondents. The sum of these should always total 100 percent.

How many new data hires do you envision making this coming year?

The majority of HR professionals responding to the survey (52%) anticipate hiring fewer data hires this year than they did the previous year. Only 16% expect to higher more.

Two common feedback variants

Likert scale

The Likert response scale attempts to gauge satisfaction or agreement from low to high. A survey may make a statement and provide response options such as:

Here are the responses by 200 of our HR professionals in Europe.

Although counts and percentage summaries are most common, you may also see mean scores reported by assigning a numeric value to the ordered response sequence.

In our simple example, the mean score for the question would be calculated by either taking the weighted average of the rank ordered result table above or by converting each response from its response label to its appropriate ordered rank value.

In the latter approach we then simply take the average of the converted values to find a mean score for the question of 3.6, a one-number indication that the average respondent leans toward agreeing with the statement. A mean value of exactly 3 in our case would indicate the typical respondent neither agreed nor disagreed.

Although this technique mathematically possible because Likert scales can be converted to rank-ordered numbers (e.g., Strongly disagree = 1, Somewhat disagree = 2, etc…), many statisticians advise against using mean values in these situations.

In spite of such concerns, mean score summaries can help simplify general sentiment, especially when communicating a long list of questions with the same ordinal scale.

Finally, you might want to compare intensity at the edges of the response scale. This is referred to as Top Box when focusing on the top of the scale (e.g., Strongly agree). If you collapse the top two options (e.g., Strongly agree and Somewhat agree) it is called Top 2 Box. Conversely, if you look or compare with the bottom of the scale it would be Bottom Box or Bottom 2 Box.

In our example, Top 2 Box would show that 53 percent of respondents either somewhat or strongly agree with the statement compared with just 34 percent who either somewhat or strongly disagree (Bottom 2 Box).

Net Promoter Score

Net Promoter Score (NPS) is a popular and simple business metric created by Bain & Company. Respondents are asked how likely they would be on a scale from zero (not at all likely) to ten (extremely likely) to recommend a specific product, service, or company to other people.

Let’s say the survey responses to the NPS question came back like this:

Respondent percentages are then grouped as follows.

• Promoters: Responses of nine and ten = 46 percent
• Neutral: Responses of seven and eight = 29 percent
• Detractors: Responses of zero through six = 25 percent

The final NPS score is the percentage of respondents who are promoters (46) minus the percentage who are detractors (25). NPS can range from negative 100 percent to positive 100 percent and is often reported without the percent sign. In our example, we found an NPS score of 21 (46 minus 25). The higher value, the better. Depending on the industry, any score above zero is considered pretty darn good.

2. Multiple select

Sometimes you don’t want to limit your respondents to only one choice. A multiple select question allows people to choose as many options as are applicable.

Unlike select one multiple choice questions, summary percentages generally sum to more than 100 percent as respondents are able to pick all options that reflect their reality.

What data roles do you plan to hire this year?

Here we find that it is more common for companies to anticipate Data Scientist hires (77.0%) when compared with Data Architect hires (28.5%).

3. Rank order

Select one multiple choice questions limit the respondent to one response. Although select all questions enable a respondent to choose as many options as relevant, it doesn’t reveal how much more important one selection is compared with another. That’s where rank order questions come in.

Rank order questions ask respondents to explicitly make value judgments against a series of options. There are a few ways to analyze such responses. Assuming everyone responds to each option, you should have a rank for each strategic option. In our example, this ranges from a rank of one to a rank of four. Here are the responses from the first ten respondents.

Please rank the importance of the following strategic actions for your organization from most important (1) to least important (4).

Mean rank

The mean rank calculates the average selected rank for each response. A lower average value indicates a higher rank and a higher average value indicates a lower rank.

Launching new products has a higher average rank (1.96) when compared with entering new markets (2.73), a glimpse into the strategic priorities for these companies.

Proportion of ranked values

You could also look at how many times a specific rank was assigned to a given option. For instance, 63 percent of respondents assigned launching new products as their top priority. Meanwhile, only 6.5 percent of respondents indicated that entering new markets was the top goal.

Although both approaches show the same directional finding, the proportion of ranked values measure is likely more impactful in this case as it demonstrates wider numeric differentiation.

4. Text entry

Text entry questions work well when you don’t have a great idea for which response options frame a certain topic or for when you simply want to give respondents a chance to provide more flexible, open-ended input.

Analysis of text data is not as straight forward as the other question types. However, we can use basic text analytics and Natural Language Processing (NLP) to get a sense on what people are thinking.

Here are some customer reviews from an e-commerce platform to use as an example with responses shown for the first ten respondents who provided a review.

Please leave a review from your purchase

It would be difficult to sift through the 19,676 records manually to tag certain words or emotions. Thankfully we can turn this into a data problem by putting each word in its own row and then analyzing the adjusted data series.

This makes it easy to count which words appear most often or conduct sentiment analysis to get a broader sense of customer satisfaction. For now we’ll look at the most frequently mentioned words, excluding common terms such as the or and.

Overall, customers look pretty happy. A wordcloud is a common way to visualize which terms dominate the conversation, aligning word size and color with the number of mentions.

Recap

We’ve now covered key aspects of survey planning and distribution, as well as analysis for core question types. Despite remaining one of the primary tools for organizations to understand the world, many people underestimate the effort required to run surveys and collect meaningful results.

We hope this practical guide has provided some ideas to help with each step of your next survey project.

This article is the second of three-part series on practical survey considerations with guidance from Gregg Schoenfeld, survey guru and founder of MNoet, a boutique research consultancy.

• Part 1: Logistics
• Part 2: Distribution
• Part 3: Analysis