How to design an excellent survey questionnaire?

Dear Reader! I wish you a Happy New Year full of energy, passion, inspiration, and curiosity! We are driving forward in 2022 with a strong set of research and evaluation capabilities to help clients fast track delivery of organizational learning, monitoring, and capacity development. This year, we have a strong kick-off with old and new clients, so I’m very excited about 2022 so far.

Today I picked up a subject that has been on my agenda lately. Not so long ago I was working on a dataset collected via a questionnaire that was distributed before my involvement in the survey. I had huge struggles with data cleansing and handling some issues that could have been avoided by editing the questionnaire in a careful way. In the case of another survey project, I will seem to have the possibility to join the team already before distributing the questionnaire, and I’m sure, that with a forward-thinking approach we will be able to reach high quality and efficiency.

Why it is important to pay a lot of attention to the wording?

First of all, the data analytical methods will largely depend on the level of measurement; in other words, on the types of response categories. If we want to investigate the relationship between two variables, for example, linear regression does not work with ordinal dependent variables, since one of the assumptions of SLR (Simple Linear Regression) is, that the variables need to be measured on a continuous scale (interval or ratio level of measurement). If our dependent variable is a Likert-scale item, we can forget about SLR. As soon as the data collection has been done, we cannot increase the level of measurement anymore, so at this point, either we perform ordinal regression, or we transform our response variable to a binary one and apply logistic regression, which is losing information in a way. It is clear, that a simple decision during the designing phase can result in complications later on.

Second, we need to ensure validity and reliability. The former measures the degree to which our results agree with the real world, the latter refers to consistency, thus getting the same results if the measurement were to be taken again under the same conditions. The point is, that our questionnaire has to capture the topic under investigation effectively, avoid confusing or misleading items, and the expressions we use have to mean the same for all the respondents.

I give you some tips that can guide you to phrase your questionnaire in a better way:

  1. Use closed-ended questions whenever it is possible!

When it is about survey method, our aim is usually to gain quantitative insights into certain phenomena and get the results in a numeric format. Analyzing and interpreting open-ended questions in case of large samples can be tiresome work, so there must be serious benefits in order to opt for it. Performing quantitative interpretation in case of such questions is not only hard but incorrect. Any time we want to use an open-ended question, we have to be sure, that the qualitative nature of our analysis will serve our goals, thus we want to explore, understand the backgrounds, gain nuanced descriptions or we want to get answers expressed in the exact words of the respondents, but we don’t need to generalize our findings. If you have more questions like this, probably survey method is not the best tool for you.

2. Be careful with the scales!

Ordinal questions can be used only when the topic is well-defined and the response options represent a gradation along a single dimension. Likert scales are widely used, since they measure the attitude directly simply expressing the individual’s level of agreement, and can grab other variations simply too, like frequency, quality, importance, or likelihood. When it is about Likert-scale items, in most cases we want to have a balanced scale by default (Agree, Undecided, Disagree), meaning that there is an even number of points on the low, or negative side, and on the high or positive side. Normally the respondents can give a neutral answer too. On the other hand, it is also an option, that we want to force the respondent to decide. In this case, we can leave out the intermediate option, thus we will have even-numbered categories.

It is also a realistic scenario that we know in advance, that the respondents will skew heavily to one side. One might say, that in such a case we can use an imbalanced structure, however, I would still recommend using a balanced scale here too. If you want to have a more fine-tuned picture, you can just simply use more categories in both directions. My experience is, that a balanced structure works better because of the cognitive patterns of the respondents. We tend to think symmetrically about negative and positive perceptions of a question, and if we see a different arrangement, it is easy to get the feeling that the researcher wants to orient our answer in a certain direction. Not convinced? Test this hypothesis with your specific target group using cognitive interviews (see more details in point 10)!

3. Less can be more!

Having more response categories can give a more precise picture, however, it is not always necessary, and if we have fewer respondents, the frequency within one category may fall into a critically low range. Think ahead, and focus on the benefits you can gain with more response categories! Sometimes a five point-Likert scale is good enough because we just want to know whether the respondents like something in general or not. Here we don’t necessarily need the difference between “somewhat agree” or “strongly agree”.  However, there are other cases when you need a more accurate reflection of the respondent’s true evaluation. At the very worst, you can collapse the scales later on, but again, this will mean extra work in the analysis phase.

4. Think about the atypical answers!

Sometimes the respondents don’t want to answer the question, or they don’t find the correct category, that applies to their specific situation. You have to be conscious of how to handle these aspects. I would recommend letting most of your questions be optional to answer, and offering ‘other’ or ‘not applicable’ response categories too. Missing values and an ‘other’ category can be handled statistically easily. When you design the survey you might have a huge knowledge about the phenomenon you want to assess, but still, it can happen, that you may miss covering all the possible alternatives. If the respondent doesn’t find the best answer, and cannot skip the question, he or she will probably select a false category (it happened to me several times as a respondent), which will result in bias finally.   

5. Use neutral and precise language!

This has major importance if you want to assess emotions or perceptions. Questions should not be suggestive or orient the respondent. Answers should be as specific as possible so that the interpretation of the results will be clear. If you work with a multi-cultural target group (for example employees of a multinational organization or you conduct international research), it is reasonable to apply a culturally responsive approach already during phrasing the questionnaire to avoid bias. Always tailor a questionnaire to the interests and style of the respondents, and be compliant with the legal and ethical considerations of the specific country!

6. Be as clear as possible!

One question should focus on one type of information. Do not try to combine two things into one item! If the item is complex, you can provide a brief instruction to explain how answers should be provided. If you are not sure, whether the respondents have enough information to answer the question properly, you can give some background information in the questionnaire, or test their knowledge with one or two questions. However, you should remember, that the survey must be relevant to the respondents!

7. Maintain the respondent’s interest and attention!

With good wording, you can give the respondents the impression that they participate in an interesting conversation, and their opinion matters. Most people like to tell their standpoint, but filling a questionnaire requires time, and the more questions you have the higher the dropout can be. If you experience a high dropout rate (so you have a lot of respondents who quit the survey without completing it) with a questionnaire that you want to use more times, it is worth doing a dropout analysis to unveil the reasons. Don’t forget, that the decision to participate in the questionnaire is influenced also by the length and the flexibility of response time!

8. Collect socio-demographic data!

When you conduct a survey, in most cases you will be interested in descriptive statistics like frequencies, means, and standard deviations, etc. But also in these cases, it can have added value to see the distributions by gender, age, place of living, seniority, management level, or any aspect that makes sense from the aspect of your research. Any pattern or association you discover in the answers can offer you new insights and action points. Don’t miss the opportunity to see the results from different perspectives, and collect socio-demographic data too!

9. Design the questionnaire in a participative manner!

As a consultant, it happens many times, that you have to conduct a survey within an organization or among a specific target audience as an outsider. The best way to utilize the organizational knowledge related to the respondents is to create the questionnaire in a participative manner. Collaborate with the HR department, professionals, or subject-matter experts to ensure that you obtain the data required for analyses in a usable format, the questionnaire uses correct terminology, and the audience is addressed in a proper style or tone!

10. Do a pilot and use cognitive interviews for complex issues!

A major problem with a representative survey is that the questions are answered by respondents who might have different interpretative backgrounds or different frames of reference in connection with the research field. There are problematic issues, where some of the related terms are only recently used (e.g. in the field of social media), so they can have a different meaning for different social groups. In such cases, it is reasonable to test the questionnaire items in advance to avoid cognitive bias. Pilot questioning is a good solution, but it might be not enough if the research topic is complex. Although a pilot can unveil some basic interpretational problems, it does not contribute to reliability significantly.

Cognitive interview is a method that helps researchers in getting a deeper understanding of the interpretation processes of the respondents and gives a precise list and description of the necessary modifications. The methodology is based on the oral reports of the interviewees about their own mental processes as they answer questions (Blair & Presser, 1993). The cognitive interviews aim to unveil whether the respondents from different social groups comprehend the questionnaire items in a similar way and whether the questionnaire is able to record their responses precisely. They help in resolving measurement problems too and in filtering misleading indicators or finding missing ones (Berends, 2006).

If you do an online survey or an electronic questionnaire, don’t forget to test it before distribution!

+1 Think out of the box!

Sometimes it is difficult to measure a variable using a questionnaire item because the respondents’ estimations do not necessarily reflect the reality accurately (e.g. they have to specify their daily time consumption for making phone calls or the frequency of checking their mailbox). In these cases, we can combine the survey with other methodologies, e.g. diary method or in-depth interviews to obtain reliable and objective information. If it is necessary, dare to combine the survey with other methods to increase reliability and validity.

Recommended literature:

Blair, J., & Presser, S. (1993). Survey procedures for conducting cognitive interviews to pretest questionnaires: A review of theory and practice. In: Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association 370, 75

Berends, M. (2006). Survey Research Methods in Educational Research. In J. Green, G. Camilli & P. Elmore (Eds.), Handbook of complementary methods for research in education 623-640. Mahwah, NJ: Lawrence Erlbaum Associates.