How do we choose a technique? Should we immediately launch into a nice questionnaire , or is a series of focus groups better ? If you've followed the previous reflections, you already know the answer is complex: techniques —I suggest—come long after we've meditated on the mandate and context of the research, after we've reasoned about the research object and appropriately conceptualized it, an essential phase following which we can imagine "operational definitions," or empirical processes that allow us to gather useful information. And these empirical processes, which are defined now, after all that has preceded, lead to the choice of techniques that can be qualitative or quantitative , or, in complex research, a mix of both ( Mixed Method ). Having said all this, which I hope is clear and convincing, we can imagine a small guideline to help us stay on track amidst the enormous range of methodologies and the truly tangled meanders of the debate on Quality and Quantity . Take what follows as a guideline, rather than a dogmatic prescription. Thought, analysis, reflection – as I have already written in this column – are always and only the basis of every choice of method.
How to make a questionnaire
Here we will talk about the questionnaire while, in future appointments, we will address other techniques.
1) Sample research should not be conducted with fewer than 100 cases . It becomes ridiculous (as well as extremely unreliable) to say, for example, that 23,3% of the sample does or says something, if the sample is made up of 30 individuals and 7 of them gave that specific answer. Corollary: percentage values must be at least one unit; therefore: 100 respondents = no percentage in the results; 1.000 respondents = only one percentage value; 10.000 respondents = two percentage values.
2) Social research cannot be conducted without at least a bivariate analysis . If the data analysis yields a collection of data such as: so many percent are male; so many percent love football; so many percent read a book a month... then the research will have been useless. The questionnaire is used to conduct a variable analysis, looking for correlations, usually between independent variables (for example, gender, age, educational qualifications... which are not usually dependent on the object of study) and dependent variables (which vary—for reasons we want to understand—perhaps due to the object of study). Therefore, the analysis will be (at least): so many percent of males read books compared to so many percent of females; so many percent of regular theatergoers have a college degree, compared to only so many percent of TV series viewers.
3) Don't conduct questionnaires unless you understand what a data matrix is and how to solve the main data entry problems (even if you have someone else do it). Corollary: You must know how to "clean" a matrix to identify congruence and plausibility errors.
4) It's hard to write a good questionnaire if you don't know exactly how to put each item into a matrix and then process it . To give you a simple example, a multiple-choice question: "What do you do in your free time, blah blah?"; answers: "Cinema, soccer, TV, reading, sleeping..." How is it put into a matrix? How is it processed? I won't give you the answer, but it's one of the most difficult things if you don't know the matrix well and how to use it in bivariate analysis (at least).
5) It's equally difficult to create a good questionnaire if you don't know the scales . There are countless errors with scales (Likert type): how many intervals? Even or odd intervals? With numerical, semantic, or self-anchored anchors? Each of these questions generates a different type of scale, all correct, but each suited to different cases and with different management problems.
6) Never—and I repeat, never—use open-ended questions if you don't know exactly how to post-code them and insert them into the matrix. Open-ended questions seem more open to listening to the interviewees' thoughts without the constraints imposed by the researcher; absolutely true. But then you have to process them, especially to insert them into the matrix. First, a lot of information is lost this way (is it really worth it?), and second, post-coding requires significant hermeneutic skills, as well as fundamental classification skills. Think carefully.
7) In social research, a principle of economy applies , which, in the case of questionnaires, translates into including only questions that are necessary (or presumed to be necessary) for the purposes of the survey. If, while preparing a questionnaire, a member of your group says something like, "While we're here, let's also ask [such and such]!" or "I'd also like to ask [such and such]!", shoo them away with the excuse of making coffee and reject the proposal without even considering it.
8) The only decent way to administer a questionnaire is face-to-face ; only this way can you include more sophisticated instruments than just closed questions or scales, ensure the average quality of the completed questionnaires (if the interviewers are qualified), and much more. Self-administered questionnaires are generally poor, unless the researcher is present (for example, in the case of questionnaires distributed to a school class); CATIs are less than mediocre; CAWIs are a joke. And in any case: the method of administration largely determines the format of the questionnaire; they are not the same questionnaire, they are different questionnaires.
9) In all cases (especially operational research such as evaluation) where all of these things cannot be done (particularly those in points 1 and 8), and you find yourself having to create a small questionnaire for a small group, to be processed quickly, self-administered, etc., then, absolutely, respect this rule: only closed questions , few, with few answers, preferably 5-interval scales well explained to the respondent. Because there is another general principle of social research that I will express as follows: the more you do, the more you get wrong ; the more questions, the more scales, the more particular items, with sophisticated processing, and so on, open many doors to multiple possible errors. If you don't have the organizational capacity, or the time and money to do it well, or you are a beginner, the better is the enemy of the good , and therefore it is far preferable to do a little, but decently, than to pretend to do a lot and produce a disaster.
10) Finally, you only use a questionnaire if you also know how to do something else . If the questionnaire is the only technique you know (you think so), then you'll use it out of necessity, without understanding whether it's appropriate, whether the context, the mandate, the nature of the hypotheses, and so on, lend themselves to taking that path. If, however, you also know some other technique (especially very different ones, such as qualitative ones based on groups), then it's reasonable to assume that you'll choose the questionnaire simply because you have a research strategy in mind (although, obviously, that's not necessarily the case!). Good luck!
Previous contributions from “ The method is us ”
- We are the method
- Data information techniques and format
- Numbers are a language
- Speak well to think well
- The method is thought. Slowly
Claudio Bezzi
