A mathematical method for analysing questionnaires*

Bull. Org. mond. Sante 1972, 47, 87-97 Bull. Wid Hith Org. A mathematical method for analysing questionnaires* A. LEVINE,1 P. ROIZEN,2 P. ROZI,3 & H. CHRISTENSEN' Investigators using questionnaires are usually confronted with an enormous number of different responses whose significance with respect to a particular characteristic is not immediately clear. This paper presents a simple computational technique for determining the relative merits of each question and a score for each respondent. The rationalization for and the formalization of the method are discussed and some practical examples illu- strate how it can be used. The strengths and weaknesses of the method are discussed in relation to those of other methods. INTRODUCTION During the last three decades, the questionnaire has become important in many fields of research- e.g., psychology, sociology, and economics-and it is used frequently in diverse epidemiological studies. The questionnaire's rise to prominence has been aided by the development of high-speed computers, which can easily handle the large number of data usually obtained. Standard statistical results can easily be calculated and displayed. This convenience has encouraged investigators to employ long questionnaires designed to collect as much potential- ly useful information as possible. The result is often chaotic-a complex questionnaire that so thoroughly mixes the relevant and irrelevant that it is often difficult if not impossible to obtain substantial conclusions using ordinary statistical techniques. Methods are needed for the systematic examin- ation of questionnaires, methods based on a theo- retical foundation incorporating the peculiar nature of the questionnaire. The theory should be mathe- matical and parallel to a series of operations that can be performed on a standard high-speed computer. Research in this direction has been going on for a number of years. Lazarsfeld & Henry (1968) * From the Division of Research in Epidemiology and Communications Science, World Health Organization, Geneva, Switzerland. 'Mathematical Statistician. Programmer. 3Numerical Analyst. ' Statistician. Hans Christensen died on 8 August 1971. and Meshalkin et al.5, among others, have obtained useful results from their methods. At the end of this paper we indicate the similarities and differences between the work of these authors, factor analysis, and the approach presented in this paper. The type of questionnaire 6 for which our analysis is intended is frequently used in the social sciences and medical research, and is defined by the following characteristics: 1. The number of questions is greater than 1. 2. The different responses to each question are in the form of or can be translated into real numbers. 3. Each question measures the same characteristic or different aspects of that characteristic. In addition, it is assumed that for any particular question, the higher the real number assigned to a response, the greater the presence of the characteristic exhi- bited by the response. 4. The relative importance of the questions in calculating respondents' scores is not decided before the questionnaire is used. 5. The questionnaire has not been designed so that the response to one question logically excludes possible responses to other questions. 6 Meshalkin, L. D. et al. (1971) Mathematical methods in studies of chronic diseases (WHO unpublished document RECS/MM/71.5). ' A questionnaire is sometimes defined as " a device for securing answers to questions by using a form which the respondent completes himself." We shall not be restricted to this definition; a questionnaire for our purposes is a set of questions together with the responses. 2877 -87 A. LEVINE ET AL. Characteristic 5 excludes the following question- naire (Goode, 1952): Yes No A. Are you over 6 feet tall? ... ... B. Are you between 5 feet 6 inches and 6 feet tall? ... ... C. Are you between 5 feet and 5 feet 6 inches tall? D. Are you under 5 feet tall? ... ... Also eliminated are network questionnaires (Picard, 1965) and questionnaires designed for analysis by the Guttman scalogram method (Stouffer et al., 1950). Characteristics 1-4 are assumptions underlying the present analysis; characteristic 5 is not. It is included to ensure that the analysis developed here is not applied to trivial cases or cases that can be better analysed by other methods. The analysis we propose is a research aid that should clarify confusing situations. It is not intended for questionnaires that yield self-evident or easily understood results. It often happens that a newly devised question- naire is used to obtain responses from a large sample of a given population. The investigator is immediately confronted with an enormous number of different responses whose significance is not immediately clear. A natural desire would be to determine (a) the relative merits of each question, and (b) a score for each respondent. There are several advantages to be gained from (a). Irrelevant questions could be eliminated. The improved questionnaire would be shorter, and would thus afford future savings of both the interviewers' and the respondents' time. Furthermore, as fre- quently not every respondent answers every question, removing questions could, in effect, increase the size of the sample for which complete information would be available. As regards (b), the investigator may be interested in the distribution of the general population or subgroups with respect to a particular characteristic. Clearly, a well defined score for each respondent for the entire questionnaire would be essential for this purpose. This paper presents a simple computational tech- nique for determining (a) and (b). NUMERICAL ASSIGNMENT TO RESPONSES In this section the problem of assigning real num- bers to responses is discussed. Consider the fol- lowing hypothetical questionnaire designed to measure a respondent's favourable attitude towards health services: 1. With respect to research in health, do you believe that: A. a great deal more money should be spent on it, B. the level of expenditures should be main- tained, C. the level of expenditures should be signi- ficantly reduced? 2. Should hospitals be: A. improved or maintained, B. abandoned? Let us attempt to propose conventions (i.e., rules for assigning real numbers to the responses) that will satisfy assumptions 2 and 3 as presented in the introduction. Assumption 2 is easy to meet; it requires that all responses be in the form of real numbers. Let us suppose that response A to question 1 has the value 3; responses B and C to the same question will be assigned the numbers 1 and 2, respectively. In question 2, let response A take the value 5, and response B the value 3. These response conventions are displayed below: question response 1 1 1 2 2 A B C A B number assigned 3 1 2 5 3 Assumption 3 requires not only that all questions measure the same characteristic, but that for any particular question, the higher the real number assigned to a response, the greater the presence of the characteristic exhibited by the response. With respect to question 2, the stated conventions are adequate, since response A shows a more favourable attitude than does B and 5 (the value assigned to A) is greater than 3 (the value assigned to B). The conventions for question 1, however, are not satisfactory. Response B demonstrates a more favourable attitude than C yet 1 (B) is less than 2 (C). This violation of assumption 3 has been amended in the following table of conventions: question response 1 I 1 2 2 A B C A B number assigned 3 2 1 5 3 88 MATHEMATICAL ANALYSIS OF QUESTIONNAIRES Clearly, assumptions 2 and 3 are not very restric- tive. Our analysis begins with the matrix A: all a12 . a,m A a2l a2m ani . . . anm where aij is the value associated with the response of the ith person to the jth question, m is the num- ber of questions, and n the number of respondents. Let us begin by pointing out the inadequacy of the matrix A for an immediate comparison of columns (i.e., questions). Note that for some fixed values of i, j, and k, aij> aik does not imply that person i has more of the characteristic measured by question j than by question k. Compare response B to question 2 with response A to question I; then compare the values assigned to these responses. Assumption 3 requires a numerical relationship between the responses to a single question and the characteristic under measure, but not between the responses to one question and those to another. We might have attempted to assign numbers to responses in the same manner for all questions-5 for responses that show a great presence of the characteristic, 2.5 for those that show only half as much presence, and so forth-but such rules force a prejudgement of the worth of the responses that may not be justi- fied. To compare columns it is necessary that the entries in one column of A be somehow related to those in another. Thus, we must first transform the elements of each column of A to a common form. Consider column k of A, denoted by Ak =I a . i < m The elements ak of Ak can be ordered by magni- tude, and this order is significant since ak > ak does , Z~~~~~J imply that person i has more of the quality measured by question k than does person j. Now, since the characteristics of each respondent are not expected to change from one question to the next, and as the questions are expected to measure the same or similar characteristics, the ordering lent to our sub- jects by the magnitude ordering of their responses to a particular question should be roughly the same for all questions. Moreover, the final assigned responses should depend on the order of the initial assigned responses but not on the arbitrary conven- tion adopted to assign the initial responses. Clearly, the assignment of values to the responses to questions should also depend on the proportion of respondents giving each response. Obviously, a yes-no question to which 90% of the respondents reply in the affirmative is discriminating differently from one to which only 50% respond in this way. Our analysis below is intended to develop an assignment function that satisfies the characteristics discussed above. Let us consider one question, say j, containing k parts (permitted responses). Let pi represent the proportion of respondents who answered part i; i = 1,.. ., k;pi +P2 + . . . +pk = 1. Assume the responses have been put in order with respect to the characteristic under measure. Let P = (P1, P2, ... pk) represent the distribution of respondents among the parts and let hk (iP, represent the value assigned to part i when the distribution of proportions is p. Let us state some additional useful properties that the function hk should have. Clearly, if a question has only one item, this item cannot discriminate. Hence, we re- quire of any chosen assignment function h that hi (1,p1) = 0; Pi = 1 Since it is desired to preserve the ordering of the initially assigned responses the following characteris- tic is required of our function: hk (i,.p) <_ hk (j, P_; i < j It is also necessary that if there is a small change in the values of pi, then the values assigned to the items should not change much. Thus, it can be assumed that hk(i4p) is a bounded continuous function of jp. If a particular response is subdivided into two responses, it can be assumed that the average value associated with persons giving either of the new responses should equal the value associated with persons giving the original response. Also, this subdivision should not affect the value given to responses that were not altered. The following example illustrates these ideas. Consider the question How many children do you have? 89 90 A. LEVINE ET AL. Set no. 1 Set no. 2 Original response set New response set proportion proportion Pi 1. 0-3 children Pi 1. 0-3 children P2 2. 4-10 children p2g, 2. 4-7 children P3 3. 11 or more children? P2g2 3. 8-10 children p, 4. 11 or more children? where g+g2= The following natural conditions should be satisfied by our assignment function: Set no. 1 Set no. 2 value of response 1 = value of response 1 value of response 2 = (g1 x value of response 2) + (g2 x value of response 3) value of response 3 = value of response 4 Under the above assumptions and some very general conditions there is only one assignment function possible-namely, hk (i,P)= £pj-Pi pjj<i j>i This now gives a rational method for assigning values to question parts,' and this method when combined with the operations on the response matrix described below has some useful statistical properties. The following points are important to an under- standing of all that follows. It should be recalled that one of our objectives is to give a score to each respondent for the entire questionnaire. Behind this idea was the intention to separate out groups of respondents. The obvious approach is to allo- cate to subgroups those respondents who give exactly the same answer to each question-in other words, to form subgroups by response patterns. In practice, this approach is often not feasible, as the following example illustrates. If there are 10 ques- tions with three parts to each question, then there are 310 possible response patterns. With a sample of size say 200, it is possible that each pattern response subgroup would contain too small a num- ber of respondents for reasonable concJusions to be drawn about the statistical differences between the groups. A more reasonable approach is to combine pattern response groups by separating respondents I This assignment method was introduced by Bross (1958) but without the mathematical justification described here. by their score. Now it is clear that different res- ponse patterns can yield the same score, so that respondents with different response patterns can be in the same score group. This fact is trouble- some only if widely different patterns can result in the same score. A little reflection will show that such is the case whenever there are a large number of questions each with two parts, " 1 " being assigned to one part of each question and " 0" to the other. One way to ameliorate this fault in using the scores is to devise a method that will not necessarily assign the same values to corres- ponding parts of different questions. The method we have devised above satisfies this characteristic. Another way to ameliorate the difficulties in using scores is to reduce the number of questions, since the smaller the number of questions, the closer the scores correspond to the response patterns. The methods described in the next two sections enable questions to be eliminated in a rational manner. ASSIGNMENT OF WEIGHTS TO QUESTIONS We have described a procedure for the assignment of values to question parts that depends on the " discriminatory power " of these parts. In effect, we have weighted the parts of a particular question in relation to each other, and in a manner that is consistent for all questions. We now consider the problem of comparing different questions with each other; that is, of weighting questions to permit an easy determination of those questions that appear MATHEMATICAL ANALYSIS OF QUESTIONNAIRES to act alike and those that do not. For example, consider the following matrix of assigned responses: R 1 2 3 4 1 -½'/ - ½/2 /2 /2 Q 2 -/2 1/2 '/2½ 3 1/2 /2 -/2 - 1/2 Score -/2 - ½/2 /2 /2 Q = questions R= respondents It is clear that question number 3 acts differently from questions 1 and 2. How are we to characterize and measure this inconsistency? Once we answer this question, can we develop a simple computational method by which we can order the questions so that those questions that seem to act together appear close together in the ordering? We take as our starting point a technique often used in the field of psychological testing (Selltiz, 1964) and elsewhere. This technique is used to test the consistency of questions by correlating the scores of the highest and lowest quartile with each question. Those questions having low correlation with these scores are said to be inconsistent with the purpose of the questionnaire. (It is assumed, of course, that characteristic 3 listed in the first part of this paper is applicable to the questionnaire.) We generalize this method as follows: Let fll fl2 . fim fnL ... fnm represent the matrix of final responses; fij is the response of respondent i to question j after the assignment procedure described above is used. The score, si, of the ith respondent is defined as m Si = 2 fijj=1 A weight wj for question j can be defined as n w= £ ss i=l where n is the number of subjects. In our example, n = 4 and W1 = 1 W2 W3= -1 (These values can be taken as measures of question " consistency ".) Questions 1 and 2 have the same "weight " while 3 has considerably less " weight ". The weights can be normalized by the square root of the sum of their squares to obtain: wj = Wj/ZWi2 * * wl = 1/i3=W2 * W3 = - (1/ V3) New scores incorporating these weights can be cal- culated as follows: m * s1M= E fj . wjj=l It may be necessary to order respondents by their scores in order to classify respondents as well as questions. The new set of scores (si(1)) is sJ(1) = S2(1) = (-V-) / 2 s3(1) = SP) = (3/) / 2 In this case the score pattern remains the same as the original pattern; however, in more general cases a shift may occur in the scores that may change the order of the respondents. Repeating this process (calculating new weights on the basis of the new scores and then recalculating the scores on the basis of those weights) might permute the order of the respondents. Hence it is necessary to repeat this procedure until the ordering becomes independent of additional iterations. In some cases the ordering may always depend on the number of iterations, no 91 A. LEVINE ET AL. matter how many times the procedure is repeated. However, it is easy to show that under some general conditions there is a number t such that the ordering is independent of any additional iterations greater than t. We repeat our procedure on the example to obtain: n Wi= Si(1)fi i=1 so that WI' = 3/ = W2 W3(0) =-VJ W.1 ( =)* = -W3(1)* = 1 / V/ It may be seen that in this case immediate con- vergence is obtained. (We have found that the speed of convergence is associated with how well questions discriminate in common between groups.) In practice one continues as above to obtain the sequence {WJ(t)*}. Note that, at each iteration (k), wj(k) represents the covariance of the adjusted scores {5s(k)} with the responses (fij) to question j. Thus, the weights can be interpreted as a measure of the consistency of the questions; i.e., those questions with high positive weights can be interpreted as those for which respondents with high scores have had high responses and for which those with low scores have had low responses. Those questions with large negative weights are those for which low-score res- pondents have high responses and for which high- score respondents have low responses. We shall see in the next section, after discussing the statistical properties of the weights, that they have a far more significant interpretation than that given above. EXAMPLES OF THE EMPLOYMENT OF THE ANALYSIS The following examples of the use of the methods described above are given here to aid the reader in understanding their use in practice. Before describing some practical applications, we discuss below an experiment performed on the computer in which we simulated several question- naires. Consider the case in which there are two groups of respondents to be distinguished and a question- naire consisting of ten dichotomous questions. Let Table 1. Probability structure of the simulated questionnaire probabilities probabilities Question no. pk of Group 1 to k of Group 2 tolJj: 2 J : - part I part I1 part I part 11 1 50 50 50 50 2 60 40 40 60 3 70 30 30 70 4 80 20 20 80 5 90 10 10 90 6 99 1 50 50 7 50 50 1 99 8 75 25 75 25 9 70 30 30 70 10 30 70 70 30 Pi k represent the probability that a member of the ith group answers part k of thejth question; i = 1,2; k = 1,2; j = 1,2,.. .,10. The entries in Table 1 below are the values ofpi We note from Table 1 that questions 1 and 8 do not distinguish the groups, and that questions 2,3,4,5, 9, and 10 do distinguish them. We would suspect then that questions 1 and 8 should have weight near zero; it is not clear what weights 6 and 7 should have; the other questions should have high weight with 10 having a sign opposite to the rest. Now let Ti represent the number of respondents in the sample from group i; i=1,2. We considered four cases; the total sample size was 200 and the values of T1/T2 selected were as follows: Case 1 2 3 4 T3/T2, 1 1.86 4 9 Each case was run three times in the computer thereby simulating three different random samples for each case. Table 2 summarizes the results for one of the samples. Some interesting results are evident from this simulation. First the weights do represent the discri- minatory power of the questionnaires but this repre- sentation becomes cloudy as T2 approaches zero. This is as it should be, for in the limiting case, when 92 MATHEMATICAL ANALYSIS OF QUESTIONNAIRES Table 2. Question weights of the simulated questionnaire Question weight Question no. Case 1 Case 2 Case 3 Case 4 1 0.006 -0.070 -0.120 -0.200 2 0.104 0.146 0.192 0.100 3 0.351 0.255 0.282 0.465 4 0.383 0.405 0.430 0.370 5 0.533 0.515 0.463 0.431 6 0.350 0.380 0.264 0.196 7 0.278 0.36 0.419 0.368 8 0.023 0.045 0.123 0.053 9 0.333 0.365 0.343 0.333 10 -0.351 -0.264 -0.303 -0.349 Case no. 1 2 3 4 Number of iterationsa 10 11 16 23 a The number of iterations is the number of times that the computer repeats the matrix multiplications described above until there is no difference in the calculated weights in the fourth place. T2 = 0, the questionnaire has no discriminatory power. Also it is evident that the number of iterations increases as the discriminatory power of the question- naire decreases. This is an extremely important observation for it turns out that the number of iterations is related strongly to how well the ques- tionnaire can discriminate between groups. If the number of iterations is high (as a rule of thumb use 25) then it should be suspected that the questionnaire is poor for discriminating between subgroups of the sampled population. To understand how well the questionnaire above discriminates between the two populations for the four cases described above, consider the data in Table 3. The cumulative distribution for group 2 is given for cases 1, 2, and 4 in relation to the distri- bution of group 1; for example, for case 4, 80% of group 1 has a score less than or equal to that value, say S, of the score while 6% of group 2 has score values less than or equal to S. Thus it is clear that the questionnaire described above does increasing- ly well going from case 4 to case 1, which is the result one would expect intuitively. Now we shall consider another questionnaire (Table 4) denoted questionnaire II. It is difficult to tell a priori from Table 4 whether there are any questions that discriminate between the populations or whether there are some questions that collectively permit such discrimination. In other words, this table does not give the type of clear-cut case pre- sented in Table 1. Several different sample sizes were simulated in the computer and each case was run several times. In order to save space the results of two cases are presented in Table 5: case 1, population size T1 of group 1=120, popu- lation size T2 of group 2= 38; and case 2, popula- tion size T1 of group 1= 448, population size T2 of group 2= 152. (Later we shall justify the use of these rather unusual experimental sample sizes and the use of this particular questionnaire.) Columns 2 and 3 in Table 5 present some typical results with respect to the calculated question weights while entries in the small table underneath indicate the average number of iterations required in each case. (Columns 3 and 4 will be explained later.) It should be remembered that when the number of Table 3. Percentage cumulative distribution of group 2 in relation to percentage cumulative distribution of group 1 Percentage cumulative distribution of group 1 10 20 30 40 50 60 70 80 90 95 100 case 1 (sample size 100/100) 0 0 0 0 0 0 0 0 4 6 100 case 2 (sample size 130/70) case 4 (sample size 180/20) 0 0 0 0 0 0 0 0 3 4 100 0 0 0 0 0 0 2 6 17 34 100 93 94 A. LEVINE ET AL. Table 4. Questionnaire 11 Probability of response of Probability of response of Question group 1 to part group 2 to part no. I II III IV V I II III IV V 1 17 37 46 0 0 0 29 66 5 0 2 76 22 2 0 0 55 40 5 0 0 3 69 25 6 0 0 47 45 8 0 0 9 32 17 14 12 25 16 28 19 13 34 10 70 1 29 0 0 53 0 47 0 0 11 68 29 3 0 0 45 50 5 0 0 4 93 7 0 0 0 87 13 0 0 0 5 19 16 65 0 0 10 24 66 0 0 6 39 15 23 23 0 32 18 16 34 0 7 24 15 22 26 13 11 34 21 18 16 8 92 8 0 0 0 99 1 0 0 0 12 63 37 0 0 0 76 24 0 0 0 Table 5. Comparison of weights of the simulated and the actual questionnaire Questionnaire II Questionnaire IlIl Question Weights: Weights: Weights: Weights: case 1 case 2 actual actual case 1 case 2 1 0.196 0.349 0.409 0.369 2 0.194 0.072 0.338 0.4 3 -0.08 0.175 0.382 0.429 4 0.744 0.687 0.008 - 5 -0.158 0.095 -0.023 - 6 -0.06 0.104 0.132 - 7 -0.02 -0.003 0.120 - 8 0.036 0.005 -0.005 - 9 0.485 0.011 0.439 0.379 10 0.244 0.578 0.416 0.429 11 0.014 -0.031 0.417 0.439 12 -0.199 -0.130 0.012 - Case1 Case 2 Actual Actual| Case 1 | [ case 1 case 2 No. of 60 54 12 7 iterations (average) (average) iterations is large the discriminatory power of the questionnaire is poor. In addition, as predicted by the theory, the effect of total increases in the sample size, T1/T2 being maintained, is to decrease the variance in the weight values; that is, with increased sample size, changes from sample to sample in the weight and scores will decrease. We now describe questionnaire 111.1 For the purposes of analysis, the respondents were divided into two groups. From a large questionnaire, twelve questions were selected because intuitively they appeared relevant to the differentiation of the groups. Complete information for the twelve questions was available for 154 of 952 persons; 37 of these persons were in group 2. The question weights calculated in this case are presented in the third column (marked " weights: actual case 1 ") of the main part of Table 5. The number of iterations required was only 12 indicating that the question- naire discriminates well between the groups and that the question weights are stable. The factors that distinguish the groups are those represented by the highly weighted questions 9, 11, 10, 1, 3, and 2. Certainly, questions 4, 5, 8, 1 Because of a contractual agreement we are not able to disclose any details of this questionnaire: it is not a simulated questionnaire (as were questionnaires I and II) but is one that has actually been employed in field trials. MATHEMATICAL ANALYSIS OF QUESTIONNAIRES Table 6. Distribution of two populations by score interval Score interval 1 2 3 4 5 6 7 8 9 10 A. All questions (37 members in group 2: 117 members in group 1-Ho: Fi = F2) F2 a 0.000 0.108 0.189 0.351 0.459 0.540 0.702 0.783 0.864 1.000 F, a 0.094 0.282 0.419 0.504 0.607 0.692 0.795 0.855 0.914 1.000 Absolute difference 0.094 0.174 0.230 0.153 0.148 0.152 0.093 0.072 0.050 0.000 (Ho rejected at 0.05 level of significance) B. Low-weight questions (37 members in group 2: 117 members in group 1-Ho: Fi = F2) F2 0.000 0.081 0.216 0.351 0.378 0.513 0.621 0.756 0.891 1.000 Fl 0.017 0.119 0.124 0.333 0.478 0.598 0.726 0.850 0.957 1.000 Absolute difference 0.017 0.038 0.092 0.018 0.100 0.085 0.105 0.094 0.066 0.000 (Ho not rejected even at 0.1 level of significance) C. High-weight questions (37 members in group 2: 117 members in group 1-Ho: Fi = F2) F2 0.081 0.162 0.216 0.378 0.486 0.594 0.702 0.756 0.837 1.000 Fi 0.213 0.316 0.487 0.527 0.624 0.726 0.786 0.837 0.914 1.000 Absolute difference 0.132 0.154 0.271 0.149 0.138 0.132 0.084 0.081 0.077 0.000 (Ho rejected at 0.05 level) D. High-weight questions (152 members in group 2: 411 members in group 1-Ho: Fl = F2) F2 0.026 0.092 0.157 0.210 0.414 0.546 0.657 0.901 0.967 1.000 Fi 0.147 0.236 0.304 0.418 0.547 0.659 0.763 0.907 0.982 1.000 Absolute difference 0.121 0.144 0.147 0.198 0.133 0.113 0.106 0.006 0.015 0.000 (Ho rejected at 0.001 level of significance) a F2 = cumulative distribution of group 2; Fi = cumulative distribution of group 1. and 12 can be eliminated without significantly changing the scores. It appears that the question- naire distinguishes between groups but we are not cer- tain that it distinguishes group 1 from group 2. To aid us in making this determination, Table 6A shows that these two populations are separated by the questionnaire in the sense that the hypothesis Ho, that their cumulative distributions with scores are the same, is rejected at the 0.05 level of significance using the Kolmogorov-Smimov two-sample test. On the other hand, Table 6B shows that the low- weight questions 4, 5, 8, and 12 do not distinguish between these two groups since Ho is not rejected even at the 0.1 level. Table 6C shows that the high- weight questions alone distinguish between these two groups. In order to increase the sample size, the low- weight questions 4, 5, 6, 7, 8, and 12 were removed from the questionnaire. Information on all the remaining questions was available for 563 cases; group 2 in this case consisted of 152 persons. The weights for the questions in this example are summa- rized in Table 5 in the column marked " weights: actual case 2 ". The number of iterations needed in this case was only 7, indicating that the question- naire discriminates well and that the weights are 9 95 A. LEVINE ET AL. more stable than in the previous case. The former implication is borne out by Table 6D where it is shown that Ho is rejected at the 0.001 significance level. The latter implication is one that would be expected with larger sample sizes. In fact, something would appear to be wrong with our theory if in- creasing sample size did not increase the stability of the system. Now let us relate the questionnaires II and III discussed above. The values used for the proba- bilities of responses in questionnaire IL were taken from the proportions of the responses to the various parts actually obtained using questionnaire III with a total sample size of 154. Thus the estimates of the probabilities of the responses obtained using ques- tionnaire III in the field were used as the theoretical probabilities in our simulation (Table 4). One asks immediately why the results obtained are so much better with the field questionnaire III than with the simulated questionnaire H. To begin to answer this question, it should be noted first that in question- naire II, the simulation is such that the response of any respondent to any question is statistically inde- pendent of this respondent's answer to any other question. As a result, the correlation matrix con- sisting of the values of the correlation coefficient between questions contains off diagonal values that are nearly zero. On the other hand, in the actual case the correlation coefficient between many questions is high. The theory indicates that when this is so, convergence is rapid and under certain conditions discrimination is good. Thus, the corre- lations that appear in practice may be important factors in determining the discriminatory power of a questionnaire. As our examples illustrate, we have provided a method that (a) helps to determine the relative worth of questions and thus to improve the questionnaire design and (b) helps to identify respondents who appear to behave alike with respect to their responses. Our work is not complete, however, and the dis- cussion below indicates that some theoretical prob- lems still remain. We now summarize how the weights should be interpreted. A high positive weight means, in general terms, that a question with this weight is well corre- lated with the scores. Thus, if a high score means, for example, " high risk ", then the question is measuring " high risk ". If a question has a sig- nificant negative weight then the question is inver- sely correlated with " high risk ". If it has a weight so low that the assigned values after being multi- plied by this weight do not contribute significantly to the respondents' scores, this question is not distinguishing groups and can be eliminated from the questionnaire. COMMENTS ON OTHER METHODS The literature on questionnaire analysis has grown rapidly in the last ten years, and some brief comments on the relationship between our work and that of others are called for. First, most of the literature investigates dichotomous questionnaires. To our knowledge, our work is the first to study a question- naire composed of questions not having a fixed number of parts. Some of the previous investi- gators-e.g., Lazarsfeld & Henry (1968) and Meshalkin (op. cit.)-assume the existence of a " latent variable " and assume a model of linear dependence of the responses on the latent variable. We do not make these assumptions here. It is true that we have suggested that questions measuring the same " thing" should be examined together; how- ever, such a suggestion is not necessary to the theory underlying our method. It is made so that it will be easier for the researcher to interpret respondent scores and question weights. The concepts and models developed by Lazarsfeld & Henry and by Meshalkin are nevertheless extremely useful and in certain cases are more appropriate than those pre- sented here, as for instance when one can assert through knowledge of the physical conditions under- lying a problem that the probability of responses will be linearly dependent on a latent variable. We do wish to comment in detail upon one ana- lytical technique that is in some ways similar to those of Lazarsfeld & Henry and Meshalkin methods and that described in this paper. This technique is factor analysis, which is a clustering technique. The theory assumes that there are some underlying factors and that the data are linearly dependent on these factors. In the sense that it clusters the data, factor analysis resembles the method described here. In the sense that it assumes a linear structure on nonmeasured " factors ", it resembles the model of " latent variables ". There are some immediate objections to the use of factor analysis in the problem of questionnaire analysis. Factor analysis demands that the data be linearly related to each other, measured by con- tinuous numbers, and approximately normally dis- tributed. None of these demands can reasonably be met in most of the complicated problems involving 96 MATHEMATICAL ANALYSIS OF QUESTIONNAIRES 97 clinical information, ecological phenomena, etc. Certainly the responses to most questionnaires are qualitative. We have pointed out the faults of factor analysis in order to indicate more clearly the usefulness of our methods. We hasten to add that under certain special circumstances where one well understands the phenomenon under investigation, factor ana- lysis can produce a useful summary of results. However, in our work we do not assume the linear structure described above. The result of our analysis is directly related to the physical problem as it is clear that most questions will relate to the problem: this is not always the case with factor analysis. We relate the weights, the scores, and the impli- cation of a large number of iterations directly to the questionnaire and not to some intermediary ficti- tious factors. In addition, we have some knowledge that we hope will be increased by future research concerning the effect of sample size on the weights, scores, and iterations. Much research is necessary. The exact sampling distribution of the weights must be obtained and our analytical research must be extended to the case of dependent questions. The weakness of our approach is that the complete theory has as yet not been developed. RESUME UNE METHODE MATHEMATIQUE D'ANALYSE DES QUESTIONNAIRES Le questionnaire (defini ici comme une serie de ques- tions accompagnees de leurs reponses) contenant des informations qualitatives est l'un des plus importants instruments de recherche dans beaucoup d'etudes epid& miologiques. Le developpement des ordinateurs a grande vitesse, capables de traiter un volume consid6rable de donnees, a incite les investigateurs a recourir 'a de longs questionnaires afin de rassembler le plus grand nombre possible de donnees potentiellement utilisables. Le r6sul- tat est souvent chaotique, consistant en un ensemble complexe de donnees oiu les informations pertinentes et les informations sans rapport avec l'objet de l'enquete sont si intimement melees qu'il est souvent difficile, voire impossible, d'en tirer des conclusions substantielles par les techniques statistiques usuelles. Dans le present article, les auteurs exposent des me- thodes qui permettent une analyse systematique des donnees afin a) de d6terminer la valeur relative de chaque question et b) de caracteriser numeriquement les divers types de r6ponses. Grace a une methode simple de calcul, les donnees qualitatives sont transform6es rationnelle- ment en donnees quantitatives qui fournissent les ele- ments conduisant a la solution de a et de b. La m6thode a e mise a l'epreuve au cours d'une experience simulant sur ordinateur divers questionnaires et les caracteristiques de deux populations. On montre qu'il est possible de determiner la valeur relative des questions et d'identifier les repondeurs qui donnent des reponses semblables. Les applications pratiques de la methode sont examinees ainsi que ses avantages et ses limites au regard d'autres procedes mis au point dans ce domaine de recherche. REFERENCES Bross, I. D. J. (1958) Biometrics, 14, 18-38 Goode, W. J. & Halt, P. K. (1952) Methods in social research, New York, McGraw-Hill Book Co., Inc. Lazarsfeld, P. F. & Henry, N. W. (1968) Latent structure analysis, Boston, Mass., Houghton Mufflin Co. Picard, C. (1965) The6orie des questionnaires, Paris, Gauthier-Villars Selltiz, C. et al. (1964) Research methods in social re- lations, New York, Holt, Rinehart & Winston Stouffer, S. A. et al. (1949) Measurement and prediction, studies in social psychology in World War II, Vol. II, Princeton, N. J., Princeton University Press