META-ANALYSIS OF SURVEYS FROM A QUALITATIVE PERSPECTIVE.
Capitolina Díaz-Martínez
Pablo Navarro
1. INTRODUCTION: WHY ARE SURVEYS SO USEFUL?
The use of surveys based on the production of data by means of a questionnaire, is perhaps the most common tool of social analysis. Several factors explain the appeal of surveys to social researchers. First, this analytical tool allows the researcher directly to focus her subject of interest, by just questioning people about it. In this way, sociological investigations can have a content as specific as intended, and can gather homogeneous data supplied --in principle at least-- by exactly the kind of people who is the target of the research. Second, the scores produced by the survey are amenable to statistical description and analysis, through the use of correlation and regression techniques. Third, by relating the surveyed sample to the corresponding population, the set of data provided by the survey can be the basis of statistical inferences. These three main reasons would explain the widespread use of surveys in social research.
2. CLASSICAL PRESUPPOSITIONS IN THE USE OF SURVEYS.
Useful as they are, though, surveys face a host of interpretive problems, which chiefly arise from the very presuppositions that buttress their power as an analytical tool. These presuppositions, generally assumed by researchers in an unconscious way, compose what may be called a 'positivist outlook', which would lie on several basic tenets. First, the 'objectivity postulate': the realities that the survey tries to investigate would exist "out there", and would have a reality independent from the way in which individuals perceive them. When these individuals ignore such realities, or portray them in an idiosyncratic manner, this mismatch is merely conceived as the result of a 'measurement error'. In fact, the very notion of 'error' presupposes the existence of an independent reality that any description would approximate at best.
A second tenet of the positivist outlook on surveys would be the 'transparency postulate': words, and linguistic expressions in general, simply denote those objective realities which the survey tries to grasp, and they do so in an unambiguous way: each semantic unit has a unique meaning --or, at least, such units can be worded, with some ingenuity, according to this principle. The search for a perfect, unambiguous, referentially transparent, purely denotational language, is one of the characteristic longings of the positivist frame of mind. But however useful this search for semantic clarity is when it concerns the theoretic discourse of a discipline, the idea that this semantic transparency can be imposed on the very subject of that discourse is unwarranted. Sociologists must explain social phenomena as clearly as possible; but they should not assume that those phenomena, which are shaped by language, are unambiguous.
The third positivist tenet would be the 'generality postulate': the unequivocal meanings of the words and expressions involved in the research process, are known and accepted by all --at least, by the overwhelming majority-- of the individuals who are the subject of that process. Of course, this postulate is closely related to the previous one: the question is not the possibility, for nearly all the surveyed individuals, to attach some meanings to the topics asked by means of the questionnaire; the question is whether or not these individuals ascribe the same meanings --the allegedly unambiguous, transparent ones referred to before-- to those topics.
These three postulates, usually implicit but firmly entrenched in the epistemological background of many social scientists, work as the necessary underpinnings of the fundamental principle assumed by classical survey research: the 'principle of equivalence', according to which equal answers to the questionnaire carry identical meanings. This principle is certainly entailed by the three postulates described above. Besides, it is the only reasonable basis for the statistical aggregation and analysis of these answers. And yet this principle, maybe self-evident for a majority of social researchers, is not granted.
3. THE LIMITS OF THESE PRESUPPOSITIONS.
Many surveys researchers know, with a certain degree of epistemological awareness, that the postulates in question are arguable, to say the least. They seem only to fit and work properly --and just in a tentative manner-- under very specific 'boundary conditions'. For instance, when the individuals questioned in the survey are interrogated about plain facts which are in the public eye, and these individuals are not interested in misleading the interviewer about their attitudes towards such facts. Now if some of those postulates were untrue as a general rule, the corresponding 'equivalence principle' would collapse, devoid of all epistemological legitimacy. In point of fact, the postulates of objectivity, transparency and generality demarcate a limit case of the process of semantic interaction between the researcher and the subjects investigated that is typical of surveys --an interaction implemented through the questionnaire.
It is a limit case in which the researcher achieves a perfect neutralization of these subjects interpretive capacities about the matters asked, by imposing her own 'official interpretation' upon them. In these conditions , the subjects of the research would only act as mere processors --not as genuine producers-- of an information predesigned by the researcher. However, human beings are not mere 'information processing machines', but 'information producing' realities, which generate new, idiosyncratic meanings in order to understand their natural and social environment. A typical questionnaire asks questions and provides possible answers to them. But in doing so, it predefines the sense of these questions, by curtailing the set of admissible answers. Jumping out of this set of pre-packed answers is strictly forbidden. This means that the questions can only be construed in one way --precisely the way that entails the admissible set of answers. In general, the sense of a question --its meaning as such a question-- is determined by the gamut of possible answers that the questioner would accept as appropriate when she puts this question to somebody in a specific situation.
Thus, in the case of a questionnaire, the sense of each question is coded in the corresponding answers, and this sense is dependent on a certain interpretation of the matters involved --on a particular, implicit ontology--, which is provided by the author of that questionnaire, and imposed on its respondents. This 'official interpretation' can be more or less compatible with the personal, spontaneous views of the respondents. But in any case it represents a system of semantic and, in the last analysis, ontological constraints enforced on them by the researcher. To maintain the stability of these constraints imposed by the researcher on the subjects, and maybe incompatible with their pragmatic as well as a semantic schemata, is very difficult indeed.
And the price the researcher has to pay even when she seems to achieve her aim is high. Because an effective interpretive neutralization of the subjects requires a drastic impoverishment of the questions addressed to them: specifically, all the questions that allow interpretive differences should be banished from the questionnaire, when these questions usually are the more interesting ones from a sociological perspective. Sociological analysis is about differences between people and groups of people; consequently, a method that systematically blurs those differences is hardly appropriate for the purpose of social research. Now this is what classical survey analysis tries to do, with respect to interpretive differences, as far as it strongly constrains the semantic diversity of the subjects by assuming the 'equivalence principle'.
4. SEMANTIC ANOMALIES IN SURVEYS.
Needless to say, real questionnaires, irrespective of the intentions and epistemological assumptions of their authors, rarely accomplish that perfect neutralization of the semantic and, in fact, ontological diversity featured by the individuals investigated. This diversity percolates through the literal expressions of the questionnaire and shows its presence in an indirect, rather elusive way. For instance, the rate of 'no answers' may substantially vary along the questionnaire. This fact would supply a clear indicator of the adequacy, or lack of adequacy, of the questions --and their corresponding, predefined answers-- to the interpretive schemata of the individuals surveyed. The changing rates of 'no answer' would suggest different degrees of semantic (cognitive and emotional) as well as pragmatic adequacy of the diverse items of the questionnaire. Other possible indicators of the degree of 'interpretive adequacy' between the researcher's ontology, as it is featured in the questionnaire, and the ontologies presupposed by the individuals surveyed, are more subtle in nature. In this respect, apparent inconsistencies between responses to different questions semantically related --at least according to the researcher's interpretation-- would be highly significant.
The 'official interpretation' of the questionnaire presupposes a network of logical relationships between different configurations of answers, which depends on the meaning attached to them. Specific control questions are usually designed to evaluate the adequacy between those relationships of logical consistency and the actual answers produced by the subjects. The idea behind those control techniques is to foist on the 'official interpretation' as the only acceptable one, an to discard all the others as 'measurement errors'. But inconsistencies may be understood in a different light. Not as 'errors', but as anomalies, which would play a fundamental heuristic role. Which is the difference between an 'error' and an 'anomaly'? Both are expressions of some mismatch, in the course of a research process, between actual results and expected results. Now this mismatch is interpreted as an 'error' when it is attributed either to the subject of knowledge or to her instruments of knowledge. In the first case, we may name it 'subjective error', and in the second case we may talk of 'instrumental error'.
Let us focus our attention on this second kind of error. If we believe that the cause of an instrumental error is the lack of precision of our measurement instruments, we think of it as a 'measurement error'. To the extent that we view any mismatch between expected and actual results as an effect of measurement errors, we are protecting our theories by means of a sort of sanitary belt which makes them immune to any refutation by reality. We can always say that the poor quality of our measurement instruments is entirely responsible for any mismatches between expected and actual results that might appear through the research process. On the other hand, we may conceive those mismatches as 'anomalies'.
An 'anomaly', contrary to a 'measurement error', is not a phenomenon affecting the instruments of knowledge. It is a phenomenon occurring in the very object of knowledge, that is, in reality itself. An anomaly indicates that reality does not behave according to our theoretical assumptions, but in an entirely unexpected way. The detection of an anomaly implies two things: first, the recognition of some dose of 'subjective error' on the part of the researcher. Second, the transformation of a certain amount of the referred mismatch between expected and actual results, from the condition of 'measurement error' into the status of new phenomena, previously ignored by the researcher. In a way, the advancement of modern science may be conceived as a process in which purported 'measurement errors' are reinterpreted as 'anomalies', and hence as the basis for new theoretic developments.
When this perspective is applied to the analysis of questionnaires, we can distinguish two basic attitudes among survey researchers: the classical, positivist attitude of those who project onto the questionnaire their 'official ontology' and 'master interpretation', by accepting a strict version of the 'equivalence principle'; and the nonclassical, postpositivist view which would conceive the questionnaire as a heuristic instrument, as a tool generating a condition of resonance between the idiosyncratic ontologies and interpretive capabilities of different sorts of people --including the researcher itself. A condition of resonance that is only partial: each subject involved would add her own peculiar overtones to it. The positivist attitude would debase, in an epistemological sense, any apparent inconsistencies present in the responses to the questionnaire, by considering them as the outcome of simple 'measurement errors'.
The postpositivist perspective would upgrade those inconsistencies to the epistemological status of anomalies, and hence this perspective would conceive them as substantive phenomena, which remain unexplained within the ontological frame presupposed by the questionnaire. These two opposed views favour two different strategies concerning the use of questionnaires as a research tool. While the positivist perspective tries to minimize 'error' through the minimization of the interpretive freedom of the respondents, the postpositivist stance would foster that freedom, in order to increase the interpretive overtones present in the process of semantic resonance that the questionnaire would implement. Thus, instead of avoiding ambiguity, the designer of the questionnaire would take advantage of it. Instead of presupposing congruency, the researcher would look for incongruencies among those respondents and among their respective answers.
The methodological negation of the positivist assumptions would lead to a conception of questionnaires very different from the usual one. A conception according to which a questionnaire would not be a meeting point for objective, transparent and universal meanings, but a tool for the refraction of the diversity of social meanings which dwell in linguistic expressions. A tool capable of distinguishing between the different perceptions that individuals and groups produce about reality. These concurrent perceptions, qualitatively different in nature, coexist unnoticed in the answers to the questionnaire, and generate several types of apparent inconsistencies and tensions in it.
5. AN EXAMPLE OF SEMANTIC ANOMALY.
How can those differences, present in the data but neglected by the usual interpretive techniques, become visible? To start with, anomalies can be detected in an intuitive way. For instance, in a survey on elderly people from Gijón (Asturias, Northern Spain), several of those anomalies became apparent. A substantial majority of the respondents to the questionnaire (75%), asked to name three main problems in their lives, identified 'financial problems' as their first cause of concern. And yet, only 12.4% answered, in response to another question, that their main need was to have money. This lack of correlation between the answers to two questions seemingly close in semantic content, is astonishing. It shows that the content given by elderly people to the expressions involved may be different from the meaning usually attached to them by younger people.
This interpretation is supported by another survey on elderly people from Avilés, a town near Gijón. When examining the answers to this second questionnaire, other puzzles became apparent. Thus, the size of the houses and flats in which elderly people live does not correlate with their income level. The smaller sizes correspond to the lower and to the higher income levels, whereas individuals with intermediate levels of income live in larger places. The presence of this sort of apparent inconsistencies seems to point to some reality which is being overlooked by the 'official ontology' of the researchers, but that the individuals surveyed would express in their answers, albeit in an indirect, cryptic way. On closer inspection, this missing aspect of reality turns out to be that money does not seem to play the same role for the elderly as it does for younger people. For younger people, money is chiefly a consumer commodity. It is used mainly to purchase material goods.
Among elderly people, money is also a means to buy those goods, of course. But it plays another role: it is a means to underpin their social acceptance as old people; it is a way to maintain their basic social links --otherwise endangered--, an instrument to buy social security, in the literal sense of the expression. Now, that social acceptance has, as its natural environment, the domain of family relations. For the elderly, money is a commodity that allows them to be, on the one hand, financially independent with respect to their children and, on the other hand, valuable relations for those children. In point of fact, the financial flows between generations seem to go more from the elderly to their children than the other way around: according to the Avilés survey, 37.6% of the elderly suggest that they help their children financially, whereas only 2.7% would recognize that they receive financial support from them --certainly, this last figure is suspect of underestimation on the part of the elderly.
A substantial aspect of this financial help to the younger generation is the selling of the house or flat owned by the older generation, who buys then a smaller place to live. Marketable houses --whose owners are, in general, rather well off elderly people-- are more likely to be sold. So, in the light of these considerations we should redefine the ontology presupposed by the questionnaires, in order to include --and hence, in order to explain-- what looked as anomalies in the answers: when elderly people think of 'financial problems' they are referring to something different from simple 'need of money'. Whilst 'need of money' would refer to 'need of money as a consumer', the notion of 'financial problems' would point to their need of 'strategic money': the money that, as a potential or actual financial flow towards the younger generation, may be instrumental in maintaining the social status of the elderly within the domain of family relations.
6. CONCLUDING REMARKS.
Besides this kind of intuitive analysis of the anomalies that appear in the responses to questionnaires, some other methods could be available in order to visualize the semantic diversity present in surveys. As a general rule, the more alien the subjects of the research are regarding the 'social ontology' that the researcher represents, the more likely it is that their answers appear as anomalous and inconsistent. Similarly, the wider the sociological scope of a survey, the higher its semantic diversity is likely to be, and the less the 'equivalence principle' will work. For this reason, most statistical techniques that presuppose the 'equivalence principle' and aggregate equal expressions as identical in meaning, are unable to detect the semantic differences in question. Nevertheless, some kinds of correspondence analysis may be capable of revealing those qualitative distinctions in meaning, by defining, for instance, a measure of 'sociosemantic distance' between the items of the questionnaire .
REFERENCES.
Cano Lorenzo, S., (1990), La vejez: Integración o exilio, Gijón: Fundación Pública de Servicios Sociales.
Arenas Martínez, M., (1995), Las personas mayores de Avilés. La generación de la emigración, Avilés, Excmo. Ayuntamiento de Avilés.
Díaz-Martínez, C., (1992), Sociosemantic Analysis: a bridge between quantitative and qualitative techniques, paper presented at the International Conference of Social Science Methodology, Trento (Italy).
