One question, two answers: What happens when different methodologies produce different results?

The case of women’s empowerment, food security and nutrition of pastoral communities in Tanzania

By Alessandra Galiè, senior gender scientist, ILRI

In our research on livestock for better livelihoods we often argue for mixed-method approaches: Combining quantitative and qualitative research methodologies will, we think, enhance the quality of our research and the validity of our findings. But what happens when the two methods lead to conflicting results?

Take, for example, a complex research problem we are currently addressing: How does the empowerment of women in livestock systems correlate with the food and nutritional status of their households? Do empowered women live in households where food is more available and where household members are less under- or mal-nourished—compared to households with less empowered women?

Qualitative researchers will tackle this question by talking in depth with a small number of women and men from livestock communities to explore what they themselves understand empowerment to mean, how they think women’s empowerment can affect the availability of food within the household, and how women’s empowerment may affect the way food is prepared and consumed in the household (which influences nutrition). They look to establish how local communities understand and experience the link between women’s empowerment, nutrition and food security.

Quantitative researchers, on the other hand, will typically conduct large-scale surveys. The surveys will ask specific questions to determine the degree of women’s empowerment, the availability of food throughout the year within the household and whether they and their families eat enough nutritious food. In addition, the researchers may take height or weight measurements to assess the incidence of under- or malnutrition. Back in the office, they will seek to quantify the extent to which empowered women live in households that are food and nutrition secure.

Results from the two methods typically inform and elucidate each other. Qualitative work can provide an initial exploration of what livestock keepers consider as key factors affecting women’s empowerment as well as food and nutrition security of their households. Quantitative work can determine if and how far the qualitative findings are applicable to a wider population. Qualitative work can then explore in greater detail the hows and whys of the findings from the quantitative work.

If the quantitative data show, for example, that empowered women generally live in households that are more food secure but where the nutritional status of the girls is low, a follow-up qualitative study may explore: ‘Why does this happen and how?’ More quantitative research can then be undertaken to test the wider relevance of the new qualitative finding — in an alternation between the two approaches that builds solid evidence.

This complementarity of methods generally works well—provided the two approaches reinforce and strengthen each other. But what if the findings simply don’t match? What if the quantitative analysis shows that an increase in women’s empowerment does not correspond with greater food security—while the qualitative study shows that it does? And what if no amount of additional analysis using either method leads to a convergence of results? What data do we think is more accurate, the qualitative or the quantitative? Which approach do we trust as more reliable? How do we reconcile the mismatch? Mixed-method approaches are often recommended in research—but rarely do researchers discuss what to do if the findings from the two approaches contradict each other.

This is what happened in our 2015 study on the relation between women’s empowerment and household food and nutrition security in 373 livestock households in Tanzania. Both the quantitative and qualitative findings showed, reassuringly, that households with the more empowered women were also more nutrition secure. However, the quantitative findings showed that the empowerment of women had no effect on the food security of their households. (Food security here refers to the availability of food in the household, while nutrition security refers to whether this available food is nutritious, and is prepared and consumed in ways that result in a good nutritional status of a given individual.)

The qualitative findings, on the contrary, showed that households with more empowered women were also more food secure. The women participating in the qualitative study explained that if they were more empowered, they would control the milk of their household and sell it to earn cash which they would then invest in food. By contrast, they told us, when the husbands controlled the milk and the revenue it generated, the men would spend it on priorities other than food—including also beer for themselves.

In trying to make sense of these contradictory findings our research team decided not to debate which methodology is per se more reliable. These discussions inevitably begin and end with polarised beliefs about what knowledge and science are—with no way out.

Rather, we decided to engage with this difference to improve our study. We examined and re-examined the approach we had adopted in each methodology. For example, we tested whether the results we had obtained would change depending on whether we were studying extensive or intensive livestock systems. But this analysis only confirmed our initial results, increasing the validity of our findings.

Next we questioned whether the definitions of food and nutrition security that we had used in the two studies were compatible. We realized through the qualitative findings that the participants thought of food security as a responsibility society assigns to men (who need to provide enough cash for food to be available) while nutrition security is a responsibility society assigns to women (who need to ensure the provided food is prepared in nutritious ways for all household members). In the qualitative study women expressed their concern over the difficulty in providing nutritious diets while not having control over household food expenditure, as this was the men’s responsibility.

Through greater control of food expenditure, women’s empowerment was linked positively to food security. The quantitative study, on the other hand, did not include such a distinction of roles and, as a consequence, respondents may have provided answers (for example on decision-making about food purchases) that reflected the local norm about men being in charge of food security – possibly under-reporting women’s role (see Galie and Farnworth 2019 on the ‘gender norms façade’). Also, the measurement of food security did not consider the within-household decision-making on food availability and access. The distinction between food and nutrition security in regard to gender roles also became an important finding in its own right and has helped us better understand other ongoing studies on the topic.

As we deepened our analysis, we also came to question the definition of empowerment that the two approaches had adopted. We used the new insights to recommend a new methodological approach to exploring women’s empowerment: one that combines both universally (such as, for example, control over productive resources) and locally (such as social status) accepted domains and indicators of empowerment. It needs to be assessed whether this approach may help to provide more converging results. Nonetheless, such an approach would produce a truly mixed methodology (by bringing in both quantitative and qualitative findings in the data gathering phase of the study – not only at the analysis stage); one that explores the complexities of empowerment more comprehensively because grounded in domains of empowerment that are accepted across cultures, and also in ones that the respondents consider relevant to their context.

In short, when we realized that our qualitative and quantitative findings disagreed, we engaged with the contradiction and used it to revisit and strengthen our methodology and findings. We did not shy away from engaging with the contradictions, nor did we try to paper them over, nor did we decide a priori which methodology to trust. Our deeper engagement with the findings and methodology resulted in an enriched analysis and understanding of our research topic. It also helped our team appreciate the strength of each approach and learn how we can collaborate even more effectively as a team in the future.

Among Alessandra Galiè’s recent co-authored articles are ‘Power through: A new concept in the empowerment discourse‘, in Global Food Security 2019; and ‘Women’s empowerment, food security and nutrition of pastoral communities in Tanzania,’ also in Global Food Security 2019 (see here for a ILRI News discussion of the paper).