What's stopping knowledge synthesis? A systematic review of recent practices in research on smallholder diversity

A systematic review of recent publications was conducted to assess the extent to which contemporary micro-level research on smallholders facilitates data re-use and knowledge synthesis. Following PRISMA standards for systematic review, 1,182 articles were identified (published between 2018 and 2020), and 261 articles were selected for review in full. The themes investigated were: (i) data management, including data source, variables collected, granularity, and availability of the data; (ii) the statistical methods used, including analytical approach and reproducibility; and (iii) the interpretation of results, including the scope and objectives of the study, development issues addressed, scale of recommendations made relative to the scale of the sample, and the audience for recommendations. It was observed that household surveys were the most common data source and tended to be representative at the local (community) level. There was little harmonization of the variables collected between studies. Over three quarters of the studies (77%) drew on data which was not in the public domain, 14% published newly open data, and 9% drew on datasets which were already open. Other than descriptive statistics, linear and logistic regression methods were the most common analytical method used (64% of articles). In the vast majority of those articles, regression was used as an explanatory tool, as opposed to a predictive tool. More than half of the articles (59%) made claims or recommendations which extended beyond the coverage of their datasets. In combination these two common practices may lead to erroneous understanding: the tendency to rely upon simple regressions to explain context-specific and complex associations; and the tendency to generalize beyond the remit of the data collected. We make four key recommendations: (1) increased data sharing and variable harmonization would enable data to be re-used between studies; (2) providing detailed meta-data on sampling frames and study-context would enable more powerful meta-analyses; (3) methodological openness and predictive modeling could help test the transferability of approaches; (4) more precise language in study conclusions could help decision makers understand the relevance of findings for policy planning. Following these practices could leverage greater benefits from the substantial investment already made in data collection on smallholder farms.