A culture of mistrust
In her sanitation projects, Molly Lipscomb worked closely with Water and Sanitation for Africa, a multinational organization with broad support from national governments across the continent. For Isaac Mbiti, however, cooperation from different levels of government was at times a little harder to find.
Mbiti was researching the outcomes of wide-ranging educational reforms in Kenya, including the decision to abolish all school fees in public primary schools. At first, media skepticism of the proposed change was widespread, with “apocalyptic newspaper stories saying the educational system was going to collapse.”
The collapse never happened; Mbiti’s research indicated that the reforms largely worked. Poor students got access to education, often for the first time. The reforms also spurred increases in private schooling among well-off families. Despite all these changes, there were minimal declines in test scores, suggesting that the program succeeded without the significant reductions in learning levels that had been widely predicted within the media.
Still, getting the vital data was a constant challenge, as is common across the developing world.
Often, researchers find a “culture of mistrust” from government officials, a fear that researchers will make the government look bad or that researchers will contradict official government statistics. As a result, taxpayer-funded data is not available to the public.
At other times, the data simply does not exist. And then, Mbiti goes to work, collecting his own survey data. This often involves implementing randomized controlled trials or, in some cases, augmenting data with additional sources. For instance, in his study that examined the free primary education program in Kenya, Mbiti had to assemble a combination of household survey data, census data, school-level information and individual test score results from the entire country in order to assess the effects of the program.
Another time, Mbiti and his collaborator designed a highly targeted study combining administrative data on test scores with surveys to find “precise answers to precise questions” about Kenya’s network of elite “national schools.” Do the national schools actually deliver better outcomes, or do their students excel because only the best students are allowed in?
As good research often is, the results were surprising.
Kenya’s prestigious, secondary-level national schools admit the best students from each region of the country. Only about 3 percent of students have test scores high enough to qualify. These prestigious schools spare few expenses, with elaborate facilities and state-of-the-art computer labs. Some even boast of airplanes and landing strips, to train students in aviation.
Mbiti’s study targeted students on both sides of the test-score admission threshold, comparing the outcomes of students who were barely admitted to those who barely missed. By narrowing the focus to closely matched students, Mbiti could evaluate the actual impact of the elite schools in test-score outcomes.
“When we rigorously analyzed this, we found that these schools basically did nothing,” Mbiti says. By the end of their secondary schooling, the students who just missed getting into national schools did just as well as the students who barely got in.
“We were very surprised. When you look at the difference in facilities between these schools and the other secondary schools, it’s massive, but it did not translate to better test scores.” This is consistent, Mbiti points out, with a number of studies showing that—in the absence of reforms in accountability and teaching methods—merely increasing school inputs does little to improve learning outcomes in developing countries.
In these research projects, which ranged over years, Mbiti and his colleagues battled media assertions that public-school reform had failed, as well as central government assumptions that elite national schools had triumphed. Sometimes it falls to policy researchers to do the hard work of skewering intuition—of telling people that the bad news isn’t that bad and the good news isn’t that good.
For such times, the right data and rigorous analysis are essential.