by Molly Hanrahan, Monitoring & Evaluation Intern, and Merritt Frey, Executive Director
In this post, we explore this year’s DEF exam results and their variation between and among our 21 (soon to be 23!) schools. In a companion post last week, we discussed the wide fluctuation in results on the DEF exam from year to year.
As described last week, the DEF exam in Mali tests 9th grade students as they graduate from middle school. If students pass the test, they are eligible to continue on to higher education. If they fail, they must redo 9th grade until they pass. This reality combined with the fact that the DEF test results are the only true data measure we have of how much our students are learning means that it’s incredibly important to us to have strong pass rates.
Looking at this year's DEF results we see something we've seen in previous years -- an incredibly wide variation in pass rates among our schools. Two schools did incredibly well with 98% of their students passing; two schools did incredibly poorly with 0% of their students passing. Nine of our 19 middle schools beat the average national pass rate this year (70%), while 10 did not.
“The more textbooks per student (i.e. a lower ratio), the higher the DEF pass rate. Not surprising, right? But seeing the hard numbers makes our campaign to invest in textbooks that much stronger!”
Seeing this variation year after year is frustrating! What could account for such a difference in results at the schools? As any parent or teacher in the U.S. knows, this is an enormous question -- why do some schools thrive and others struggle? There are so many inputs and factors to consider...and there is always the question of just how meaningful any test result is (or is not).
Despite the daunting nature of the question, we want to understand the variation so we can help our struggling schools. We analyzed DEF pass rates against some of our other data to see if we could find an "a-ha!" moment to account for the wide variation in DEF pass rates at our schools. We found DEF scores don’t have a strong correlation with gender ratios, school age, or even -- surprisingly -- class size.
One intriguing hint of a connection did surface -- textbooks. We found a moderate negative correlation between DEF pass rates and a school's ratio of textbooks to students. The more textbooks per student (i.e. a lower ratio), the higher the DEF pass rate. Not surprising, right? But seeing the hard numbers makes our campaign to invest in textbooks that much stronger!
To truly figure out the DEF scores, we believe we will have to go deeper in our analysis. We can try looking at specific details, like the number of French textbooks a school has, or we can try layering different factors—maybe good DEF scores are correlated with a combination of small class size and good book ratios. We are also excited to look at the differences between the schools who do well and those who don’t -- we want to learn from our "bright spots." We will also continue to discuss the scores with our staff and teachers in Mali, who will most likely have the best insight into the variations in scores.
Of course, the score variation might be due to something that is really hard to quantify, like teacher quality. Or they may be due to other factors that we don’t currently measure, like attendance, the teacher turnover rate, etc.
But we'll keep asking questions, collecting data, and sharing our analysis. Our generous supporters invest in education as a way to empower Mali's children, and we will do all we can to make sure that education is delivering all it can for our students. Stay tuned to learn along with us...
