Rwanda Pilots an Improved Benchmarking Method

USAID and its partners have used many different methods to set performance benchmarks for primary school students over the years, with the vast majority of benchmarking exercises relying on student reading assessment data to identify proficiency levels. However, this is not best practice in developed countries, which typically use a method sometimes referred to as policy linking. Recently, in Rwanda, this method was piloted in the USAID Soma Umenye project. 

According to Sharon Haba of USAID Soma-Umeneye, formerly with Rwanda’s Ministry of Education, there are several reasons why the Government of Rwanda decided to give this benchmarking method a try. There’s currently a greater emphasis on early grade literacy and numeracy, a desire to have a strong basis for evaluating student performance, and they want to measure progress against the Sustainable Development Goals. 

“We wanted to have a strong basis and move away from very general statements like ‘kids are not reading the way they should be reading’ or ‘they move from one grade to another without the necessary skills.’ We did not have strong and specific, teacher-owned and education-system-owned benchmarks,” said Haba during a recent presentation in Washington D.C.  

Developing New Benchmarks: The Process
Policy linking requires that experts select the best method for setting cut scores based on the type of assessment a country uses. In Rwanda, experts selected the Angoff Method to help test creators determine the percentage, or “cut score,” necessary for a student to be considered proficient. The method relies on subject-matter experts who examine the content of each test question and then predict how many minimally-qualified candidates would answer correctly. 

In Rwanda, the modified Angoff Method was developed in three phases. In the first phase, Ministry of Education learning assessment specialists came together to discuss how many performance categories there should be and how to categorize these. The discussions led to four categories to describe pupil performance in all subject areas, including reading proficiency:  

1) Does not meet expectations

2) Partially meets expectations

3) Meets grade level expectations

4) Exceeds grade level expectations

In the second phase, reading curriculum specialists defined what performance looked like for each performance category from first to third grade. “Having accurate descriptors and identifying the borderline pupil or the point-of-reference child are critical steps to getting this improved method right,” said Norma Evans, an education expert supporting the use of the modified Angoff assessment method in Rwanda.

In the final phase, master teachers and curriculum specialists established benchmarks for oral reading fluency. Master teachers are early primary school teachers who have well-established reputations, are currently teaching, and have at least five years experience teaching Kinyarwanda in public schools. For the very first time, these teachers collected data on reading fluency. Their task was to identify the “borderline pupil,” or the student on the cusp between “partially meets expectations” and “meets expectations” based on the descriptions they developed for each of the levels.  

They did this by first estimating what score a borderline student would achieve and then conducting a fluency assessment of the students’ reading level for grades 1 to 3 to check their accuracy. 

Results and Implications for Rwanda
The previous oral reading fluency benchmark for third graders in Rwanda was the ability to read and comprehend 33 correct words per minute. With the improved method, the new draft benchmark is now set at 41 words per minute, which raises the bar for both Rwandan students and teachers but also brings the country more in line with regional benchmarks, thus making Rwanda more competitive in the long run.

Because the decision-making process is rooted in local expert knowledge, there’s a high level of confidence in the new benchmarks’ accuracy. The master teachers who participated in developing these standards are now acting as local champions. They are not only able to explain and defend them, but can also teach others how to monitor progress toward them.

The Government of Rwanda has ownership of the new benchmarks and standards because they were involved in the development process from the very beginning. 

“The Ministry of Education will now have something that can be used throughout the country, has been developed by teachers, and in a language that everybody can understand. This tool will help the Government of Rwanda place children in categories where they can be properly supported,” said Haba. 

Recommendations for Setting New Standards in Other Countries
When it comes to setting new performance standards in other countries, first look at the lessons learned from Rwanda. Developing new benchmarks should start toward the end of the school year when teachers have a better sense of their students’ abilities and performance levels. Also, government ownership and use of new benchmarks are more likely to happen if the Ministry of Education is involved in creating new standards from the start. 

When selecting master teachers, they should be chosen carefully with input from the local community. “It is critical to value and credit the knowledge that local teachers bring into developing new benchmarks,” said Evans. “This will go a long way in strengthening local capacity and building teacher confidence.” She also added that teachers should be encouraged to assess a wide range of pupils before setting fluency standards.  

Rwanda clearly demonstrated that involving assessment specialists from various subject areas is more likely to result in building systemic capacity to set standards, thus making the education sector overall stronger.

Lastly, countries and USAID Missions should consider using the new Global Assessment Framework (coming soon!) established by a multilateral group of reading and math experts, led by USAID, in June 2019 to ensure country benchmarks are linked to USAID Foreign Assistance indicators and the Sustainable Development Goal indicators.