Monitoring, Evaluation, Research and Learning (MERL) for Language Issues in Reading Programs

This is the fourth and final blog post in our series on language of instruction. In this post, we review important considerations for monitoring and evaluation in reading and language programs. 

Monitoring, evaluation, research, and learning (MERL) is an essential component of reading programs to help improve teaching and learning practices and outcomes. USAID’s Reading MATTERS Conceptual Framework provides practitioners with guidance when designing MERL plans and conducting analyses. 

Language-focused MERL activities play an important role in gathering evidence and garnering support for adopting languages children use and understand outside of school for reading instruction. Effectively designed MERL can guide policy formation, promote changes for improved teaching practices, and inform curriculum and materials development within reading and language programs.

Developing a MERL Plan
Given the importance of high-quality MERL for reading programs, The Handbook on Language of Instruction Issues in Reading Programs, developed by USAID and the Global Reading Network, presents key steps that programs should take in developing a comprehensive and appropriate MERL plan that integrates language issues:

• A MERL plan should be developed at the same time the program is being designed rather than after. This will ensure that the design is driven by stakeholders and that the desired MERL activities will be possible. 
• Specialists knowledgeable about language issues should be involved in MERL plan development to help identify the language-specific issues to monitor and evaluate and to contribute to the design and development of MERL questions, indicators, tools, and data collection methods. Language-specific issues to consider include the use of language in the classroom and children’s reading outcomes. 
• The plan should integrate cost analysis as part of the monitoring and evaluation process. Collecting cost information has important implications for modifying, maintaining, expanding, and/or sustaining interventions. Cost analysis should also include a comparison to the typical approach to teaching reading and language without programming to better understand the cost-effectiveness of program interventions.

• The MERL plan should call for disaggregated results of student learning data by language and other sub-populations (gender, geographic area, refugee status, and disability status). This should include a comparison between students who are learning to read in the language they use and understand and those who are learning to read in an unfamiliar language. Disaggregated results are important for helping programs identify gaps in reading achievement between groups of students and how the languages of instruction may be benefiting or hindering students’ acquisition of reading skills. 

Fidelity and Quality of Implementation
Monitoring the fidelity and quality of implementation are critical components of a MERL plan in a language-focused reading program. Monitoring the fidelity of implementation helps ensure that programs are implemented as they are intended to be. If a program is not implemented as designed, it can lead to discrepancies between what is being implemented and what MERL activities are actually evaluating. 

Closely tied to the fidelity of implementation is the quality of implementation. Programs need to monitor not only that activities are being implemented as intended but also that the implementation is high quality. Monitoring this regularly allows programs to identify and address gaps in programming quality.

These gaps in fidelity and quality of implementation can provide important insights and MERL opportunities. When faced with gaps, programs can design studies to explore why such gaps exist. This can lead to programmatic learning and help identify other issues that may hinder the effectiveness of the reading interventions. In fact, “How can countries most effectively address issues related to language of instruction to improve learning for all students?” is a key learning question in the USAID Education Foundational Skills Learning Agenda. This understanding can then inform modifications and adaptations to make interventions more effective.  

Quantitative and Qualitative Approaches
According to the Handbook, MERL plans should include both qualitative and quantitative approaches to collecting monitoring data to develop a more robust understanding of what aspects of a program are working and what needs to be adjusted. Quantitative data to gather includes:

• The change in the amount of instructional time used to teach reading in specific languages over time.
• Changes in children’s reading and language skills. 
• Changes in teachers’ instructional practices for teaching reading and language. 

Qualitative methods can help dive deeper into these areas and extend understanding about the quality of the programming, challenges, and successful implementation practices. For example, qualitative research might consider questions such as the following:

• What is the quality of instruction provided?
• In what areas do teachers experience challenges with implementing the reading program? Why are they facing these challenges?
• In what areas is the reading program successful? What factors help facilitate this success?

To collect both qualitative and quantitative data, programs need to develop tools—such as questionnaires, surveys, language assessments, and classroom observation instruments—to facilitate data collection. USAID’s Foundational Skills Development Learning Agenda includes other important questions that programs should consider including in their research agendas to better understand issues related to language of instruction. 

Developing a MERL plan takes careful planning to ensure that it’s comprehensive and appropriate to the program and context. MERL plays an important role in identifying and evaluating language issues in reading programs and informing ways for improving and adapting approaches and activities. Investing the time and energy to design a thoughtful MERL plan as a program is under development can lead to important implications for improving policies and programming as well as for identifying areas where further research is needed to ensure that programs are using effective, evidence-based approaches.