Measurement, Data Generation, and Use: Takeaways from CIES 2019

Exciting new approaches for how we measure our efforts, track progress, and apply evidence and learning in the Education sector are on the horizon. This year's gathering of Education specialists at the Comparative International Education Society (CIES) 63rd conference showcased promising trends in this area. 

Every spring, the CIES brings together thousands of educators, researchers, donors, and foundation representatives from over 100 countries to share research findings, approaches, dialogue, and learn from each other. This year’s theme was “Education for Sustainability.” 

Here are a few takeaways from some of the conference’s forums. 

Better Data for SDG4
The United Nations Sustainable Development Goal 4 (SDG4) seeks to ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.

Having the right data to track the progress toward achieving SDG4 is essential. Unfortunately, there are many data gaps among the 43 indicators associated with SDG4, but efforts are underway to close them. A couple of notable measurement tools are included below.

The UNICEF Multiple Indicator Cluster Survey (MICS) child questionnaire includes learning and child functioning modules that provide childrens' reading and numeracy proficiency rates and disability prevalence statistics. MICS and other household-based surveys are generally only conducted every 3-5 years, but models can be used to estimate current school completion rates by pulling in data from different independent surveys conducted at different points in time.

Models can also be used to estimate adult literacy rates using Programme for the International Assessment of Adult Competencies (PIAAC) data.

Session Materials: Better data for SDG4: Recent methodological developments

Measuring Costs of Education Programming
In support of the 2018 USAID Education Policy, USAID’s Office of Education collects, analyzes, and uses comprehensive cost data on USAID education programs to improve accountability, transparency, value for money, and successfully transfer investments in international education to host government ownership.

Despite a general consensus in the donor community on the need for program cost data and analysis to help measure the effectiveness of education programs, there’s still a lack of agreement for one primary approach on how to achieve this. USAID has been working on a coherent approach to improve both value for money of their investments and provide host governments with the necessary information for sustainment of effective interventions. 

USAID’s Cost Reporting Guidance puts forward a method for collecting cost data of donor-funded education activities in order to inform program design and better understand cost economy, effectiveness, and sustainability across USAID’s global portfolio. This approach, along with the methodology for cost analysis developed by USAID, lays the foundation for the new Cost Measurement Guidance Note produced by the Building Evidence in Education Donor Working Group. The Note is designed to harmonize how funders of international education projects collect and analyze cost data to enable systematic evidence building on the cost-effectiveness of donor-funded programming across the entire sector. 

Session Materials: Cost effectiveness analysis for sustainable education policy, What does value for money mean when it comes to educating a child?, and Cost measurement of USAID education programs: Practical guidance

Measuring SEL: A Systems Approach to 21st Century Skills 
Social and emotional skills, or soft skills, are essential for success in school, career, and life. Students who participate in school-based SEL programming perform measurably better in school than peers without exposure to SEL. 

As socio-emotional learning (SEL), or soft skills, has gained prominence in recent years, there’s a greater consensus that a child’s social and emotional skills correlate with their ability to learn in school. It’s also an important predictor of future success in life. The USG Basic Education Strategy and 2018 USAID Education Policy promote SEL along with literacy and math. 

Several donors and implementing organizations shared their lessons learned from taking on the challenge of measuring social and emotional skills:

• Instruments must be adapted locally to ensure that cultural norms, behaviors, and skills are reflected in these instruments, not just language.
• An iterative process of cognitive testing, piloting and revising is needed to develop effective measures.
• UNICEF has worked with partners to develop a skills framework. UNICEF and the World Bank are currently developing a companion assessment tool.

Session Materials: Reimagining life skills: UNICEF’s global framework on skills, from theory to practice & Measuring soft skills: concepts, methods and results

Setting Minimum Proficiency Standards
Social Moderation (or the policy linking method) was introduced as a non-statistical method that might offer value in linking national assessments to USAID indicators and other global standards, helping to compare and aggregate results of childrens' reading and mathematics assessments across and within countries.

Policy linking is a non-statistical, rigorous method for linking education assessments. It uses expert judgment to directly match performance levels on assessments. USAID has pursued this approach to measure indicators under its new Education Policy. Policy linking will link national and cross-national assessments to a global scale of minimum proficiency in reading and mathematics.

USAID will pilot this method in two countries in the coming few months. The method is an improvement over the status quo since there is no other psychometrically acceptable method in use that can compare national and cross-national assessments to a global standard. 

Session Materials: Linking assessments to a global definition of minimum proficiency using policy linking

Innovations in Early Grade Reading Benchmarks
Oral reading fluency (ORF) has become the popular proxy measure for childrens' overall reading abilities and for the quality of an education system.

The processes for setting oral reading fluency benchmarks are typically data-driven or content-focused. Data-driven approaches use mean, median, and/or regressions of student assessment data to set benchmarks. In contrast, content-focused methods, such as modified Angoff, consider the items on the assessment to create benchmarks. 

Data-driven methods and content-focused methods will yield different benchmarks. Regardless of the approach used, the stakeholders’ application of the benchmarking exercise helps legitimize the benchmarks and should help create the political will needed to achieve benchmark targets. 

Session Materials: Rethinking early grade reading benchmarks: Innovations in setting oral reading fluency standards

Utilizing Data for Collaborating, Learning, and Adapting (CLA) 
USAID’s Collaborating, Learning and Adapting (CLA) approach involves using strategic collaboration, continuous learning, and adaptive management to improve organizational effectiveness and development results.

It’s not sufficient to just collect data. We need to work actively to integrate the utilization of data into decision-making. It’s also important to ensure that actionable data is being presented in a way that is tailored to the purpose and to the individuals who will be using them.

The CLA framework is purposefully flexible, and different projects have been able to adapt CLA to their particular projects and context. 

For future sustainability, it’s beneficial to work closely with ministries to develop a design that is integrated into existing systems. 

Session Materials: Data use for improved programming and sustainable outcomes: Case studies in applying the collaborating, learning, and adapting (CLA) approach

Laying Foundation for Local Data Ownership: What Does it Take?
USAID’s pivot toward supporting our partner countries on the journey to self-reliance means that our programming is increasingly driven by the gaps in capacity and commitment of partner governments. Such capacity and commitment with regard to education data collection, analysis and utilization are central toward ensuring sustainment of USAID investments. 

USAID is shifting focus from donor-funded monitoring systems designed to inform upstream donor reporting to government-managed monitoring systems that drive the development of robust education data eco-systems at the country level. A country's journey to self-reliance is not possible without strong data systems and the government's commitment and ability to use education data for policy and fiscal decision-making. 

There are significant challenges that countries face in their efforts to improve education data systems. Existing organizational structures and process are frequently designed to obfuscate rather than facilitate the use of data in decision-making. Education data are inherently political and can be used to promote agendas of some groups to the exclusion of others. It is essential to consider the political economy of the data life cycle when attempting to improve data and evidence uptake. 

Session Materials: Better M for better E: What is the price of data ownership?