A process evaluation tells you whether your program is being implemented as expected. In other words: process evaluations help you see whether the assumptions in your theory of change holdup in practice.
A process evaluation can help you:
· Track whether the iron supplements you purchased are reaching the people they’re meant for
· Assess whether farmers are remembering the lessons from their extension trainings
· Learn whether students are using the computers that you distributed for classroom instruction
“This sounds a lot like a monitoring system. How is a process evaluation different?”
Which aspects of program design are not working as planned?
Process evaluations often identify elements of program design that should be modified based on gaps observed in the field. For example, you may run a program that trains veterinary health workers to provide services such as vaccination and deworming to all livestock farmers. A process evaluation may tell you that veterinary health workers are only selling to medium- and large-sized livestock farmers. As a result, you may consider adding monetary incentives for reaching smallholders.
Which version of my program should I use?
Process evaluations can help you identify which version of a program is more effective at delivering services. For example, say your program aims to notify parents when a vaccination camp is taking place in their village. You run a process evaluation to decide whether to send SMS reminders or broadcast information over the radio. The process evaluation reveals that most parents do not check their SMS messages regularly, but community radio is popular. You thus decide to broadcast information over the radio.
Can my program be salvaged?
A process evaluation may reveal that it is not possible to implement the program in accordance with the theory of change. For example, if an assumption critical to the success of your program does not hold, and is outside of your control, you may need to consider terminating the program. Say you have a program that offers doorstep banking services for women. Banking agents take deposits from customers and drop the money they collect at the bank daily. However, when you conduct a process evaluation, you learn that the roads have broken down after a recent storm, and rebuilding will take a while. It is not safe for agents to travel on these roads. In this case, you would need to consider terminating the program (or significantly changing how it is set up).
Which aspects of my programs do I need to monitor closely?
Process evaluations often point out areas for improved monitoring and oversight. For example, if a process evaluation for a program providing extension training to farmers reveals that you are not reaching smallholder farmers, you may want to start regularly collecting customer profile data to track program reach.
Doing a process evaluation involves two steps:
1. Assessing program implementation: Data on implementation is collected through activities such as direct observation of program activities, surveys of staff and participants, focus groups with participants, audits of documentation or monitoring data, etc.
2. Comparing these process descriptions to expectations: This determines whether what is happening on the ground is in line with the path to impact outlined in the theory of change.
Should you do the process evaluation yourself or hire an expert?
Organizations with the capacity to run surveys, conduct interviews, and analyze data can do process evaluations on their own; however, you may want to hire an external party to reduce the risk of bias.
1. Is it credible if I run the process evaluation in-house?
If you are conducting a process evaluation to meet high-stakes funder requirements, the evaluation would be more credible if it were conducted by an external evaluator, as the risk of bias is lower. If, however, you are conducting a process evaluation for internal learning, you can do the exercise in-house if you have capacity and are confident that you can collect unbiased data.
2. Does my staff have the necessary technical skills?
To run a process evaluation in-house, your organization should be able to collect quality data reliably. Your team will need to design questionnaires and responsibly survey implementers and/or participants on a variety of topics. We generally recommend hiring surveyors externally so as to minimize response bias.
Your organization should also be able to analyze qualitative and quantitative data. Qualitative data analysis requires the ability to organize high volumes of information and the ability to read interview answers with a careful eye. Quantitative data analysis may require only simple summary statistics in a spreadsheet program like Microsoft Excel, or more nuanced econometric analyses using specialized software like Stata.
If you do not have internal capacity to reliably collect, analyze, and interpret the data, you should work with an external party.
3. How much funding and staff time can I allocate to this exercise?
If you choose to conduct the process evaluation externally, you’ll save on staff time. Note that someone from your organization who is familiar with the program will still need to spend time working closely with the evaluator and supporting them in data collection.
A process evaluation tells you whether your program is being implemented as expected. In other words: process evaluations help you see whether the assumptions in your theory of change holdup in practice.
Examples
A process evaluation can help you:
· Track whether the iron supplements you purchased are reaching the people they’re meant for
· Assess whether farmers are remembering the lessons from their extension trainings
· Learn whether students are using the computers that you distributed for classroom instruction
“This sounds a lot like a monitoring system. How is a process evaluation different?”
Which questions can a process evaluation answer?
Process evaluations often identify elements of program design that should be modified based on gaps observed in the field. For example, you may run a program that trains veterinary health workers to provide services such as vaccination and deworming to all livestock farmers. A process evaluation may tell you that veterinary health workers are only selling to medium- and large-sized livestock farmers. As a result, you may consider adding monetary incentives for reaching smallholders.
Process evaluations can help you identify which version of a program is more effective at delivering services. For example, say your program aims to notify parents when a vaccination camp is taking place in their village. You run a process evaluation to decide whether to send SMS reminders or broadcast information over the radio. The process evaluation reveals that most parents do not check their SMS messages regularly, but community radio is popular. You thus decide to broadcast information over the radio.
A process evaluation may reveal that it is not possible to implement the program in accordance with the theory of change. For example, if an assumption critical to the success of your program does not hold, and is outside of your control, you may need to consider terminating the program. Say you have a program that offers doorstep banking services for women. Banking agents take deposits from customers and drop the money they collect at the bank daily. However, when you conduct a process evaluation, you learn that the roads have broken down after a recent storm, and rebuilding will take a while. It is not safe for agents to travel on these roads. In this case, you would need to consider terminating the program (or significantly changing how it is set up).
Process evaluations often point out areas for improved monitoring and oversight. For example, if a process evaluation for a program providing extension training to farmers reveals that you are not reaching smallholder farmers, you may want to start regularly collecting customer profile data to track program reach.
Table of Contents
About
A process evaluation tells you whether your program is being implemented as expected. In other words: process evaluations help you see whether the steps and assumptions in your theory of change holdup in practice.
Examples
A process evaluation can help you:
- Track whether the iron supplements you purchased are reaching the people they’re meant for
- Assess whether farmers are remembering the lessons from their extension trainings
- Learn whether students are using the computers that you distributed for classroom instruction
“This sounds a lot like a monitoring system. How is a process evaluation different?”
Which questions can a process evaluation answer?
Which aspects of program design are not working as planned?
Process evaluations often identify elements of program design that should be modified based on gaps observed in the field. For example, you may run a program that trains veterinary health workers to provide services such as vaccination and deworming to all livestock farmers. A process evaluation may tell you that veterinary health workers are only selling to medium- and large-sized livestock farmers. As a result, you may consider adding monetary incentives for reaching smallholders. Similarly, you may want smallholder farmers to make up at least 50% of the health workers’ clientele, but your process evaluation reveals that it is only 10%. Therefore, it helped you recognize that your program was falling short of a target during implementation, and you may now take up steps to rectify that.
Can my program be salvaged?
A process evaluation may reveal that it is not possible to implement the program in accordance with the theory of change. For example, if an assumption critical to the success of your program does not hold, and is outside of your control, you may need to consider terminating the program. Say you have a program that offers doorstep banking services for women. Banking agents take deposits from customers and drop the money they collect at the bank daily. However, when you conduct a process evaluation, you learn that the roads have broken down after a recent storm, and rebuilding will take a while. It is not safe for agents to travel on these roads. In this case, you would need to consider terminating the program (or significantly changing how it is set up).
Which aspects of my programs do I need to monitor closely?
Process evaluations often point out areas for improved monitoring and oversight. For example, if a process evaluation for a program providing extension training to farmers reveals that you are not reaching smallholder farmers, you may want to start regularly collecting customer profile data to track program reach.
Conducting a process evaluation
Doing a process evaluation involves two steps:
- Assessing program implementation: Data on implementation is collected through activities such as direct observation of program activities, surveys of staff and participants, focus groups with participants, audits of documentation or monitoring data, etc.
- Comparing these process descriptions to expectations: This determines whether what is happening on the ground is in line with the path to impact outlined in the theory of change.
Should you do the process evaluation yourself or hire an expert?
Organizations with the capacity to run surveys, conduct interviews, and analyze data can do process evaluations on their own; however, you may want to hire an external party to reduce the risk of bias.
1. Is it credible if I run the process evaluation in-house?
If you are conducting a process evaluation to meet high-stakes funder requirements, the evaluation would be more credible if it were conducted by an external evaluator, as the risk of bias is lower. If, however, you are conducting a process evaluation for internal learning, you can do the exercise in-house if you have capacity and are confident that you can collect unbiased data.
2. Does my staff have the necessary technical skills?
To run a process evaluation in-house, your organization should be able to collect quality data reliably. Your team will need to design questionnaires and responsibly survey implementers and/or participants on a variety of topics. We generally recommend hiring surveyors externally so as to minimize response bias.
Your organization should also be able to analyze qualitative and quantitative data. Qualitative data analysis requires the ability to organize high volumes of information and the ability to read interview answers with a careful eye. Quantitative data analysis may require only simple summary statistics in a spreadsheet program like Microsoft Excel, or more nuanced econometric analyses using specialized software like Stata or R.
If you do not have internal capacity to reliably collect, analyze, and interpret the data, you should work with an external party.
3. How much funding and staff time can I allocate to this exercise?
If you choose to conduct the process evaluation externally, you’ll save on staff time. Note that someone from your organization who is familiar with the program will still need to spend time working closely with the evaluator and supporting them in data collection.