MEL advisor Lisa Zook discusses piloting a new user feedback analysis system with UNICEF and Ghana Health Service
While it is a best practice to include stakeholders in the evaluation process, it's not always easy to do, especially in rural communities. Project teams often face barriers due to geographic remoteness, illiteracy and language abilities, and power dynamics.
The Amplio Talking Book offers an innovative way to collect community voices and perspectives. With the built-in microphone, Talking Book users can record their feedback directly onto the device. Each Talking Book can collect hundreds (or even thousands) of user feedback messages — typically recorded in a community's local language.
That's a lot of qualitative data! As you can imagine, it takes some effort to process.
To address this challenge, Amplio recently released new software to help our partners analyze the Talking Book user feedback they collect from program participants. We partnered with UNICEF and Ghana Health Service to pilot the new system.
In May, our monitoring, evaluation, and learning advisor, Lisa Zook, presented a case study about the pilot at IAIA22 in Vancouver, BC, Canada, a conference organized by the International Association of Impact Assessment.
Watch a video of Lisa's talk or read a summary below.
Note: the video volume is quite low at the start but adjusts after a few seconds.
Watch a video of Lisa Zook's IAIA22 presentation on a user feedback case study.
Partnering on a Pilot Project
In 2021, Amplio partnered with UNICEF Ghana and Ghana Health Service to launch a Talking Book pilot project at 15 Community-Based Health Planning and Services (CHPS) compounds in Jirapa District. The project involved working with community health workers to strengthen access to accurate, consistent, and engaging health education materials, including maternal and child health, WASH, and child protection topics. Other goals were to generate demand for health services and to build capacity for the community health nurses and volunteers who deliver health education.
For the CHPS pilot, community health nurses and volunteers used Talking Books to deliver health messages, facilitate discussion, and record community feedback.
The team designed an impact assessment to look at how the Talking Book helped the community health workers create more impact. Were they seeing improvements in rural health outcomes? Did the technology support or improve their work? Was the device a scalable method for strengthening the health system?
Impact Assessment Data Sources
Lisa said the team identified four data sources for the impact assessment. The first three would be considered traditional impact assessment methods. These included usage statistics, community health worker surveys, and household surveys. The fourth component was Talking Book user feedback and that’s where the team wanted to capture emergent themes and build participation into the assessment.
Talking Book User Feedback
User feedback is generated when program participants record their questions or commentss directly onto a Talking Book. Most feedback is from adults but depending on the program we also get feedback from children and youth. User feedback recordings are stored on the device until someone goes to the field and connects that device to a laptop or smartphone to collect it. The data is automatically uploaded to Amplio’s server the next time their laptop or phone goes online.
Amplio and its partners can access the user feedback recordings to hear directly from program participants in their own words and voices.
At Amplio, we talk about prompted and unprompted user feedback. People can pick up the Talking Book and record feedback on anything they want to say. We can also build some prompts into the content.
"At Amplio, we talk about prompted and unprompted user feedback. People can just pick up the Talking Book and record feedback on anything they want to say. We can also build some prompts into the content," Lisa said.
"Let’s say a project has a message on hand washing behavior. At the end of the message, we could ask 'What are your thoughts on hand washing?' or 'What are your barriers to hand washing?' and that would be prompted user feedback because we would direct the program participant to record their thoughts about a topic."
1,200 User Feedback Messages
For the CHPS pilot, Talking Books collected 1,200 user feedback messages after about six months. People asked a wide range of questions about health services, the cost of health care, corporal punishment, child protection issues, and more.
At that point, the team needed to decide how to process all of the information.
"Keep in mind, the user feedback was in local languages. We had to work with local language speakers to sort and process the recordings, and make sure the data was fed into the impact assessment, and more importantly, into the program design."
Amplio's Solution
To process the user feedback, Amplio's product team tried using existing software systems (Transana and Nvivo) but found that they were too cumbersome for audio messages and too complex for the local language speakers to navigate.
So Amplio developed a web-based user feedback processing system. Analysts can log in to access all of the messages that are collected by a project. The dashboard shows how many messages have been analyzed and how many still need to processing.
The system randomly selects a user feedback message for the analyst to listen to. A short series of questions helps them categorize the message and identify issues and trends. Multiple analysts can log in simultaneously, and the questions can be customized to align with a program's monitoring and evaluation strategy.
Want to give it a try? Log in to Amplio's user feedback processing demo. Enter "amplio.demo@gmail.com" for the email address and "TalkingBook2021" for the password. (Note: The password is case sensitive.)
Lessons Learned
Lisa said the new user feedback processing system helped the team overcome some of the barriers to participation that are common with traditional impact assessments, including issues with gender discrimination, language, and geography.
"We had access to community members and perspectives that we never would have had access to previously. We were able to identify emergent themes that broke us out of the traditional impact assessment framework. Instead of a rigid framework, where we didn't know the right question to ask, all of a sudden we had an open door for people to give their perspective on anything that was on their minds."
We had access to community members and perspectives that we never would have had access to previously. We were able to identify emergent themes that broke us out of the traditional impact assessment framework.
Next Steps
Amplio's user feedback processing tool will continue to evolve as more Talking Book partners and programs use it and provide their feedback. The team is looking at how artificial intelligence and/or voice recognition could be used, especially for widely spoken languages, to help process user feedback more quickly. They are working on how to use visualizations to better present data. They are also exploring more intentional work with the analysts who process the feedback.
Got a question or comment?
If you're interested in learning more about Amplio Talking Books or leveraging user feedback for community participation in impact assessments, contact us here.