Note: This blog post references an early version of SurveyCTO Collect for Android which is now available from the Google Play Store. Please see our release notes to learn more and download the latest version here.
Decision-makers all over the world need rigorous data and evidence to understand how COVID-19 has impacted their countries and make data-driven policy decisions to mitigate those impacts. Innovations for Poverty Action (IPA) has launched Research for Effective COVID-19 Responses, or RECOVR, and developed the RECOVR survey, a rapid response panel survey to directly inform key government partners on the health, economic, and social ramifications of the pandemic.
The RECOVR survey is being conducted via phone surveying, or computer-assisted telephone interviewing (CATI), and has already been launched in nine countries. This blog post presents how IPA has deployed SurveyCTO to conduct the survey in the Philippines, along with best practices and recommendations for remote data collection.
How was the RECOVR survey conducted?
1. Random digit dialing for obtaining contact details
The project consisted of one survey round that takes about 35 minutes to conduct over the phone. The sample size was 1,389 respondents. IPA called around 8,400 phone numbers from a pool of 40,000 numbers generated by random digit dialing, which is a method of generating phone numbers at random, thanks to the services of global sample provider Sample Solutions. Since there were so many phone numbers, enumerators didn’t call back those who didn’t pick up the first time. Out of all the respondents who were reached, 38% completed the survey. As an incentive, IPA offered to top up respondents’ phone credit.
2. Case management workflow for easy tracking of calls
The IPA team in the Philippines used the components of the SurveyCTO CATI starter kit to design and implement its CATI project. Enumerators called respondents using SurveyCTO’s “phone-call-dialer” field plug-in that launches the dialer application on devices and they captured responses in SurveyCTO forms. (SurveyCTO now offers the “phone-call” field plug-in with expanded functionality.)
They also used the case management feature of the CATI starter kit to track the progress of enumerators making contact with respondents and manage their multilingual sample. Case management refers to a data collection workflow centered on cases as opposed to forms. Cases can represent any unit of evaluation in the data collection project (e.g., households, patients, water pumps, etc.). When enumerators started the survey, they found out the respondent’s preferred language and noted it in the form. The form pulled from a separate dataset that captured the languages spoken by each enumerator, and if it calculated that the current enumerator didn’t speak the preferred language, it presented a list of enumerators who did and the updated numbers of cases they had. Enumerators were then instructed to chose another enumerator with a lower number of cases for fair workload distribution. Once they did so and finalized and submitted the form, the case got transferred, and the form sent that enumerator a pre-populated SMS message, configured with the SurveyCTO’s “launch-sms” field plug-in, that informed them of their new assignment.
3. Pre-survey SMS messaging to introduce the survey to respondents
IPA also used Sample Solutions to send bulk SMS messages to potential respondents before enumerators called them to build credibility, introduce IPA and the RECOVR project, and explain how their phone numbers were obtained and why they were being contacted. However, everyone didn’t receive the same message. The IPA team created the following three treatment groups to test if sending messages and their content increased the odds of respondents answering calls and participating in the survey:
-
- Comparison Group: Didn’t receive a text message before they were called by enumerators.
-
- Information Group: Received the introductory text message.
-
- Information & Promised Incentive Group: Received the introductory message with mention of the participation incentive.
How did they recruit and train enumerators?
4. Virtual recruiting to avoid in-person interactions
The Philippines team is comprised of 18 enumerators, nearly half of who are new to IPA and slightly more than a third who are new to phone surveys. Luckily, before the COVID-19 pandemic, IPA already had the practice of remotely recruiting enumerators in the Philippines via online job postings, Facebook groups, and referrals. Therefore, the lockdown didn’t present serious recruitment challenges.
5. Remote training and camaraderie-building
IPA held three days of virtual training using Zoom. They found it useful to break up the team into three groups for more intimate discussions. The training consisted of 1) an overview of the guidelines, survey protocol, and the full questionnaire, 2) mock interviews, and 3) Google Forms-based quizzes on the survey protocol.
The IPA team in the Philippines offers the following training recommendations based on their experience:
-
- Produce training materials that can be reviewed offline. In countries with poor internet service, this is critical. For this project, IPA created a PowerPoint slideshow with audio narration that enumerators could download and access when offline. They also recommend recording a mock interview demonstrating how to handle challenging respondents and complicated scenarios, and sharing the video for offline review.
-
- Set aside time during the training for self-guided learning. They set aside 3 hours for enumerators to study the training materials for more in-depth review.
-
- Create messaging groups for support. Each of the three enumerator groups had their own Facebook Messenger group where they could ask questions and build camaraderie. This allowed for peer problem-solving and alleviated feelings of isolation during remote work.
How did they manage for security and quality?
6. Device management for data protection
Enumerators used IPA tablets that were delivered to them (the majority are based in Manila, which facilitated ease of delivery). Before delivery, the IPA team installed the early release version of SurveyCTO Collect for Android, which allowed them to configure an app passcode that prevents unauthorized access and set up data storage on a private app directory, as opposed to the device itself.
7. Monitoring for quality
IPA used the phone-call-log() function in the form design to capture a full call log of phone events while the form is being filled out, including numbers dialed, call duration, and form activity, for greater insights into survey administration. Using SurveyCTO’s support for publishing to Google Sheets, they also built a dashboard that tracked enumerator progress in real-time. More information on setting up real-time project monitoring can be found in this overview. This example Google Sheets CATI dashboard also illustrates a helpful approach to maintaining oversight of survey progress and adherence to protocol.
IPA used audio audits, which are recordings of some or all survey administration, and device sensor readings that measure light, movement, sound, and conversation for greater insights into how the surveys are actually being conducted. For this survey, IPA only measured sound and conversation (light and movement aren’t relevant for phone surveys). Surveys with low levels of both were flagged to help identify lower-performing enumerators and crosschecked through back-checks.
IPA used text audits, which provide a field-by-field record of the amount of time spent on each field, to check if enumerators weren’t spending enough time on certain sections, especially the consent script. In one instance, they suspended an enumerator because they determined that he spent much less time on the consent script and was likely skipping the content.
Learn more
More information about RECOVR, including the full downloadable questionnaire, can be found here. For any questions about this project, reach out to Yuna Liang at yliang@poverty-action.org or info-philippines@poverty-action.org.
To learn more about how to use SurveyCTO for phone surveys, explore the CATI starter kit and watch this webinar recording for a helpful overview.
If you’re a SurveyCTO user and need help, our support team is available 24×7. Go to the Support Center and click on the link to “Submit a support request.” No question is too small.
