The COVID-19 crisis has deepened the need for effective governance, whether its source is the state or non-state actors. In territories that are heavily controlled by gangs and organized criminal groups, even without a health crisis, it is often hard for state institutions to provide much needed health care and economic relief services. But could the COVID-19 pandemic help governments reestablish preeminence in these areas? A team of researchers at Innovations for Poverty Action (IPA) in partnership with The University of Chicago and EAFIT University, and with funds from the Crime and Violence Initiative at the Abdul Latif Jameel Poverty Action Lab (J-PAL), conducted phone surveys in Medellín, Colombia under a project titled Mi Barrio to investigate this question and others. Read on to learn about this project and how they used SurveyCTO.
Adapting phone survey design and workflow for targeted sampling
From May to June 2020, the team at IPA Colombia interviewed 1,197 people from middle- and low-income neighborhoods in Medellín via computer-assisted telephone interviewing (CATI). The team obtained respondent phone numbers from a combination of sources, including from a neighborhood-level government organization, programs run by the Medellín city government, and social contacts of IPA research assistants and other outreach workers.
The survey focused on the following factors related to the COVID-19 lockdown: criminal and state governance (who solves different problems of the community), combos (local youth gangs) and state legitimacy (how citizens perceive the actors and their performance), changes in the supply and prices of basic goods, changes in the extortion market, and changes in the drugs market. The IPA team also leveraged a recently completed representative survey and field experiment on the relative strength and legitimacy of state vs. gang governance in Medellín to inform the survey.
To minimize contact, IPA used a local delivery service to deliver mobile devices to enumerators. The devices were installed with the latest version of the mobile app SurveyCTO Collect, which now offers integrated call management capabilities and enhanced security features. The team used the SurveyCTO CATI starter kit, which offers an automated and scalable approach to using SurveyCTO as a CATI solution on both mobile devices and web forms, and incorporated SurveyCTO’s case management capability to design its needed workflow.
The team wanted to focus on specific neighborhoods (barrios) and previously defined experimental sectors in Medellín. Specifically, in each neighborhood and experimental sector, the team planned to survey up to (and not more than) four people. To ensure that this condition was met, the team added two tracking datasets to supplement the case management cases datasets on their server. One dataset tracked the number of surveys already conducted in each targeted neighborhood and the other tracked the number of surveys conducted in each sector. When an enumerator began an interview with a respondent, they noted the neighborhood and sector for which the respondent was participating. The form then automatically calculated the number of surveys that had been completed for that neighborhood or sector. If the number of respondents had already reached four, the enumerator was prevented from continuing with the survey. The flow diagram below demonstrates the workflow used for this survey.
Positioning enumerators for a successful phone survey with remote training
Recruiting and training enumerators to work in remote locations can be challenging for large data collection projects. To overcome these challenges, the IPA team recruited 23 enumerators who had worked on previous IPA projects. The team then used the following strategy for training:
- Before scheduled training sessions, enumerators were provided with training materials, which included a detailed manual for the survey, links to important resources, and pre-recorded training videos via email.
- The training sessions themselves were conducted online using Zoom. To improve enumerator engagement and understanding, the training included questions after every two or three slides. Enumerators were also divided into smaller groups to further promote asking questions and participation.
- To support continued, self-directed learning after the training, the IPA team anticipated parts of the enumerator manual that would be harder to understand and created videos and other resources that enumerators could access at any time in a shared Google Drive during the course of the survey. These resources included a frequently asked questions (FAQ) document based on questions asked during training, video recordings of the training, and other materials used during the training.
Three-pronged data quality monitoring approach: Back-checks, audio audits, and high-frequency checks
The IPA Colombia team put in place three quality control mechanisms: back-checks, call recording, and high-frequency checks. See below to learn more about each:
- Back-checks: Back-checks are a quality control method implemented to verify the quality and legitimacy of data collected in a survey by re-interviewing some respondents with a short subset of survey questions. An experienced enumerator who was not involved in this survey was hired as an auditor to perform back-check interviews on a randomly selected 15% of the respondents that were interviewed. The auditor reasked these respondents some of the questions included in the survey in order to compare responses. The auditor also used these back-check interviews to gather data on the performance of enumerators. This process revealed a few cases where enumerators did not properly read the consent or skipped specific questions.
- Call recording: With the proper consent from respondents, all calls were recorded using SurveyCTO’s audio audits, which allows recording both sides of a phone call launched using SurveyCTO Collect. The IPA team randomly selected and listened to 15% of daily calls to evaluate the quality of responses and to identify focus areas where more training was needed for enumerators.
- High-frequency checks: The team also developed multiple tests that were run daily to evaluate the quality of incoming data, also known as high-frequency checks. Examples of these checks included missing data checks, incomplete variables, skip patterns, survey logic, distribution of continuous variables, and others. The most common issues that were discovered were related to typos in the prices of basic goods and incorrect unit of quantity. The team performed an outlier analysis and used soft and hard limits to identify these issues and then used recordings to make corrections.
Planning for internet connectivity challenges on the ground
Due to the fact that this survey used case management and relied on enumerators working remotely on mobile devices, it was important to have a reliable internet connection at all times. This was particularly challenging because some areas in Colombia have poor access to reliable internet connection. To minimize the impact of internet connectivity limitations, the IPA Colombia team recommends the following strategies:
- Teams should verify the access, bandwidth, and speed of the enumerators’ internet connection during the planning stage of the data collection project. This is important for understanding the best approach to designing the survey workflow and making contingency plans for enumerators.
- Enumerators must be provided with backup internet access, such as mobile device internet, to be used in cases where the main internet connection fails. For this project all enumerators were provided with mobile device internet and some used this as their primary internet option.
- Survey instruments and the survey workflow process must be designed with these internet restrictions in mind. For example, this might include minimizing the amount of the data that must be transferred between enumerator devices and the SurveyCTO server by controlling the file size of image and audio recording attachments.
More information
The IPA Colombia team is currently working on the analysis of the data collected for this survey and hopes to publish findings later this year. Please see this page for updates. For any questions about this project, reach out to IPA’s press office at press@poverty-action.org.
To learn more about how to use SurveyCTO for phone surveys, explore the CATI starter kit and watch this webinar recording for a helpful overview.
If you’re a SurveyCTO user and need help, our support team is available 24×7. Go to the Support Center and click on the link to “Submit a support request.” No question is too small.
