In early 2021, Assess Transform Reach (ATR) Consulting won the bid for a contract with the World Bank to conduct four (4) rounds of telephone surveys, with a sample of 3,500 per round. The intent of the phone surveys was to track the evolving economic implications of the Coronavirus pandemic in Afghanistan. To date, the measured impacts of the pandemic have adversely affected the Afghan people’s welfare through at least three broad channels:
-
- Income and employment
-
- Prices and the availability of essential goods and services
-
- Health consequences, including the ability to access healthcare
The World Bank wanted to understand two things: the economic consequences of COVID-19 for individuals in Afghanistan, and economic trajectories as they sought to recover from the joint health and economic shocks. They were particularly interested in hearing from individuals who were already vulnerable to economic hardship, and they wanted to ensure women and people from regions throughout Afghanistan were represented in the data.
The surveys would be completed over four high-frequency survey rounds covering a nationally representative sample of 3,500 individual respondents in each round. No more than 25 percent of respondents could come from a single province, and at least 30 percent of respondents needed to be women. The World Bank also requested real-time updates on the data being collected.
The challenge: conducting high-frequency phone interviews while ensuring broad coverage and gender representation
ATR was given a baseline sample of 5,000 respondents for the first round of their survey, allowing for high rates of attrition common with phone surveys. Due to the COVID-19 pandemic, the survey was required to be conducted via phone.
Before using SurveyCTO’s case management feature, the ATR team used standard data collection forms for remote data collection. In this method of remote data collection, phone numbers were first collected from the field and then verified, cleaned, and then distributed to the call center agents. They would provide each agent with records containing all of the phone numbers collected from the field or a set of Random Digit Dialing (RDD) numbers.
“We used to make standard forms for remote data collection, providing each call center agent with a pre-established list of numbers when they were working from the HQ,” recalls Ahmad Khalid Haidar, QA Manager at ATR Consulting. “This was extremely inconvenient because any error resulted in multiple interviews and there was no log of attempted calls.” In addition, the lack of real-time tracking meant that there was no way to generate any live data on the progress of the data collection.
ATR needed a new system to manage their data collection process that would minimize error, reduce unnecessary calls, track survey progress, and focus their data collection efforts so they would meet the gender and regional criteria the World Bank required.
“We used to make normal forms for remote data collection, providing each enumerator with papers. This was extremely inconvenient because any error resulted in multiple interviews and there was no log of attempted calls.“
Ahmad Khalid Haidar, QA Manager at ATR Consulting
The solution: using case management and CATI to automate enumerator workflow and track progress in real-time
ATR implemented three strategies to conduct the rounds of surveys and meet The World Bank’s criteria. First, ATR used SurveyCTO’s case management feature to build case lists for each of their call center agents. Then, they utilized computer-assisted telephone interviewing (CATI) to automate each agent’s workflow and reduce the number of errors. Finally, they built an XLS Form to automatically receive data from SurveyCTO in real-time, so they could adjust case lists as needed and share live updates.
Here’s how ATR automated their workflow:
1. Build a case list, and assign each call center agent with cases
Step one was implementing case management. Case management allows survey designers to organize their data collection into cases, and a case can be anything — a household, a participant, etc. — depending on the unit of observation for the project. ATR created a case for each respondent on their list. Then, they used SurveyCTO’s case management workflow to assign a list of cases to each call center agent. Respondents were assigned a unique ID, a label the enumerator would see, and a list of forms to be filled out. These forms asked how many times an enumerator had attempted to reach the participant, the survey questions themselves, and whether the respondent had refused to participate.
This allowed ATR to track the status of each respondent, so the call center agent would call the correct respondent the correct number of times, and then move on if the respondent couldn’t be reached or refused to participate.
“This was our first time using the case management feature,” says Asma Rahimi, Call Center Agent at ATR Consulting. “It was a great experience that enabled us to streamline the process and collect high-quality data.”
2. Use computer-assisted telephone interviewing to automate the research workflow
Computer-assisted telephone interviewing (CATI) is an interview method where call center agents use an electronic device such as a computer or tablet to read the survey script and log the responses to the survey.
For this project, ATR tested five different smartphones and tablets to select a model that could be used for recording both sides of the phone interview (the call center agent and the respondent) and completing the SurveyCTO form.
It took four weeks for ATR researchers to review, translate, and back translate the computer-assisted telephone interviewing (CATI) tools their field researchers would be using, create an XLS Form that would display their data, and complete user acceptance testing.
They also set up the Survey CTO audio audit feature to record specific portions of interviews for quality assurance purposes.
Then, their interviewers set off collecting data with an automated CATI workflow.
Four steps of ATR’s CATI workflow, built with SurveyCTO’s case management feature:
-
- First, the enumerator would open or refresh their case list. When they were ready to make a call, they would open a case and check that the respondent’s information had been verified.
-
- Then, the enumerator would call the respondent. If the respondent was available and consented to the data collection, the enumerator would proceed with the survey. If not, the case would automatically be rescheduled.
-
- If the phone was not answered, the enumerator would classify the call as out of coverage, no answer, dropped call, or incorrect number, and the case status would be updated.
-
- The case would be designated as closed when the survey was conducted successfully, the respondent refused to participate or the number of tries to reach the respondent exceeded 5 attempts.
Once the enumerator had completed their list, they would automatically be assigned new cases.
“We have been conducting phone interviews for quite a long time through SurveyCTO, but this was quite special where calls were logged and we were able to track the number of calls through the case management feature,” says a staff member from the ATR call center.
“SurveyCTO’s CATI toolkit and case management features helped us track the number of attempted calls and call statuses at different intervals,” says Ahmad Khalid Haidar. “They also helped us reschedule calls, record interviews for quality assurance purposes without having to use a third-party app, and assign new cases for enumerators when they had completed their lists.”
3. Send live data to a dashboard in Google Sheets, fed by SurveyCTO
Then ATR monitored the XLS Form they had coded to display the data being collected from SurveyCTO. They used this to track the progress of the project, make adjustments as necessary, and deliver updates to the World Bank in real-time.
ATR used the dashboard to track:
-
- How many survey respondents they had interviewed in each region
-
- Gender representation amongst survey respondents
-
- Quality assurance status of each case
-
- Enumerator performance
-
- Number of overall participants whose cases had been completed
This data ensured ATR knew exactly how many cases they had completed from their list of samples. Once cases for all of the sampled phone numbers were complete, they created a new set of cases using random digit dialing (RDD) to reach their target number of interviews.
“The CATI system allowed us to easily adapt when we switched to RDD,” says Ahmad Khalid Haidar. They assigned a new case list to each enumerator, and the project continued to flow smoothly.
ATR completed the first round of data collection in just over a month and conducted the second round one month later. They completed their project on August 11, just four days prior to the fall of the previous Afghanistan government.
“SurveyCTO’s CATI toolkit and case management features helped us track the number of attempted calls and call statuses at different intervals. It also helped us reschedule calls, record interviews for quality assurance purposes without having to use a third-party app, and assign new cases for enumerators when they had completed their lists.”
Ahmad Khalid Haidar
The result: easily tracking calls and ensuring quality for a smooth workflow
Minimizing mistakes and keeping track of progress with CATI
“Owning SurveyCTO’s CATI system allowed us to distribute a pool of unique numbers to each enumerator, minimizing mistakes,” says Ahmad Khalid Haidar. “We also have a record of all attempts to reach respondents, organized based on their statuses such as wrong-number, refused, out-of-coverage, etc.”
What’s next for ATR
Even as the political situation in Afghanistan rapidly changes, ATR is confident that the systems they have in place will allow them to obtain the highest possible quality of data. They can easily modify surveys or case lists using case management if they need to, and they can update their questions with CATI so the call center staff knows exactly what to ask.
“SurveyCTO’s case management minimizes the risk of multiple data collectors surveying the same case and streamlines the process of data collection. Additionally, this feature seems to be the best option for conducting remote data collection due to the availability of internet at the provincial centers, the enumerators can easily track the status of the phone surveys and it would help the Data Management Team and Quality Assurance Team to process the data by the time the data is available on the server.”
Eager to speed up your workflow, reduce error, and hit data collection targets?
Try SurveyCTO for your next project
