This post was authored by Melissa Kuenzi, Associate Product Manager at SurveyCTO.
Reflecting on the pandemic’s long-term impact on SurveyCTO data collection methods.
Since the beginning of the COVID-19 pandemic, we’ve been deeply interested in its impact on survey research, especially on how it has changed the way data is collected. For SurveyCTO users, that impact has largely been experienced by moving from in-person data collection to computer-assisted telephone interviews (CATI). Two years after the onset of the pandemic, we set out to understand more deeply what that impact looked like. We created the “Pandemic Pivot” project: A series of qualitative interviews, an administrative data review, and a broad survey of users focused on the impact of the pandemic. We explored the shift to phone surveys during the pandemic, users’ preferred methods of CATI, and their plans for future data collection in a post-pandemic world.
We were honored to share this project at the annual American Association of Public Opinion Research (AAPOR) conference in Chicago this month. Along with sharing our findings, we also spoke with other survey researchers about their own experiences with mode changes during the pandemic. Now, we’re excited to share this project here on our blog with our wider community.
Prior to the COVID-19 pandemic, computer-assisted personal interviewing (CAPI) was the most popular data collection mode for SurveyCTO users, who generally work in areas where face-to-face interviews are the most reliable way to collect high-quality data. Typically, users would take their mobile devices into the field to interview respondents and fill in questionnaires. Other data collection modes, such as computer-assisted telephone interviews (CATI) were employed by a minority of users. As such, CATI surveys were not a major product focus.
But in early 2020, everything changed
As the pandemic unfolded, we realized that users needed to transition from in-person data collection to remote modes, and they had to do so quickly if they were still going to collect quality data. We were determined to meet their needs.
Since most users were already set up for CAPI and accustomed to conducting surveys on mobile devices, we decided to design a feature that would allow them to do remote interviews while maintaining as close of a setup to CAPI as possible. Prior to the pandemic, a CATI setup would have involved some combination of a device used for making calls, and another device or computer for filling in survey forms. While this is certainly feasible for some users, such a setup requires two devices per enumerator and is quite a departure from the mobile device-based CAPI mode that SurveyCTO was primarily designed for.
To make phone surveys feasible for users, we turned the SurveyCTO Collect Android app into a phone app that could make and manage phone calls. This allowed for a single, streamlined interface for holding a phone call while filling in forms. We also created our phone-call field plug-in, allowing users to call respondents directly from a questionnaire inside SurveyCTO Collect and complete a computer-assisted telephone interview all on one mobile device. This data collection setup functioned similarly to how it would have before the pandemic, except the interviews would occur over the phone instead of in-person. To help users get started with this transition quickly, we created the CATI starter kit, which featured sample forms, templates, and documentation on switching to CATI. To further assist with the longer-term impacts of the pandemic, The Safe People + Data Initiative was formed, connecting users with great remote data collection resources from SurveyCTO and organizations like JPAL and IPA, who were also working to assist with this challenge.
The “Pandemic Pivot” research project
As the pandemic progressed, we wanted to know more. How many people were using CATI? Was the single-device method enabled by the field plug-in helping? Were users finding ways to conduct CATI surveys using different methods? To learn more, we decided to collect a wide range of data on the impact of COVID-19 on SurveyCTO users’ work, and the “Pandemic Pivot” project was formed.
We gathered data on CATI deployments on SurveyCTO, surveyed a random sample of 310 paid users through an online survey, and spoke in-depth with a smaller number of users. We wanted to understand users’ experiences with data collection modes over the course of the pandemic, their preferences on different methods of CATI, and what they thought about the future of their data collection work.
Here’s what we found
Our administrative data tracked usage of the phone-call field plug-in designed for single-device CATI. We looked at active SurveyCTO subscriptions — meaning those currently receiving survey submissions — to see how many forms contained a phone-call field plug-in, and then compared those with the total number of active forms and subscriptions on the platform. We first collected this data in September 2021 and ran the same reports in February, March, and April 2022.
The number of active forms and subscriptions containing the single-device CATI feature remained remarkably consistent across this entire span of time, with 4-5% of active forms and 11% of active subscriptions containing the single-device CATI feature in any given 30-day span. When we blended administrative and survey data, we saw that 34% of subscriptions who responded to the survey used single-device CATI.
While survey respondents’ reporting of single-device CATI usage is higher than what we observed monthly on the platform, not all active subscriptions have forms collecting data at all times. Also, many users deploy multiple modes of data collection, and while the single-device CATI feature is the single most popular method of CATI, many do use other setups. Based on this evidence, single-device CATI turned out to be a useful option for users but did not become the dominant method of data collection on the platform.
For our qualitative interviews, we spoke with users from diverse professional backgrounds, representing universities, NGOs, and private companies. They live and work in a wide variety of global locations, including the United States, Nepal, Ethiopia, Ghana, and India. Their data collection projects ranged from collecting data from government employees about their country’s federal Covid-19 response to interviewing female business owners in rural India. We found that all these users had pivoted to CATI because of the pandemic, and they reported varied but overall positive experiences with CATI, despite challenges.
The third tier of the project was the online survey conducted on the SurveyCTO platform. This asked users about their use of CATI/phone surveys, the impact of the Covid-19 pandemic on their data collection decisions, and their plans for the future. Findings are summarized in the images below:
A couple of key findings stood out from the survey that we want to highlight
Usage increase in phone surveys: One key finding involved the overall rate at which users increased their use of phone surveys, and switched to them specifically because of the pandemic. As widely reported and expected, a high number of respondents confirmed that their overall use of CATI increased during the pandemic. While some increase in CATI over the past two years could have been incidental, our data shows that 74% reported choosing to switch to using CATI for data collection specifically because of the pandemic. We knew that CATI was being used much more than before the pandemic, but it was still striking to see these percentages and confirm how widely this pivot had affected users.
Future data collection: The second key survey finding centered around how users said they planned to collect data in the future. The survey revealed that 31.6% plan to use CAPI; not surprising given that this was the mode most were using before COVID-19. But a striking 48.4% of survey respondents didn’t choose CATI or CAPI as their main mode for the future. Instead, they chose a mixed-mode approach.
What we learned
SurveyCTO users turned to CATI in high numbers during the pandemic. Deployment of single-device CATI looks consistent from our data, and users say they prefer it above other possible CATI setups. We also learned that while in-person data collection will continue to be important for many SurveyCTO users, times have changed, with more than half of questioned respondents saying they will not use CAPI as their primary mode in the future. This is a noteworthy change from pre-pandemic times — for us, and for users.
Of all our survey findings, the increase in mixed-mode data collection struck us as perhaps the most important, and this was echoed in our qualitative interviews. One user said that while they looked forward to using in-person interviews again, they now planned to do all of their pre-screening with CATI. Another said that after using CATI in rural India during the pandemic, she now saw its advantages for both enumerators and respondents. She told us that the enumerators she worked with had spent a lot of time traveling to rural areas for in-person interviews, and CATI simplified their job. For the respondents, an interview on their cell phone was often easier than an in-person meeting to work into a busy day of running a small business. While CATI wasn’t possible for all situations, she would now consider it for some future projects.
Despite the convenience and advantages that CATI can provide, we also acknowledge that they are not directly interchangeable with CAPI. There are very real coverage limitations, representation concerns, and logistical challenges that come with phone interviews replacing in-person data collection. While users in the qualitative interviews tended to speak positively of their CATI experiences, more than one said that they were happy to again be able to use in-person interviewing for at least some of their data collection, citing more complete data and better overall data quality as their reasons. With this in mind, we’ve compiled a list of resources at the end of this post to provide thoughtful research and further insight into phone vs. in-person data collection modes.
As we consider the key findings of Pandemic Pivot, it’s clear to us that the COVID-19 crisis altered data collection practices from 2020 – 2022 in very real ways. Perhaps more importantly, it has also changed what those practices will look like in the post-pandemic future. In response to the pandemic’s challenges, we created new features, users learned new skills, and now data collection preferences and best practices have evolved for many. We expect to see a wider variety of data collection modes and a rise in mixed-mode approaches from users going forward. As we learn from the past two years and look forward to the changes that the future holds, we are deeply mindful that there’s still more to learn, and that continued research is needed on the impacts of this pandemic-related pivot in data collection modes.
For further reading on what current research says about phone surveys, remote surveying, and data quality, we recommend exploring Remote Surveying in a Pandemic: Research Synthesis and its source data from Innovations for Poverty Action.
If you are considering using CATI for some or all of your data collection, or are curious about whether our one-device setup would work for your organization, check out our CATI starter kit for a host of helpful resources.
