Hearing more: Using audio audits to improve data quality

In this article, you’ll learn best practices on how to use SurveyCTO’s audio audit feature to hear field interviews and increase the quality of your data. You’ll also learn about the important ethical requirements to consider.

What are audio audits?

Audio audits are audio recordings that take place during a survey interview without an indication that the recording has been initiated. They are one of several tools that survey administrators can use to ensure that the highest possible quality of data is being collected. They are also a cost-effective means for administrators to better understand how their surveys are actually being conducted in the field.

Sign Up for Your Free Trial

How can I use audio audits?

Recording every minute of every interview would be impractical in so many ways: devices would run out of storage, it can be expensive and challenging to upload so much audio data to your SurveyCTO server, and most survey management teams don’t have the time to listen to that amount of audio anyway. That’s why SurveyCTO’s implementation of audio audits has selective recording of interviews at the heart of its design.

Audio audits can be randomly activated in a subset of interviews, for only part of the duration of those interviews. You might set the probability that an audio audit takes place at a certain percentage, then over time that proportion of interviews will have audio audit recordings activated in the background. Similarly, you can also record audio from a certain point in the questionnaire (at a set time or question) until another point. It’s also possible to record multiple audio audits in the same form. Read our documentation for more information.

If you are conducting phone surveys, using SurveyCTO’s computer-assisted telephone interviewing (CATI) functionality, recording phone calls (or strategic sections thereof) with audio audits can offer great insight into the quality of data. Depending on context, a different approach to recording calls may be required. You can find more information about the different options in this support guide.

What logistics should I consider?

Audio audits are useful in many contexts—but before you decide to use them in your digital forms, consider their viability:

• What is the mobile internet service like in the areas being surveyed? If the connection is likely to be spotty at best for most of the time, it might be difficult to submit audio audits along with survey data to the server. If you can’t get access to the audio audit files in a meaningful timeframe, they’ll be less valuable. However, SurveyCTO does make it possible for records (including survey data, media files, and audio audits) to be submitted offline to a laptop computer acting as a remote server. Explore our documentation for more information about working offline.
• Does your team have the capacity to listen to audio audits? You’ll need an office-based team member who is conversant in the language/s which is/are being spoken in the interviews. This team member should also be very familiar with the questionnaire, survey protocol (including ethics and data protection regulations, which we address below), and objectives, in order to be able to properly assess what they are listening to, whether they should be listening to the recording in the first place, and generate useful feedback.

We advise that you plan and budget accordingly so that you have the resources you need to make the best use of audio audits. In your planning, we advise that you review the audio recording quality settings on the default audio recorder application on your mobile devices. This is what SurveyCTO Collect uses to record audio. The lowest quality audio recording is usually perfectly intelligible and well-optimized for audio audits.

What probability should I use?

The probability to set for recording an audio audit depends on many factors, including your overall sample size, number of enumerators, and how many key sections of interviews you’d ideally like to listen to. If your sample size isn’t very high and you wish to get recordings from all your enumerators, that’ll mean you want a higher chance. Also, the more enumerators you have, the higher the probability you’ll want to use. For example, if a team of eight enumerators is to complete an equal share of a total of 1,000 interviews, and you used a probability of 25%, you might get around 30 audio audits per enumerator.

Remember that at lower volumes of interviews, you might not yet get the proportion you’re aiming for, due to the nature of random chance. Also, the mere knowledge that interviews are being recorded often does a lot to improve the standard of interviews, irrespective of how many you’re actually recording and listening to (just like in Jeremy Bentham’s Panopticon).

What should I record?

Consider the key sections of the questionnaire for which you have concerns about execution. Here are just a few examples:

• Perhaps your survey has an economics game that you’re deploying in your questionnaire and it’s really important that enumerators explain it precisely and don’t provide any additional information.
• You might also have a story or explanation that the enumerator should read out, before a section of questions about that story or explanation.
• The consent script is also a common part of the survey to record in an audio audit. It is important to ensure that your respondents are giving informed consent correctly.
• Perhaps in an agricultural economics survey, you ask which crops were grown by the respondent’s household last season. This is critical to your study, and under-reporting the number of crops to reduce the duration of the interview is something you’re worried about.

If you don’t have any such sections that stand out but still want to make the best of audio audits, consider parts of the questionnaire that are a bit laborious, or tedious in practice, where an enumerator might be tempted to steer things in a different direction. It can be a less than enjoyable task to ask repeated questions about numerous household members and to ask even more repeated questions about each person’s nutritional intake. However, accurate data of this sort is the substance of many studies about the well-being of farmers and children in their households. You could be keenly interested to confirm that your enumerators are, in fact, patiently asking these questions and accurately recording the answers according to the instructions. This can be particularly useful if you have outsourced your data collection to a survey firm. Check out our blog post for more of our best practices on subcontracting.

How long should I record for?

There is no fixed answer to how long audio audit recordings should be. Make sure the recordings are long enough to be useful, but not so long that they’ll be a problem to recover.

The biggest limiting factor is the likelihood of having problems uploading audio audit data to your server due to poor internet connectivity. If this is a factor, limit the duration of recordings as much as possible. This could be a reason to use time-based as opposed to question-based durations for audio audit recordings. Large recordings can be submitted to a laptop running SurveyCTO’s desktop client, SurveyCTO Desktop, as discussed above.

What are the ethical considerations?

Research ethics are an important consideration when collecting any information about people. To conduct your study, you may have been required to submit to an independent ethical review process in order to determine the scope for harm to human research subjects. Audio audits are certainly something that you should mention in such a process and in your study’s informed consent statement.

Aside from clearance from ethical review boards, there are national and international rules and laws about the treatment of human subjects in research to which your study will be subject. An authoritative list is outside the scope of this article, but most research ethics bodies should be able to offer guidance. We recommend doing research on the requirements for the country and regions in which you’ll be collecting data. Such guidelines go beyond the scope for harm, often looking at precisely how data about respondents will be stored and safeguarded, who will have access to that data, and for what purposes. We also recommend familiarizing yourself with the legalities governing the audio recordings of human subjects in the country where your research is set to take place. For those who are interested, SurveyCTO is compliant with the EU’s General Data Protection Regulation (GDPR). Should you have questions about any other sorts of legal compliance, please write to info@surveycto.com or submit a support request.

The following institutions offer helpful resources to ensure that your survey project meets ethical standards and best practices:

• Johns Hopkins University Human Subjects Research Ethics Field Training Guide
• United States National Institutes of Health Protection of Human Subjects
• University College London Research Ethics
• University of Michigan International Human Subject Research Resources
• World Health Organization Ethical Standards and Procedures for Research with Human Beings

Lastly, audio audits should be part of a broader strategy for monitoring the quality of your survey data. Read this blog post for more insight on the practical ways that SurveyCTO can help enhance your efforts, and check out our documentation on Keeping Your Data Secure for tips related to secure data collection.