If you are launching a research project in a country where you don’t have a physical presence, you may be considering the possibility of collecting data with a sub-contracted survey firm. There are good reasons for doing so! When you work with a local survey firm, you take advantage of their local expertise, pool of experienced enumerators, and other in-country resources that you wouldn’t otherwise have access to.
However, outsourcing data collection work involves a degree of trust. Choosing this path, as opposed to collecting the data with your own team, means that the service provider’s field staff will be out and about, collecting data and interviewing respondents, without your direct oversight. This doesn’t need to be a barrier to collecting high-quality, trustworthy data, but there are some things you should take in to consideration before you outsource data collection.
Problems you may encounter with sub-contracted survey firms
- To conduct the survey faster, they may include respondents that do not meet your sampling criteria (especially if respondents who meet your criteria are hard to find).
- To conserve resources, they may not monitor their enumerators as carefully as you would, and enumerators, in turn, may cut corners if they are aware they likely will not get caught.
- To conserve resources, they may not pay enumerators sufficient wages, further incentivizing enumerators to reduce their workload.
- If enumerator turnover is high, in order to stay on schedule and within budget, the firm may not expend resources to properly train replacements.
- And in order to secure your business, they may be unlikely to report any and all of the issues above.
This is not to say that survey firms are untrustworthy or that managing your own fieldwork is categorically better. Many survey firms take their work seriously and will raise important matters like the above, even if it means going over budget or time. The point is that your incentives and survey firms’ incentives are not always the same, which poses risks. If you choose to sub-contract field research, you should be aware of those risks and plan accordingly, even if you’re working with a longtime, trusted service provider.
How to avoid problems and collect better data
Get invested in the process
The first step is to choose to be invested and involved in the process rather than merely treating the data collection process as a black box. Sure—in a perfect world—it would be great to simply pay a fee and receive a clean, guaranteed problem-free dataset to start analyzing. But the world is not perfect, and the data your sub-contracted field team collects likely won’t be either. So it’s far better to know what you’re dealing with, than to simply take the quality of the data on faith.
Vet your firms properly
Before hiring a firm, determine whether the firm is the type that prioritizes data quality or a firm that is more focused on conserving resources and time. How they answer questions like the ones below can help shed light on this:
- “What steps will be taken to ensure that your team is adhering to the study design?”
- “How will you check and audit the quality of your surveyors’ work?”
- “What metrics will you look at to measure surveyor performance?”
- “Will you do spot checks and back checks?”
- “What training will you provide to your surveyors?”
- “If problems occur during data collection, what will be your process for reporting those problems to the client? In past studies, what are some problems that have occurred and how did you report and resolve them?”
- “If you find that respondents are struggling to understand any of the questions in the survey, what will you do?”
- “What process do you have in place for addressing a data breach or lost mobile device?”
And always get references!
Leverage your agreement to enforce accountability
Make sure your agreement with the survey firm gives you the most unrestrained, live access to raw data, including all indicators of data quality, as it is being collected. Directly reviewing and analyzing this data allows you to keep the data collection process honest through transparency and minimizes your blindness to events in the field.
One of the easiest ways to achieve this is to use digital data collection platforms like SurveyCTO, be the primary owner of the account, and invite the survey firm to use your account for your project. This weeds out the middleman between data collected in the field and you. It also ensures that you have full ownership of the data you are paying for.
Some firms may be resistant to this level of access because it will leave them feeling exposed to scrutiny. However, as long as outsourced service providers understand that the purpose is not to embarrass them or catch them red-handed, they should hopefully appreciate your interest in helping to promote data quality. And firms that prioritize data quality themselves should have nothing to hide, and hence no objections to such requirements.
If you do face resistance, don’t forget: you are the customer, and your business is valuable to them.
Take ownership of and control the form design
One of the greatest benefits of collecting data digitally is you can program skip logic and constraints into the tool, to control a surveyor’s path through the survey and prevent surveyors from entering nonsensical or illogical answers. This ensures that, from the beginning, the data being collected is as clean as possible.
To ensure that your forms are using all the skip logic (aka “branching”) and constraints they should be, make sure you have full control of the form design process. If you’re following our advice above to cut out the middleman, and are making sure you choose the digital data collection tool that will be used, you will be in a better position to do so. Ideally, design the form yourself in your digital data collection tool of choice and hand it over to the survey firm to use. If your in-house bandwidth is limited and you will rely on the survey firm to design the form, make sure to carefully review and test their work (again, you should have your own account on your data collection platform so you can directly see and test the form the firm is designing).
Capture metadata that allows you to audit the fieldwork remotely
Computer tablets and smartphones have a number of built-in features that can tell you more about the circumstances under which your data was collected. For example, SurveyCTO leverages the clock feature in devices to record how much time enumerators spend answering particular questions. This type of data can be stored for review, but it can also be made actionable. For instance, you might enforce a minimum amount of time to be spent on a question.
In SurveyCTO, you can even capture unique types of metadata, like audio audits (an invisible audio recording), so that you can listen in to exactly what was happening during a survey, or sensor data (to indicate whether the survey was conducted indoors or outdoors, whether the device was still or moving, etc.). You can also combine these features: for example, to trigger an audio audit if questions are answered too quickly.
And, of course, you can also capture metadata like time stamps/times of day the surveys were done, GPS locations of the survey, and device information.
Metadata such as this gives you rich details and insight into what is happening during a survey, even if you are far away from the survey team or survey firm, allows you to carefully audit the quality of their work, and catch issues as soon as they arise. In SurveyCTO, tools like the Data Explorer make it easy to monitor individual submissions, and any associated metadata, as they are submitted.
Digital data collection ensures that you receive data, even remotely, as data collection is ongoing. Monitor data in real-time to catch and correct issues as they arise.
In addition to reviewing the metadata being collected for auditing purposes, the right data collection platform will provide you with time-saving features that make it easy to review aggregated data and check for statistical patterns that are unusual. Use them!
For example, SurveyCTO’s automated quality checks allow anyone to quickly set up statistical checks that flag outliers, variables with means that are too high/too low, or unusual variation in responses by surveyor, among other checks. As data comes in, the server will run these checks for you and flag problematic submissions for easy review in the Data Explorer.
You should also create additional charts and dashboards that automatically update as new data comes in, using a tool like SurveyCTO’s Data Explorer to quickly review and track key indicators. Some indicators you should visualize and track include:
- The average interview duration by enumerator/surveyor, as that is a good general yardstick for quality data. Very short interviews are particularly suspect. Enumerators with short average durations may not be asking all the questions, or asking them correctly. Overly long interviews might also be an indication that the enumerator is struggling to administer the survey.
- Timestamps, to check if surveys are being conducted at reasonable times of day.
- Aggregates of demographics like age and gender, to determine if your sample has the intended distribution of characteristics and if the survey firm is sampling respondents correctly.
- GPS positions of each interview on a map, to monitor whether interviews are taking place where they should be. Location data can also be helpful in auditing a firm’s sampling work: a credible geographic spread of GPS points as opposed to over-concentration can help build confidence in the data and that your sampling procedure was followed.
- Photos (if you are capturing them in your survey) – of respondents, places, landmarks or other things – to confirm where the enumerator was and what they did.
- Histograms, bar charts and scatter plots to check the distribution of answers to specific questions in your form. The exact distributions you monitor will depend on your specific research and type of data you are collecting but, an example: let’s say you ask about household income from different sources and certain categories you expect to see in the data are never selected. Were you mistaken? Is there a problem in the data? Or if you ask a question with a Likert scale as answer choices, are all answers clustered on either end of the scale or do they have a more credible distribution?
- Cross tabulations of averages and distributions of answers against enumerator IDs, to check for enumerator bias, or “surveyor effects”. For example, if the average household size in your study population is 4 and the average for some enumerators is 2 after having collected more than 50 observations each, it might be that those enumerators are under-reporting household size.
Finally, conduct back checks if possible
Even with the most powerful remote monitoring tools at your disposal, there is no perfect substitute for in-person audits. Even when sub-contracting work to a survey firm, try to budget time and resources for contracting a third party that also has a physical presence in your project location to perform in-person back checks (re-visiting respondents independently to verify if they were surveyed). While the main survey firm you are working with might offer to perform these sorts of checks, we would advise that you consider employing a third, independent party for this work.
