Laterite is a data, research, and technical advisory firm based in Rwanda and Ethiopia – and uses SurveyCTO for everything from short M&E questionnaires to long, complex impact evaluations in education, health, agriculture, governance, and other sectors. Recently, they started using SurveyCTO in innovative ways to streamline and scale up their enumerator hiring and management processes.
We asked Josep Casas, Anirudh Rajashekar, and Ganesh Rao about their systems, how they managed 1700 applications in three days, and what advice they have for other researchers and managers.
How are you are using SurveyCTO for your internal hiring, managing, and testing processes?
Laterite is currently undergoing a transition. Our team and workload has almost doubled since the beginning of the year and many of the tasks of the data collection team such as the management of enumerators or developing field plans are still being done manually. With multiple projects happening at the same time and over 300 enumerators, the time it takes to complete these tasks is quickly becoming unsustainable and we needed a more efficient system.
“Since the test was going to be taken by thousands of applicants simultaneously, we had to implement several features to prevent candidates from cheating.”
We want to be able to efficiently manage our ever-growing team of enumerators and focus on those tasks that truly need our human resources. The system and tools being put in place are tailored to meet the needs of our staff, can be managed with minimal supervision, and are easily scalable. We believe these features will maximize the productivity of our staff, simplify internal processes, and maximize the value delivered by Laterite.
For a typical data collection project, there are four main requirements:
- We need to recruit enumerators
- We need to manage and communicate with them
- We need to build teams of enumerators
- We need to build field plans.
We are developing tools to automatize as many of these tasks as possible. SurveyCTO has been crucial in the development of the tools for the first two parts.
The first tool we built is for recruitment of new staff to our enumerator team. We have used SurveyCTO to automate most of the processes related to this task to make the process more fair and save time. Previously, after advertising the positions, we used to receive thousands of applications by email. Applications came in different formats and were often incomplete. Reviewing the CVs and cover letters took a lot of time and was not very effective at discriminating between the good and bad candidates.
To deal with the huge number of applications and improve our selection process, we created two forms in SurveyCTO, an application form and an assessment form. The application form is the first interaction prospective candidates have with Laterite and uses SurveyCTO’s web form feature. Candidates can access it through the link provided in the job advertisement and fill it in with all the usual information (personal information, education, experience, and references) as well as a short cover letter.
We select those candidates that have correctly filled in the application form and invite them to sit for an assessment test. The assessment (again a web form) is done remotely by each enumerator and consists of a series of questions that aim to test the skills required to be a successful enumerator (e.g. attention to detail, honesty, and capacity to follow instructions). The design of this form, although not very different from a traditional survey, presents some unorthodox yet interesting uses of SurveyCTO.
“With over 300 freelance enumerators, it is very important for us to keep track of all of them.”
Since the test was going to be taken by thousands of applicants simultaneously, we had to implement several features to prevent candidates from cheating. We randomized each of the questions so that each enumerator would be shown a random permutation of the questions. This was important given that the assessment was multiple choice test, which makes it fairly easy to cheat if all the questions are the same. We also included a time limit of 45 minutes from the moment a candidate began the test. The survey would show the time remaining at the top of the screen and this would be updated every time they moved to the next question. Finally, the score of the candidates is calculated as they answer the questions, which makes the final analysis easier. We are very thankful to the SurveyCTO team for the support provided to address the challenges caused by the large number of candidates taking the test at the same time.
The second tool we have designed using SurveyCTO is the Staff Information Management System (SIMS) (Figure 1). With over 300 freelance enumerators, it is very important for us to keep track of all of them. This involves knowing when they are available for projects and keeping their personal information up date. Previously, we wasted considerable staff time manually updating this information but with the new tool we aim to shift the responsibility to the enumerators themselves.
Figure 2: Staff Information Management System (SIMS) schema
The enumerator tool has two primary functions:
- Keep an up-to-date database of all our enumerators and their personal information
- Keep track of when an enumerator is free, busy, or allocated to a project
The design of the system is a complex network of surveys and dynamic datasets sitting on the SurveyCTO server. On the backend we have three dynamic datasets containing the information of all enumerators. The first dataset contains all of their personal details such as ID, phone number, and email. The second dataset contains their availability information. This dataset contains the status of the enumerator for each week of the year (i.e. whether they are busy, available, or engaged in a project). Finally, the third dataset contains the login details of the enumerator (more on this later).
“In addition, we are building tools on SurveyCTO for evaluating the performance of our enumerators by their supervisors and automating daily reports submitted during data collection.”
On the front end, we have the long form – the SIMS – that pulls the data from these datasets. The form is also available online and enumerators can access their information with their username and password. These login details are not SurveyCTO logins, but a username and password that reside inside the third dataset. A SurveyCTO case management system might work for smaller teams but we chose this structure due to efficiency considerations. The survey checks that the password introduced by the user matches the one in our records for the given username. If they are the same, it displays the rest of the form.
In the form, enumerators can see all of their information and update it if the details have changed or are missing. They can also update the weeks they will be available and change the password to access the platform. At the end of the survey, the form automatically detects the information that has been updated and pushes it back to the dynamic datasets in the server.
In addition, we are building tools on SurveyCTO for evaluating the performance of our enumerators by their supervisors and automating daily reports submitted during data collection. Lastly, using STATA and R, we plan to use this information as input for building field teams and efficient field plans for all our projects.
We are currently in the process of validating the systems with a small pilot group in Rwanda. Our plan is to implement it in Rwanda and Ethiopia by early 2017.
This whole exercise has been a great learning experience. It allowed us to experiment and learn how to implement some lesser-known features in SurveyCTO that will be very helpful for designing complex surveys in the future. And, technical sophistication aside, extensive feedback from our colleagues ensured that the needs of our users and local context guide the core of all these systems.
Is there anything else you’d like to share about the challenges of hiring, managing, and training staff? Any advice for others?
Hiring enumerators in Rwanda can be complicated. There are not enough skilled jobs in Rwanda for recent graduates and whenever we roll out a new recruitment drive for enumerators we are overwhelmed with applications. In fact, the number of applications we receive is so large that we have stopped advertising our position in job portals and rely exclusively on word of mouth. Even then, we still receive thousands of applications. For instance, in our most recent recruitment drive, we received over 1700 applications in less than three days.
Dealing with such a large numbers of applicants can be complex. How can we be sure to select the best candidates? With 1700 applications, it becomes impossible to review each candidate. This is why we think that the assessment form has so much potential at efficiently identifying the best candidates. It is important to think carefully about the questions so the test is balanced and the candidates spread over all the grades.
“We also plan on linking the results to the performance score in projects to identify those questions that are the best at predicting successful enumerators.”
In our most recent assessment we obtained the distribution of grades shown in Figure 2. The maximum score was 13 and the minimum score was zero, since we did not penalize for getting an incorrect answer. In the end we selected those candidates with a score greater or equal than 10 that had not copied the essays (the assessment included an essay). We also selected the candidates with the best essays that got a score of 9.
Figure 2. Grade distribution of valid candidates (n=1,172)
There are many ways we want to use this data in the future. We are particularly excited about applying Item Response Theory (IRT), which is the psychometric model behind many standardized tests, to improve the design of our assessments. We also plan on linking the results to the performance score in projects to identify those questions that are the best at predicting successful enumerators.
“We hope some of our systems simplify things but, at the end of the day, you need excellent coordination and communication.”
Managing large teams (especially when the majority of them are freelancers) is a big challenge. We hope some of our systems simplify things but, at the end of the day, you need excellent coordination and communication. Our data managers and senior field supervisors run one of the best data collection operations in the region, implementing complex survey instruments in multiple regions, even in tight timeframes. Through extensive experience, we have also developed standardized training modules that help us conduct effective trainings in just a few days.
