The Population Council conducts research to address critical health and development issues around the world. We recently connected with Andrea J. Melnikas about the Council’s work in empowering and improving the lives of adolescent girls through her research on child marriage. She is using tools like WhatsApp and Google Sheets in tandem with SurveyCTO for real-time monitoring and collaboration in India, Malawi, Mali, and Niger.
What are the challenges related to collecting data about girls and adolescents?
There are a number of challenges in collecting data from girls and adolescents. Among the most important is the protection of human subjects and the need to uphold the highest ethical standards when working with children and adolescents. At the Council, we strive to be sure that:
- The data we are collecting from adolescents are necessary (that we aren’t placing an undue burden on adolescents by asking them to participate in the research)
- These data will be used to answer research questions and develop and improve programs that may ultimate benefit these adolescents or others like them
- The data are secure and confidential so that participants can speak honestly about their circumstances without concern that the data may be shared with others through a breach of confidentiality
Using SurveyCTO allows us to feel confident in the security of the data, gives us access to the data more quickly, and gives us confidence in the quality of the data.
“At the Council we want programs and policies to be evidence-based and we want that evidence to come from high-quality data from adolescents; SurveyCTO is a tool that is helping us get there.”
There are other more methodological challenges in collecting these data. As my colleagues have written (Mensch et al, 2014), self-reported data from adolescents on the timing and sequencing of transitions to adulthood (sexual initiation, school dropout, and marriage) may be problematic and inconsistently reported. In other studies, we have found that we have significant age heaping where individuals report ages that coincide with certain milestones in that cultural context.
It is difficult to address biases in self-reported data; some bias is inherent in conducting survey research. We try to address some of these challenges by asking questions multiple ways and triangulating data when possible (for example, comparing household listing data on the age of an adolescent to her own self-reported data) but these methodological issues are not unique to the Council and we can learn from each other and from others in the field about how they are addresses these issues.
One way we are using SurveyCTO to address these challenges is in age heaping. We are asking age questions multiple ways within SurveyCTO with certain constraints built in (for example, we ask age in one question, then ask birth date and month and have a calculated field to compare these responses) and we are running analyses frequently as data come in and relaying information to our field teams when we see patterns that suggest that age reporting may be an issue. This doesn’t solve issues where respondents do not know their actual age, but it can get us higher quality data than we could previously have with pen and paper surveys and data entry.
Once data is collected and analyzed, what happens next?
This depends on the data being collected. In our current project we are running analyses frequently as baseline data come in to address any issues with data collection and to give us a sense of the basic characteristics of the communities where the program will be conducted, as well as comparison communities. After all data are in and analyzed, we develop reports and schedule webinars to discuss the findings and work with implementers to discuss how the baseline findings may influence program delivery as they move forward with implementation.
For our M&E data, there will be a much quicker feedback loop. We have built the M&E forms within SurveyCTO and used the functionality within SurveyCTO to publish to Google Sheets. We then built dashboards in Google Fusion Tables so that the data are dynamically updated whenever a form is submitted. These dashboards are available to staff in various HQs as well as field staff and management staff on the ground in countries where we are working.
“We are using very cost-effective Android phones so that as many field staff as possible have access to the SurveyCTO forms and can be filling in M&E forms frequently, rather than waiting until monthly or quarterly reports are due.”
We are still in the early stages of the M&E for this project but we hope that the dashboards will allow program staff with decision-making power to quickly see how the program is being implemented and troubleshoot any issues. For example, in our dashboard, a supervisor could see that one village was not receiving much attention compared to another (including GPS coordinates to see when program staff visited that village, as well as reported activities occurring within that village) and could contact the program staff to alert them and direct resources to that area, or troubleshoot why program staff were not making site visits as scheduled.
We are using very cost-effective Android phones so that as many field staff as possible have access to the SurveyCTO forms and can be filling in M&E forms frequently, rather than waiting until monthly or quarterly reports are due. We are optimistic that the frequency of the data collection and almost instant aggregation of the data into the dashboard may allow our program staff to follow implementation closely and flag issues early.
Can you tell us about The Girl Roster? How does this fit into the Council’s broader mission?
The Girl Roster is a program tool designed to help increase girls’ access to vital resources, facilities, and services. It is not designed to be a research tool, but instead is useful for program managers who want to assess and understand their program’s coverage. I would encourage those interested in learning more to download our toolkit “Building Girls’ Protective Assets.”
The Girl Roster fits in our broader mission of being an evidence-based organization. We believe that quality information about the universe of girls and young women in a community can help program staff to effectively reach the most vulnerable, and the Girl Roster is a tool that helps people get that kind of information.
What projects are you working on now?
The Council is the research and learning partner for the More than Brides Alliance, a program implementing interventions to delay marriage in India, Malawi, Mali, and Niger. Sajeda Amin is the Principal Investigator overseeing the work and I am the Project Manager; in my role I liaise with our data collection partners, design and program our data collection tools, and analyze the data.
“By the end of 2016 we will have collected data from approximately 5000 adolescent girls 12-19 and listed more than 15,000 households using SurveyCTO.”
We are conducting an impact evaluation of these programs including baseline, midline, and endline surveys, qualitative data collection to understand the situation of early marriage in these contexts and evaluate program delivery, and providing technical assistance on the design and use of M&E tools.
By the end of 2016 we will have collected data from approximately 5000 adolescent girls 12-19 and listed more than 15,000 households using SurveyCTO. In terms of next steps, we are piloting our M&E forms and dashboards now and then will be launching revised forms and dashboards in time for full program implementation in early 2017.
What are some of the interesting ways you’re using SurveyCTO?
Linking SurveyCTO M&E data to Google has allowed us to create program dashboards that are dynamic, accessible, and very cost-effective compared to previous projects where we hired developers to custom build similar dashboards. We have been able to do this all with a very small project team and SurveyCTO made it easy to configure.
“In addition to SurveyCTO we are using other technology platforms like WhatsApp to communicate with field staff in real time and make changes quickly and efficiently so projects stay on schedule.”
Using SurveyCTO and having instant access to our research data was a big change from other projects where paper forms were filled out and then entered manually, sometimes taking weeks or months to have a final data set that then needed to be cleaned significantly prior to analysis. By spending time upfront to build our questionnaires in SurveyCTO, we have been able to improve the quality of the data we get immediately by restricting fields and adding constraints. We need to spend less time cleaning data prior to analysis.
Additionally, having access to real-time data allows us to run analyses daily as data are uploaded from the field so we can start to get a sense of interesting patterns in the data and also identify any issues right away, rather than finding out after all data have been collected.
Is there anything else you’d like to share about your project or survey design in general? Any advice for others?
In addition to SurveyCTO we are using other technology platforms like WhatsApp to communicate with field staff in real time and make changes quickly and efficiently so projects stay on schedule. If we weren’t using these we would have to wait until scheduled Skype calls or emails from project teams to hear about potential issues and then address them. Because we can communicate with entire teams on WhatsApp around the global quickly we can make changes quickly too so that enumerators can test changes immediately and we can be sure we are all satisfied with the instrument before data collection officially begins.
“One advantage to using SurveyCTO in an organization like the Council is the ability to share across projects.”
One advantage to using SurveyCTO in an organization like the Council is the ability to share across projects. We conduct research on adolescents in dozens of countries and have been developing and improving upon questionnaires to address adolescents’ transitions to adulthood for decades, so we are drawing upon a large body of field-tested instruments.
In the past when researchers shared versions of instruments we would cut and paste sections and modules but still primarily field surveys in paper format. Now we can build an electronic database of forms specific to SurveyCTO, including data quality measures we have already put into place within the form. New projects using SurveyCTO can start one step ahead and improve upon older versions with the hope that we ultimately end up with the highest quality data we can achieve in survey-based research.
At the Council we want programs and policies to be evidence-based and we want that evidence to come from high-quality data from adolescents; SurveyCTO is a tool that is helping us get there.
Photo 1 courtesy of the Population Council. Photo 2 by Paul Hewett, courtesy of the Population Council.
