Let’s start things off with a quiz:
1. What percentage of health facilities in Kenya comply with minimum patient safety standards as per current regulation?
A. 20%
B. 3%
C. 65%
D. 90%
B. We do not have data for the entire country, but based on baseline data collection in all facilities in the Kenyan counties of Kakamega, Kilifi, and Meru (~1,100 health facilities), 3% of facilities comply with minimum patient safety standards.
2. What percentage of procedures are performed with new (as opposed to reused) needles and syringes in Kenya?
A. 57%
B. 21%
C. 100%
D. 78%
C. Based on the same baseline (~19,000 observations of healthcare-worker-patient interactions), 100% of procedures were conducted with new syringes and needles in the three study counties.
This baseline research comes out of the The Kenya Patient Safety Impact Evaluation (KePSIE), a unique partnership between the Kenyan Government and the World Bank Group (WBG). It is the largest trial to date aimed at improving patient safety in low- and middle-income countries.
We wanted to learn more about KePSIE and how SurveyCTO is playing a role in the research, so we reached out to Guadalupe Bedoya, an economist in the Development Impact Evaluation team at the World Bank’s Research Group and the Team Leader and co-principal investigator of the impact evaluation. She provided an in-depth look at how the project has evolved, what her team has accomplished so far, and where the research is headed next.
Tell us a little about the project.
The impact evaluation team members include me, Jorge Coarasa, Jishnu Das, Amy Dolinger, Ana Goicoechea, Njeri Mwaura, Frank Wafula, and Khama Rogo. The team is also supported by Benjamin Daniels (DECHD), Yvonne Machira, and Chex Yu (DECIE).
In early 2010, as a product of a multi-stakeholder dialogue, supported by the World Bank Group through the Health in Africa Initiative, public, private, and civil society stakeholders in the health sector agreed to partner to develop a health inspections regime that would be fair, transparent, and focus on guaranteeing the safety of all Kenyans.
“The scale and scope of this survey make it the world’s biggest patient safety survey conducted so far.”
Since then, all of the professional boards and councils in Kenya with a legal authority to inspect health facilities agreed to conduct joint inspections based on an agreed set of minimum mandatory patient safety standards. The following challenges related to health facility inspection were identified by the KePSIE team and stakeholders when the team joined this effort nearly one year after the first joint inspection checklist was gazetted:
- Discretionary grading in the inspections due to lack of definitions on specific items to be inspected
- Inadequate capacity by part of the boards and councils to inspect and monitor a significant number of facilities
- Lack of incentives to improve patient safety at different levels of compliance with the standards due to unclear sanctions and weak enforcement.
“…we programmed our inspection checklist in SurveyCTO, which enables our inspectors to use SurveyCTO’s interactive interface to inspect health facilities.”
After several years of work, we have accomplished two of KePSIE’s objectives, reflected in:
- An Enhanced Regulatory Framework for Health Inspections for Patient Safety, and first of its kind in Sub-Saharan Africa, including a standardized Checklist, a score system to classify facilities according to their risk level, and a system of warnings and sanctions for non-compliant health facilities. This framework now includes private and public facilities (before this reform only private facilities were inspected) and was gazetted recently in the legal framework under Legal Notice No. 46 on March 21, 2016. To give some context, in the Africa region, out of 45 countries with de jure inspection regimes, only five countries report actually carrying out any type of inspections mostly for private health facilities (IFC, 2011).
- A Toolkit of Instruments to Measure Patient Safety in Multiple Domains validated and tested during 2014-2015.
- Data collection from all health facilities in the three study counties to assess patient safety, quality of care, demand, and prices during 2015. Data collection included around 1,100 health-facility level surveys, 8,500 patient exit surveys, 1,600 health-care worker surveys, and 19,000 observations of healthcare-worker-patient interaction. The scale and scope of this survey makes it the world’s biggest patient safety survey conducted so far. The team is currently finalizing the analysis the results from the baseline data collection.
- An Inspector Training Manual developed for the first full-time cadre of inspectors in Kenya, who are currently being trained and will be deployed in the three study counties for the implementation of the inspection pilots.
What are your next steps?
The team is currently in training and preparing for implementation.
The implementation of pilots to assess the impact of inspection regimes on patient safety is planned to start in September 2016. Specifically, we will experimentally allocate the census of nearly 1,100 private and public health facilities (around 10% nationally) in three Kenyan counties – Kakamega, Kilifi, and Meru – into one of three randomly allocated intervention arms:
- High-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities
- High-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities, and public disclosure of inspection results
- “Business-as-usual” low-probability inspections (the control group)
The three counties where the study will take place were selected by the Health Management Representatives of the 47 counties in the country to represent different conditions and markets in Kenya and the details of the implementation are being developed with the regulatory boards, the Ministry of Health, and the pilot counties.
KePSIE will deploy 12 full-time inspectors to conduct inspections in public and private facilities, applying warnings and sanctions as per the March 2016 Gazette of the Joint Health Inspections Checklist. This team will be also in charge of implementing all the features of the KePSIE pilots, following strict protocols that will allow us to understand the potential of different inspection models and implementation to improve patient safety.
What are some of the innovative ways you are using SurveyCTO?
We used SurveyCTO to collect our baseline data. But it is proving to be an equally useful tool during our implementation phase. For example, we programmed our inspection checklist in SurveyCTO, which enables our inspectors to use SurveyCTO’s interactive interface to inspect health facilities.
But first we had to standardize the health inspection checklist considerably. We programmed the first version using SurveyCTO in 2014 and then we piloted a version that the regulatory boards wanted to test. When they saw the quantitative results for the first time (never before had they seen quantitative results of the health facility performance on patient safety), they decided to implement and gazette the detailed format version we had programmed in SurveyCTO.
“When we showed the electronic version of the checklist to real inspectors for the first time, they were amazed that their job could be so much easier and that they could easily access the data as well.”
Both for the pilot and the baseline data collection, we collected a survey version of this checklist.
Now we are supporting the training of inspectors to do official inspections with SurveyCTO. Because of the slightly different purpose of using SurveyCTO for government officials during health facility inspections (versus collecting household survey data), we tailored SurveyCTO’s different features to fit our project. Here are a few examples:
- We needed considerable flexibility due to the variety of health facilities, so navigating between sections was essential (they couldn’t be constrained to a specific order in which the checklist would need to be filled in). We contacted SurveyCTO and they developed the menu function that allowed us to have hyperlinks and a more organized and user-friendly navigation. This was critical for us to improve the functions we needed and we are very grateful for the prompt response from SurveyCTO. This feature is essential in the new phase we are starting.
- In addition to having a menu page with hyperlinks, we wanted easier navigation so that inspectors could click on one section title and all the sub-sections would expand as a cascade. We were able to use SurveyCTO’s Go-To-Prompt to create a menu with different levels of cascades so that it is much easier to locate specific sub-sections of the checklist. This has been a great feature as well.
- We also wanted to score each question and display every section score at the end so that inspectors can inform health facilities of their scores after each inspection. The facilities are classified into risk categories based on this score and there are warnings and sanctions associated to this score. At the end of the inspection, the SurveyCTO checklist estimates the scores per section, which the inspectors will use to fill in a summary sheet and leave with the facility. At the end, the uploaded data goes to a web-based monitoring system where all completed reports will be available. We also have indicators based on these results for different stakeholders that are available almost in real time.
Is there anything else you’d like to share about the project or survey design in general? Any advice for others?
When we showed the electronic version of the checklist to real inspectors for the first time, they were amazed that their job could be so much easier and that they could easily access the data as well. They are mostly senior government officials that do intensive but sporadic inspections (10-day waves) but have other responsibilities in the Ministry of Health or the medical boards. Now they are training full-time inspectors to do intensive inspections during our project. The combination of the electronic version of the checklist (through SurveyCTO) plus the web-monitoring system are an important part of the accountability systems we are testing.
SurveyCTO is a research tool for us as researchers to adjust the tools as we go. At this stage, we will also be fine-tuning and testing how to implement this at scale (a World Bank project is already thinking of scaling-up part of what we are doing). It would be interesting to assess whether you can design a tool that goes beyond research.
