This paper, published in Global Health Action, presents the experience gained from the implementation of the ORCA project. The authors review how the project served to enhance data quality, analysis, interpretation and use. 

The authors used mixed methods to document the fidelity, feasibility, reach, and acceptability of ORCA and identify early signs of improved knowledge and changing institutional practices. 

According to this study, the participants appreciated ORCA’s applied and practical approach as well as good communication from administrators and clear links to national strategy. They, however, expressed frustration with delays, difficulties prioritising project work over routine responsibilities, and lack of formal accreditation.

The authors conclude that knowledge and analytic skills increased and participants were able to integrate experiences from the project into their future work. Health system managers saw potential in longer-term improvements in data analysis and application to policy, although no clear changes were observed yet.

This paper, authored by ORCA analysts and team members, analysed the quality of routine Health Management Information System data using a WHO data quality review toolkit and comparing data to other sources, such as the Demographic and Health Survey. The assessment included a review of indicator definitions, completeness, internal consistency over time and between related indicators, and external consistency compared with other data sources. 

Results from this study showed a varied picture with good internal consistency for some data, such as on individual vaccinations, but very low external consistency for data on fully vaccinated children. Maternal health indicators showed increasing coverage over time, whereas indicators on child nutrition, malaria, and tuberculosis were less consistent. The data quality on neonatal mortality was particularly low. 

In conclusion, the ORCA authors point to the importance of increased attention on the quality of routine data gathered in the health system. The study authors recommend regular triangulation with other data sources, as well as a review of denominators, a reduction in the complexity of indicators and the alignment of indicators to international definitions. 

This paper, published in PLOS ONE provides an important overview of published results from interventions that aimed to improve data quality and use in routine health information systems in low- and middle-income countries. 

The ORCA team members and authors of this scoping review, conclude that the studies reviewed in this paper showed that a combination of interventions, addressing
both behavioural and technical factors, improved data quality and use. Interventions
addressing organisational factors were non-existent, but these factors were reported to
pose challenges to the implementation and performance of reported interventions.
 

After two years of successful on the job training for 36 analysts from the Ministry of Health, Ethiopia, the Ethiopian Public Health Institute and the Ethiopian Pharmaceuticals Supply Agency a virtual certification ceremony was held on July 30th 2020 to celebrate the achievements to date. It was an exciting moment for all, as Dr Ali Karim from the Bill & Melinda Gates Foundation noted during the online ceremony. All participants have now completed a series of training courses, have received mentoring and coaching and have conducted research in six thematic groups providing them with a range of analytical skills to collect, analyse and make use of data in their every-day jobs.

The ORCA project set out two years ago to work with analysts to foster their critical thinking, especially in terms of collecting, analysing, interpreting and using data and implications for health services and health policy formulation. A further goal was to establish a group of internal experts to drive improvements in the generation, management and use of data within the Ministry of Health and related institutions. Dr. Ebba Abate, Director General of  Ethiopian Public Health Institute, highlighted the special nature of the ORCA project describing it as something innovative and underscoring its importance as a part of the Ethiopian information revolution.

During the online ceremony, which around 58 people attended, government officials, representatives from the Bill & Melinda Gates Foundation and the ORCA team came together to celebrate these aims being achieved and acknowledge and celebrate the hard work done by all participants. Dr Meseret Zelealem, Director Maternal and Child Health, Ministry of Health, Ethiopia spoke about significant improvements made through ORCA, which are already observable. She said that this cohort of trainees is just the start and she would like to see a continuation and build on the investments made to date in the ORCA project. 

Professor Lars Ake Persson, the Principal Investigator on the project from LSHTM, congratulated all participants on being selected for the special project in the first place and described the journey everyone had made together to reach this point. Dr Getachew Tollera, the Deputy Director of the Ethiopian Public Health Institute joined in on the praise and added that the fact that all critical institutions were involved in ORCA as one of the reasons for its success and calling on the analysts to apply their skills to improve the federal government health metrics. 

The ORCA team has produced a first research brief on improving the quality of routine health information system data for better decision-making. Key points outlined in the brief include:

·       Data on maternal health including antenatal care, skilled birth attendance and postnatal care were consistent, both over time and when compared with external sources.

·       The coverage of several indicators was higher in the routine HMIS than in the Ethiopian Demographic and Health Surveys. This may be due to issues with the size of the targeted population or over-reporting in the HMIS.

·       Data on individual vaccinations were consistent over time. However, data on fully vaccinated children showed lower coverage in the Ethiopian Demographic and Health Surveys than in the routine HMIS.

·       Data on deaths in the first week of life were of poor quality, with only one in eight such deaths reported in the routine HMIS.

·       The uncertainty of population estimates used for calculating the coverage of health services contribute to quality problems of the HMIS data

The research brief and findings have been presented and discussed with the technical advisory group, as well as partner institutions.

Download the ORCA research brief for June 2020 (pdf). 

After two years of work on the ORCA project we hear from the team and participants about their experience, as well as their findings. Participants, who have worked on six thematic teams reflect on their original motivation to apply for the project and how the workshops, training, mentorship and research has provided them with a wealth of new knowledge and skills. Many of the participants remarked on the importance of working in teams, learning from other participants, as well as the mentors. A new appreciation for quality data and the importance of data for evidence-based decision-making was echoed by all with the hope that their strengthened skills and knowledge would support Ethiopia’s aim to improve the health management information system and achieve objectives set out in the digital transformation plan.

The ORCA participants joined a writing school in January and February 2020, where participants and coaches learnt about scientific writing and discussed findings. The participants analysed results from their desk reviews of existing data, previously gathered from all levels of the health care system in the routine Health Management Information System.  

The ORCA participants identified cross-cutting issues with regard to data quality by triangulating data with information from other sources, primarily the Ethiopian Demographic and Health Survey, and using a WHO toolkit.

Due to the COVID-19 lockdown, the next part of the analysis of data on the findings from the field studies will be performed without the physical meetings and workshops that were previously used. 

A virtual workshop was arranged on May 27^th, 2020, where the ORCA groups presented their data from the field studies. In total, the groups between August and December 2019 visited 58 health care facilities in eight districts in four different regions of Ethiopia to explore reasons for data discrepancy.  Both quantitative and qualitative data was collected. The ORCA participants meet their coaches biweekly for on-line support while they work on their data. 

After many weeks on the road all ORCA analysts completed their field work and are set to start planning their analysis. The 6^th ORCA Workshop held in Debre Zeyt, Andulala Resort from the 18th to the 20th of December provided just that opportunity.

The aim of the workshop was to prepare for the analysis of the data the six thematic ORCA groups had collected at different levels of the health system and throughout Ethiopia. During the workshop the ORCA participants learned about and discussed ways to integrate the quantitative and qualitative data to ensure results complement each other. Moreover, the workshop helped explain issues of data quality in the Health Management Information System.

Carina Källestål jointed the workshop and used templates from the groups´ data to advise on analysis of quantitative data and used the statistics programme R in her teaching.

Joanna Busza introduced ways to approach and code the rich set of qualitative data gathered by the groups.

All six thematic groups started working with their data and are now in the process of preparing a Statistical Analysis Plan to put together their final report.

The six thematic ORCA teams have all completed their field work in different regions of Ethiopia. The analysts collected and analysed performance data at different levels of the health sector. Importantly, the ORCA participants also tried to explore reasons for a lack of data and problems in the quality of routine data by performing qualitative interviews. 

Below: An ORCA analyst interviews a Health Extension Worker at her health post, where she provides women with antenatal care and family planning services. 

Maternal and child health coverage indicators in Ethiopia continue to improve, whilst newborn care and survival stagnate and issues of quality of care, equity and use of data for decision-making, require further attention.

These are just some of the results discussed during a joint MNCH learning conference held in Addis Ababa, Ethiopia, on the 7^th of November. Approximately 100 researchers, including the ORCA project team, government officials, development partners and implementers met at the conference, co-organised by the Federal Ministry of Health, the Ethiopian Public Health Institute and the London School of Hygiene & Tropical Medicine.

Participants at the MNCH Learning Conference in Addis Ababa, Ethiopia.

Ethiopian Public Health Institute Director General, Dr Ebba Abate, welcomed the strong existing research collaborations between Ethiopia and the London School of Hygiene & Tropical Medicine, and highlighted the importance of reviewing data and knowledge during the conference to support health policies and practice in the country. “The conference acts as a platform to bring together research and practice, and make research work for policy and practice.”

LSHTM Deputy Director, Professor Anne Mills remarked on the wide scope of LSHTM-led research undertaken in the country, which aims to address not just the major causes of ill health and disability, but also how these can be prevented and treated through improved services and systems.

Preliminary results from a ‘Mini’ Demographic and Health Survey presented showed and impressive threefold increase in the number of women receiving antenatal care from a skilled provider since 2005 with similar results in the number of women delivering at a health facility. The data did, however reveal significant geographical and wealth disparities, as well as the stagnating levels of neonatal mortality.

Delegates at the MNCH learning conference during a presentation.

It proved to be especially interesting to hear findings from the IDEAS-led evaluation of the Community Based Newborn Care programme, as well as the Dagu-led evaluation of the Optimizing the Health Extension Worker programme, in addition to the Mini DHS results to provide an even more comprehensive picture of the current status of maternal, newborn and child health in Ethiopia, but also highlighting some issues which require further investigation.

Sessions at the conference were dedicated to four broad themes including: Towards Universal Health Coverage, Quality of Care, Data for Health Planning and Mutual Learning for Change. Presenters included researchers, policy makers, planners, analysts and research students from the Ethiopian health system, the Ethiopian Public Health Institute, Universities, the London School of Hygiene & Tropical Medicine and other partners. The sessions offered an opportunity to compare findings from a number of quantitative and qualitative studies focusing on Ethiopia, as well as studies from another IDEAS priority country – Nigeria.

Two LSHTM-led projects in Ethiopia with a particular focus on capacity building within universities and government institutions received special attention during the conference. The Dagu project, which combines the evaluation of a health extension programme with the support to Ethiopian PhD students was seen as exemplary, as funding for PhD research can be rare. The ORCA project is especially highly regarded due to its importance for sustainability. It works directly with Ethiopian analysts, employed in government institutions, to strengthen their analytical capacities and improve data analysis and use for health decision making and planning.

Following the conference partners and LSHTM alumni took the opportunity to further network at an alumni and partner reception.

The fifth ORCA workshop took place from the 18^th to the 20^th September 2019. The aim of the workshop was for the thematic working groups to refine their national and regional secondary data analysis findings and discuss how to move forward with reporting as well as sharing their experiences from the pilot testing with each other. The workshop was held at Adulala resort, in Debreziet (Bishoftu).

One the first day, each thematic group (Malaria, Maternal Health, Neonatal Survival, Immunization, Tuberculosis and Child Nutrition) presented their analysis results, and received feedback from the ORCA coaches and participants. On the second day, groups focused on the analysis of outliers and consistency over time, consistency between related indicators, and external consistency. After a brief presentation and guidance from IDEAS team member Antoinette Bhattacharya,  six -groups, two on each analytic approach, synthesized their results across 18 health system indicators.

On the last day of the workshop, six people were selected from the three analytic analysis groups. This editorial group was responsible for compiling all presentations and prepared one final presentation. In the meantime, the rest of the ORCA participants and coaches discussed how the results of this national and regional data analysis can be further analysed, packaged and disseminated.

Finally, the group presented the results to representatives from the Federal Ministry of Health and EPHI.

With a special focus on comparing secondary data and the importance of quality routine data, a workshop took place at the Ethiopian Public Health Institute with all ORCA participants attending. To start the day participants presented their preliminary work on comparisons of secondary data on indicators of interest from different sources, including the Health Management Information System reports, the Ethiopian Demographic and Health Surveillance reports and various surveys.

A presentation by Antoinette Bhattacharya on the importance of quality of routine data opened up a new perspective for many of the participants. Ms Bhattacharya is a PhD student and member of the IDEAS project at the London School of Hygiene & Tropical Medicine who came to Ethiopia specifically to speak about her research on data quality of routine facility data for monitoring priority maternal and newborn indicators in Nigeria.

The second day Joanna Busza presented on qualitative approaches to health systems research. She described a case study and then took time for questions and answers from participants on their qualitative studies. A much-appreciated feedback session on each group's interview guides followed. After lunch the groups worked on criteria for rigor in qualitative research and mixed methods presentation of findings as well as on planning mixed methods analysis and reporting.

On the third day, participants further refined their work plans for the forthcoming months fulfilling their objectives on reporting assessments and analysis of the discrepancies between different data sources. They then presented and discussed these during the afternoon.

Overall, the workshop was well attended, and the evaluation statements from the participants showed that they appreciated the input given and that they were making good progress with their work.

Two highly competitive Positive Action for Children (PACF) Scholarships are available to applicants from sub-Saharan Africa intending to study at London School of Hygiene & Tropical Medicine on a one-year, full-time, London-based 2019-20 qualifying MSc programme. Deadline for application February 27, 2019.

Find out more 

‘Simple’ and ‘practical’ – words not immediately associated with statistical software.  The ORCA analysts who recently attended a four-day workshop at EPHI, facilitated by Katarina Selling of Statistikakademin, however said just that following an intense two day get-together. The workshop was designed to introduce the analysts to the open-source statistical software ‘R’, using ‘R-commander’ (Rcmdr) – an interface similar to other menu-based statistical software with the added benefit that it enables users to access ‘R’ without having to learn and write code.

During the workshop, the analysts covered the following themes:

• Data management. Importing data into R-commander, sub-setting data, and the functions recode and compute.
• Visualisation. To produce histograms, bar graphs, and scatterplots.
• Descriptive statistics. To compute means, medians, standard deviations, proportions and the like in R-commander.
• Basic statistical tests and their assumptions. To master certain statistical test (for example chi-square test, different t-tests, correlation test) and how to do it in R-commander.
• Introduction to statistical models including their assumptions. To conduct and interpret output from simple and multiple linear regression, and One way- and Multi way Analysis of Variance (ANOVA) in R-commander.

The workshop was intensive with a packed agenda but the analysts remained engaged throughout the process and provided lots of positive feedback. Guililat, working as part of the immunization group, noted that the workshop ‘will help [them] to conduct the analysis part of the research’. Tsion, a member of the maternal health working group, was one of the participants remarking on the ease of using the Rcmdr software.

The skills gained through the workshop will prove to be beneficial when the participants analyse the data for their projects in the coming months.

The group working on TB

The immunisation group

The group working on malaria

The second batch of ORCA participants

The ORCA project participants are grouped in two batches, completing capacity building workshops to support them in their analytical roles at the Federal Ministry of Health and the Ethiopian Public Health Institute.

The second group of ORCA participants, who had come together for a first workshop in August met again from the 7^th to the 11^th of November for a methods workshop. With 15 participants attending the workshop was held at YaYa village outside Addis Ababa.

The five-day workshop included themes such as critical reading, ethical considerations for operational research, preparing questionnaires and conducting qualitative research. The focus on qualitative research methods was well received by participants. Emiamrew Sisay, who is a PMTCT Coordinator, mentioned rapid appraisal methods and the introduction to using focus group discussions as especially useful in also being able to choose the right tool for the job and appreciating the practical coaching elements during the workshop. In parallel, the participants worked on critically reading and presenting scientific articles and on their thematic group’s protocols. Participants, such as Mesoud Mohammad found the workshop helpful in improving his protocol, as well as learning from other thematic groups: “Almost all of our project proposals need some re-working”, he said: “based on the comments we received and the lecturers time spent in training us.” The thematic groups work on data quality of indicators for immunization, chronic diseases/tuberculosis and malaria.

The first ORCA methods workshop was conducted July 29 to August 03, 2018 at Ya-Ya village in Sululta town. For the 21 participants, the same who had participated in an earlier planning workshop, the aim was to refine each thematic group’s study plan including the individual participants’ plans. The main focus was the selection of the right methodological approach for each group’s research questions.

Sessions during the workshop included:

• critical reading
• questionnaire design for quantitative interviews
• qualitative approaches including rapid appraisal approaches
• an introduction on analysis

Participants worked in their groups to refine the research questions, the choice of methods, the ethical considerations, and a work plan.

During the last day of the workshop, a representative from the Federal Ministry of Health joined the workshop. Participants presented posters with their refined plans and received feedback from the audience.

The ORCA team conducted a first planning workshop for a first batch of ORCA participants from June 26-30, 2018 at Nega Bonger Hotel, on the outskirts of Addis Ababa. The workshop established thematic groups each of the participants will contribute to, as well as a plan of activities for each of the groups and for each of the participants for the upcoming 12 months.

The 21 participants received introductions on the following subject areas:

• Objectives of the ORCA project,
• Research approaches for identifying errors in routine data
• Problem formulation and the design of studies
• Critical reading of research reports
• Ethical issues in operational research
• How to work and interact in a group

The workshop followed a participatory approach, encouraging participants to work in their respective groups and actively contribute. At the end of the workshop, participants had:

• established specific areas of thematic work
• formulated a preliminary research problem
• suggested a potential approach and prepared plans for the next 12 months

During the last day of the workshop, a member of staff from the Ethiopian Public Health Institute (EPHI) – one of the partner institutions of the ORCA project, joined in the discussions following the thematic group presentations.

Methods workshop

The first ORCA methods workshop was conducted July 29 to August 03, 2018 at Ya-Ya village in Sululta town. For the 21 participants, the same who had participated in an earlier planning workshop, the aim was to refine each thematic group’s study plan including the individual participants’ plans. The main focus was the selection of the right methodological approach for each group’s research questions.

Sessions during the workshop included:

• critical reading
• questionnaire design for quantitative interviews
• qualitative approaches including rapid appraisal approaches
• an introduction on analysis

Participants worked in their groups to refine the research questions, the choice of methods, the ethical considerations, and a work plan.

During the last day of the workshop, a representative from the Federal Ministry of Health joined the workshop. Participants presented posters with their refined plans and received feedback from the audience.

The London School of Hygiene & Tropical Medicine (LSHTM) has been awarded more than £1.8 million by the Bill & Melinda Gates Foundation to improve the collection and analysis of health data in Ethiopia.

Like many other African countries, Ethiopia has gaps in their Health Management Information System (HMIS), as well as insufficient mechanisms for data validation, which hinders informed decision making particularly at national policy level. For example, there are often errors and inconsistencies in recorded immunisation data which can impact on the delivery of vaccination programmes.

Through this new grant, researchers will be supporting Ethiopia’s Federal Ministry of Health (FMOH), strengthening technical knowledge to critically assess, interpret and accurately report the wealth of health information. This work will improve the country’s HMIS, as well as leading to improved and more robust measurement of key health indicators such as immunisation coverage, maternal, newborn and child health and nutrition.

Academic teams from LSHTM are already based at the Ethiopian Public Health Institute (EPHI) under two ongoing projects: the IDEAS project, which aims to improve the health and survival of Ethiopian mothers and babies by generating evidence to inform policy and practice, and Dagu, which is working to strengthen maternal, newborn and child health programmes in Ethiopia. Both projects are already contributing to national evaluation, research, and capacity-building activities.

The new grant will expand the existing LSHTM team and allow them to create the resources required to improve both the quality of government metrics and the analytical capacities of key ministry staff.

A gender-balanced group of 40 analysts from the Federal Ministry of Health and affiliated organisations will receive training, mentoring, and coaching through a series of workshops and applied skills-building activities, following the research process from problem formulation to reporting results, and their implications for policy.

In 2015, the FMOH published their Health Sector Transformation Plan. An Information Revolution was a key on the Ministry’s agenda, and outlined the need for timely, accurate and dependable information to allow effective decision-making. Improving quality issues in data, and training and capacity building for analysts were also highlighted as crucial. The Annual Review Meeting of the Ethiopian health sector in 2017 highlighted that inconsistencies in health data remain and need to be further addressed.

The research will be led by Lars Åke Persson, a Professor in Public Health Evaluation, who is based in Ethiopia. Professor Persson, said: “We are delighted to receive this funding from the Bill & Melinda Gates Foundation, to continue vital work in improving the quality and analysis of health data in Ethiopia.

“Ethiopia’s ability to plan and make evidence-based decisions to protect and improve health is impossible if their data have errors.

“This programme of work will allow our experienced analysts to partner with colleagues in Ethiopia to identify errors in health data collection and analysis, fixing them at the root. This will considerably increase the analytic capability at Ethiopia’s Ministry of Health.”

Through the ORCA project researchers from the London School of Hygiene & Tropical Medicine are supporting Ethiopia’s Federal Ministry of Health to strengthen technical knowledge to critically assess, interpret and accurately report the wealth of health information generated in the country.

The analysts participating in the project were recruited in a competitive process, including a background screening, as well as written and oral examinations relevant to the field. 38 were finally admitted from the following three Ethiopian institutions:

• Federal Ministry of Health (27)
• Ethiopian Public Health Institute (6)
• Pharmaceutical Supply and Fund Agency (5)

The selected participants already work at the ministry and institutions in roles related to data management and analysis, have basic knowhow and experience of the country’s health information system and were keen to contribute towards improved  quality of routine health information.

A total of 23 men and 15 women were selected, all between the ages of 20 to 49. A majority of the participants hold master degrees and have chosen careers in the fields they were initially trained in.