Non-Randomised and Quasi-Experimental Designs | Quantifying impact

Often, random allocation of the intervention under study is not possible and in such cases the primary challenge for investigators is to control confounding.

Members of the Centre for Evaluation organised a multi-disciplinary symposium in London in 2006 to discuss barriers to randomisation, review the issues, and identify practical solutions. The following two papers summarise the arguments presented, drawing on examples from high- and low-income countries:

Alternatives to randomisation in the evaluation of public health interventions: design challenges and solutions
Bonell CP, Hargreaves J, Cousens S, Ross D, Hayes R, Petticrew M, Kirkwood BR. Alternatives to randomisation in the evaluation of public health interventions: design challenges and solutions. Journal of Epidemiology and Community Health. 2011 Jul 1;65(7):582-7.

Alternatives to randomisation in the evaluation of public-health interventions: statistical analysis and causal inference
Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, Hayes R. Alternatives to randomisation in the evaluation of public-health interventions: statistical analysis and causal inference. Journal of epidemiology and community health. 2009 Aug 6:jech-2008.

Some methodological approaches beyond stratification and regression to address confounding in quasi-experimental or non-randomised designs are highlighted on this page:

Within the Centre for Evaluation at LSHTM, this type of work is carried out in close collaboration with the LSHTM Centre for Statistical Methodology, in particular the Causal Inference, Missing Data, and Time Series Regression Analysis groups, as well as LSHTM’s MARCH (Maternal Adolescent Reproductive and Child Health) Centre.

Craig and colleagues from the UK Medical Research Council have introduced new guidance on the use of some of these methods, and others, under the umbrella term of natural experiments:

Using natural experiments to evaluate population health interventions: new Medical Research Council guidance
Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, Ogilvie D, Petticrew M, Reeves B, Sutton M, Thompson S. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. Journal of epidemiology and community health. 2012 May 10:jech-2011.

Difference in Differences

This method is used to evaluate the impact of interventions that are non-randomly allocated to a sub-set of potentially eligible places. The change in the outcomes in places that got the intervention (the ‘difference’) is compared to the change in the outcomes in the places that did not get the intervention: hence the difference-in-the-differences.

This approach requires data from before and after the intervention is delivered, in places the do and do not get the intervention, and is often estimated as the interaction between the change over time and the allocation group (i.e. whether or not a place got the intervention) in a regression model.

It is possible that the places that receive the intervention are different at baseline from the places that do not receive the intervention in terms of the outcome of interest, and this method accounts for this possibility. However, the method assumes that in the absence of the intervention the change over time in the outcome of interest occurs at the same rate in the intervention and comparison places – this is often referred to as the ‘parallel lines assumption’. Therefore, while the method can account for differences at baseline, it cannot account for a varying rate of change over time that is not due to the intervention. This assumption cannot be directly tested since it is an assumption about the counterfactual state: i.e. what would have happened without the intervention, which was not observed. Researchers can look at trends in other related outcomes, or trends in the outcome of interest before the intervention started, to try to find evidence that supports the assumption about the trends that they cannot actually see.

In the following paper, Tim Powel-Jackson and colleagues investigated the effect of a demand-side financial incentives intervention to increase the uptake of maternity services in India using difference-in-differences, and a number of diagnostics to assess the assumptions underlying the method:

Financial incentives in health: New evidence from India’s Janani Suraksha Yojana
Powell-Jackson T, Mazumdar S, Mills A. Financial incentives in health: New evidence from India’s Janani Suraksha Yojana. Journal of health economics. 2015 Sep 30;43:154-69.

Regression Discontinuity

Regression discontinuity is used to evaluate the impact of interventions when allocation is determined by a cut-off value on a numerical scale. For example, if counties with a population of over one million are allocated to receive an intervention, while those with a lower population are not, then regression discontinuity could be used.

Regression discontinuity compares outcomes in places that fall within a narrow range on either side of the cut-off value. For example, any place with a population short of or over one million by, say, 50,000 people could be included in the comparison. This method assumes that places on either side of the cut-off value are very similar, and therefore, the allocation of an intervention based solely on an arbitrary cut-off value may be as good as a random allocation. The method requires few additional assumptions and has been shown to be valid.

It is important to bear in mind that the effect is estimated only for places that fall within a range around the cut-off value, and therefore cannot be generalised to places that are markedly different, such as those with much smaller or much larger populations.

In the paper below, Arcand and colleagues investigated the effect of an HIV education intervention in Cameroon that was allocated according to the number of schools in the town.

Teacher training and HIV/AIDS prevention in West Africa: regression discontinuity design evidence from the Cameroon
Arcand JL, Wouabe ED. Teacher training and HIV/AIDS prevention in West Africa: regression discontinuity design evidence from the Cameroon. Health Economics. 2010 Sep 1;19(S1):36-54.

The Centre for Statistical Methodology at LSHTM provides a guide for conducting Time Series Regression Analysis, including methodological challenges, researchers with expertise, and references on methods and to publications.

Interrupted Time Series

The interrupted time series method is used to estimate the effect of interventions by examining the change in the trend of an outcome after an intervention is introduced. It can be used in a situation when comparison places are not available as all eligible places receive the intervention.

This method requires a large amount of data to be collected before and after the intervention is introduced, and from a number of time points, to allow modelling of what the trend in the outcome would have been if the intervention was not introduced. The model is compared to what actually occurs. Any change in the level of the outcome or in the rate of change over time, compared to the model, can be interpreted as the effect of the intervention.

It is possible that changes in the trend in the outcome may be due to factors other than the intervention. This can be accounted for quantitatively: by investigating events or policy changes that took place at the same time. Alternatively, like the approach used in the difference-in-differences method to assess the counterfactual rate of change over time, researchers may investigate ‘control trends’ in outcomes. This is done by investigating other related outcomes that might be affected by most of the possible alternative explanations for the change in the trend observed, but not affected by the actual intervention.

In the paper below, the authors investigate the effect of a pneumonia vaccine on pneumonia admissions. They considered that changes in the wider healthcare system might have also affected pneumonia admissions, so they investigated the trends in another related outcome: admissions for dehydration. The assumptions made were that the majority of the possible alternative explanations, such as policy changes or changes to delivery of healthcare, would have affected dehydration admissions to the same extent as pneumonia admissions; that dehydration admissions would not be affected by the vaccine; and that pneumonia did not cause dehydration in large amounts. Using this approach, they were able to show more convincingly that the vaccine brought about the change in the trend.

Decline in pneumonia admissions after routine childhood immunisation with pneumococcal conjugate vaccine in the USA: a time-series analysis
Grijalva CG, Nuorti JP, Arbogast PG, Martin SW, Edwards KM, Griffin MR. Decline in pneumonia admissions after routine childhood immunisation with pneumococcal conjugate vaccine in the USA: a time-series analysis. The Lancet. 2007 Apr 13;369(9568):1179-86.

In another paper, below, Lopez-Barnal and colleagues used interrupted time series analysis to investigate the effect of the 2008 Financial Crisis on suicide in Spain.

The effect of the late 2000s financial crisis on suicides in Spain: an interrupted time-series analysis
Bernal JA, Gasparrini A, Artundo CM, McKee M. The effect of the late 2000s financial crisis on suicides in Spain: an interrupted time-series analysis. The European Journal of Public Health. 2013 Jun 25:ckt083.

Synthetic Controls

Synthetic controls is a relatively new method for evaluating the impact of interventions using data from places that did not get the intervention, collected over time. The method works by first looking at the trends in the outcome of interest before the intervention was introduced. The data from various places that do not ultimately get the intervention are each given a weight so that the weighted-average of their data look as much as possible like the trend in the places that will get the intervention. This weighted-average is the ‘synthetic control’. The weights, unchanged, are then applied to the places without the intervention after the intervention has been introduced, and this weighted average is compared to the actual trend in the place with the intervention. This comparison can be used to estimate the impact. Similar to the other methods discussed earlier, researchers must assume that there is not another intervention or policy change that is happening in the places getting the intervention at the same time. The method requires a lot of data, both from many places and over time. It does not use or require parameterised models, so inferential statistics are calculated using permutations rather than more traditional methods.

In the paper below, Abadie and colleagues introduced the method and applied it to investigate the impact of a tobacco control policy change on cigarette consumption in California, by comparing the trend in California with a weighted-average of the trends in the other states in the USA.

Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program
Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American statistical Association. 2012.