Causal inference methods for small non-randomized studies: Methods and an application in COVID-19.

The usual development cycles are too slow for the development of vaccines, diagnostics and treatments in pandemics such as the ongoing SARS-CoV-2 pandemic. Given the pressure in such a situation, there is a risk that findings of early clinical trials are overinterpreted despite their limitations in terms of size and design. Motivated by a non-randomized open-label study investigating the efficacy of hydroxychloroquine in patients with COVID-19, we describe in a unified fashion various alternative approaches to the analysis of non-randomized studies and apply them to the example study exploring the question whether different methods might have led to different conclusions. A widely used tool to reduce the impact of treatment-selection bias are so-called propensity score (PS) methods. Conditioning on the propensity score allows one to replicate the design of a randomized controlled trial, conditional on observed covariates. Extensions include the doubly robust g-computation, which is less frequently applied, in particular in clinical studies. Here, we investigate the properties of propensity score based methods including g-computation in small sample settings, typical for early trials, in a simulation study. We conclude that the doubly robust g-computation has some desirable properties and should be more frequently applied in clinical research. In the hydroxychloroquine study, g-computation resulted in a very wide confidence interval indicating much uncertainty. We speculate that application of the method might have prevented some of the hype surrounding hydroxychloroquine in the early stages of the SARS-CoV-2 pandemic. R code for the g-computation is provided.