A simulation study of the statistical power and signaling characteristics of an early season sequential test for influenza vaccine safety

PURPOSE: The US Food and Drug Administration monitors the risk of Guillain-Barre syndrome (GBS) following influenza vaccination using several data sources including Medicare. In the 2017 to 2018 season, we transitioned our near real-time surveillance in Medicare to more effectively detect large GBS risk increases early in the season while avoiding false positives. METHODS: We conducted a simulation study examining the ability of the updating sequential probability ratio test (USPRT) to detect substantially elevated GBS risk in the 8- to 21-day postvaccination versus 5× to 30× the historical rate. We varied the first testing week (weeks 5-8) and the null rate (1×-3×) and evaluated power. We estimated signal probability and the risk ratio (RR) after signaling when high-risk seasons were rare. RESULTS: Applying fixed alternatives, we found >80% power to detect a risk 30× the historical rate in week 5 for the 1× null and in week 6 for the 1.5× to 3× nulls. Nearly all testing schedules had >80% power for a 5× risk by week 11. To test the robustness of USPRT, we further simulated seasons where 1% were true high-risk seasons. Using a 1× null led to 10% of seasons signaling by week 11 (median RR approximately 1.4), which decreased to approximately 1% with the ≥2.5× null (median RR approximately 16.0). CONCLUSIONS: On the basis of the results from this simulation and subsequent consultations with experts and stakeholders, we specified USPRT to test continuously from weeks 7 to 11 using the null hypothesis that the observed GBS rate was 2.5× the historical rate. This helped improve the ability of USPRT to provide early detection of GBS risk following influenza vaccination as part of a multilayered system of surveillance.