May 11, 2020 Addendum: On May 4, IHME announced that its model would stop using a curve-fitting methodology — the basis of the below blog post — and would instead adopt the use of a more traditional SEIR epidemiology model (the same that has been used by Covid Act Now since March 20). While IHME’s adoption of an SEIR model negates the comparative analysis below, the underlying “more models are better” thesis remains valid, and we will maintain this blog post in our archives for posterity’s sake.
We often get asked how these two models compare. This blog post lays out the answer to that question, to help bring clarity to the conversation around COVID modeling.
Here are answers to common questions we get about how these models compare:
What do you think of the IHME model?
Most importantly, the overall message is consistent between IHME and Covid Act Now: aggressive physical distancing measures, implemented as early as possible and maintained until there are other effective ways of managing spread in the community, are critical to prevent overwhelmingly fast growth of COVID in our communities.
We believe that multiple models projecting COVID impacts is a good thing. This is the scientific method at work; having multiple teams with different approaches tackling the problem from different angles will speed up and improve our ability to understand COVID.
The differences between IHME and Covid Act Now projections are the expected result of different approaches to modeling the early and incomplete data regarding COVID. As more is learned about COVID, both models will converge on the correct answer.
What explains such a wide variance in outcomes between these models?
First, COVID is an exponentially growing disease, so seemingly small differences in assumptions can have dramatic effects on the forecasts.
Second, we do not yet have enough complete data to have a comprehensive understanding of all of the variables of the disease. Therefore, interpretation of data and human judgment plays a role in forecasting COVID at this time.
The differences between the forecasts can be explained by the combination of these two factors: human judgment amplified by the exponential nature of the disease.
However, we are learning more and more about the variables driving COVID’s growth every single day. This new information fills in the gaps in data and reduces the amount of judgment, helping us narrow in on the correct answer. This is why both models will both converge on the correct answer over time.
How do the approaches of IHME and Covid Act Now compare?
IHME and Covid Act Now share a common purpose, to enable COVID decision-makers to make better decisions faster, informed by data and modeling.
But the teams have chosen some differences in approach to the problem:
- Disease Modeling vs. Curve Fitting: Covid Act Now has taken the common approach of modeling the underlying mechanics of COVID’s viral propagation (e.g. SEIR modeling). IHME has taken a different, simpler, and more direct approach: best-fitting COVID propagation to the growth curves of prior regions, such as Wuhan, China; Italy; and Spain. Both are valid, but are fundamentally different modeling approaches.
- Scope: Covid Act Now is focused on building a model that can provide nuanced, sophisticated projections in the U.S., including down to the county level. IHME models U.S. states, but not counties, and has a more international focus, providing projections for other countries.
- Scenario Models vs. Confidence Intervals: Since it is unknown how effective U.S. public health interventions will be at this time, Covid Act Now has taken the approach of statistically modeling a range of different potential outcomes (e.g. “Limited Action,” “Lax Social Distancing,” and “Strict Social Distancing”). IHME has taken a different approach to communicate uncertainty by making projections across a range of confidence intervals.
The choices both teams have made are valid, and there is no right or wrong answer.
How do we act on these forecasts when they do not agree?
In short: focus on the worst-case outcome, and ensure we as a society take steps to prevent it.
Multiple models also help increase our understanding of the disease. For example, in weather forecasting, it has been understood that having multiple forecasts that you can compare and average yields more useful overall results for decision-makers. This is why many COVID response planners are combining the outputs of IHME, Covid Act Now, and their own internal forecasts to guide their decision making.