Forty Eight Hours to Results: The Story Behind California’s Covid-19 Model
48 Hours to Results: The Story Behind California’s COVID-19 Model | Oct 2020 | Towards Data Science
Image via @cagovernorCan you imagine planning D-Day without a weather forecast? No? Well then, how about thwarting a rapidly-spreading disease without foretelling its spread? This situation is where the state of California found itself days before Governor Newsom instituted the initial COVID-19 stay-at-home order on March 19, 2020.To put things in context, consider the state the world was in that week. California was trying to understand COVID-19 in an environment where they had only partial, incomplete, and contradictory information. At that point, California had suffered only 18 deaths, while the U.S. had lost fewer than 500 people to the disease. The Chinese government had isolated Wuhan, the hospitals in Northern Italy were overrun with cases, and New York City hospitals were scrambling for sufficient bed capacity. While San Francisco had instituted stay-at-home orders, the disease’s lethality was unclear, and it was inconceivable that a state with 40 million constituents and the fifth-largest economy would enter a partial lockdown.California was effectively flying blind. Wide-scale testing was unavailable, as the CDC had not approved tests yet, and models of COVID suffered from numerous issues. Such issues included resolution only at the aggregate state level and not at a granular county level, no forecast for how mitigation steps like social distancing could affect the epidemic, and no modeling of critical resources like ICU beds and ventilators.In two days, a ragtag group of California public health officials, Johns Hopkins epidemiologists, and Silicon Valley engineers came together to retrofit and operationalize a relatively sophisticated model of the epidemic — providing a line of sight on how policy interventions could affect spread and allocate critical resources like ICU beds and ventilators. This information was instrumental in the decision to enact the first stay-at-home order in the country, which studies show may have averted as many as 1.7 million cases in California by April 6, 2020. Additionally, the model — which has been used to assist numerous state and national governments — was quickly scaled up in order to provide rapid forecasts at the national level. In May 2020 alone, it used over 100- million computer hours for its forecasts.The model supports fine-grained mitigation scenarios within specific geographic boundaries and computes infection forecasts and hospital capacity. To illustrate, one intervention scenario might be moderate social distancing and school closures for four weeks in urban areas, coupled with light masking in rural counties. One can interact with the CalCAT COVID assessment tool to see part of this policy-making apparatus. The truly adventurous can peruse the model source code, which is open source.Screenshot of the CalCAT COVID assessment toolOn March 16, 2020, four days before the stay-at-home decision was issued, all California had was an out-of-date internal forecast and limited, state-level-only projections from outside research labs. Our group initially consisted of representatives from the California State Government, along with a handful of Silicon Valley volunteer technologists. Luckily, one of the state epidemiologists had an existing relationship with Johns Hopkins Bloomberg School of Public Health, where the infectious disease dynamics group had devised a COVID model, which was our best shot. It would allow us to develop county-level projections and create different situational scenarios such as compliance rates with stay-at-home orders. Unfortunately, the model wasn’t tuned for California’s demographics and took more than half a day to compute — consuming valuable time against an exponentially growing virus. We needed a per-county California model that was quick enough to compute and improve as the disease progressed, and that could test scenarios as fast as we could think of them.Already sleep-deprived, we set 48 hours as the deadline to deliver the first batch of results, which reflected the urgency necessary in a fast-moving pandemic. With these time constraints, we had to quickly identify roles and responsibilities and iterate on them every hour. We didn’t have time to create an org chart; our key goal was to find the best-qualified person to take on a task and then support them to the best of our ability.By the evening of that first day, we had built our team and devised the initial deliverable: by how much would California surpass its hospital capacity if the exponential growth continued unchecked, and what could we do about it? Given the urgency, we hyper-focused on the order-of-magnitude differences and settled on three major scenarios: an unmitigated scenario where no drastic measures are taken, which provides an upper-bound; a Wuhan-style lockdown with enforced quarantine, which would be unrealistic in a democracy but provides a lower bound; and a middle-of-the-road solution — a partial lockdown, San Francisco-style. That city had instituted early quarantine a couple of days prior. We didn’t have data for San Francisco, so the epidemiology team creatively modeled San Francisco after Kansas City from the 1918 Spanish influenza pandemic, the next best comparable data.The software team optimized the model and performed a lift and shift onto cloud computing resources to scale the simulation onto thousands of computers.Modeling and forecasting the need for critical public health resources like ICU beds and ventilators requires knowing each hospital’s capacity. Unfortunately, after scouring various government agencies, we learned that this data didn’t exist. To get it, we built a coordinated program in short order, regularly phoning hospitals and capturing this information into spreadsheets. The entire endeavour was akin to the “epidemiological Olympics,” where the team applied, at the highest level, the same well-known epidemiological techniques and software-scaling principles that undergraduate students are taught.In the late evening of Wednesday, March 18, 2020, our team’s model had produced the first set of results for California, and we presented these projections to California’s COVID Response Teams. It was the first time that policymakers had seen models with this level of granularity, and the implications were downright frightening. The next day, Governor Newsom announced California’s stay-at-home order. A few weeks later, on April 10, the governor held a press conference to present the model to the public and tout the value of modeling.Prediction intervals for models are rather large, which is challenging, and it is necessary to internalize in any policy response. Our understanding of the disease is continually improving, from understanding vectors of transmission to hospitalization outcomes, which will tighten estimates. More importantly, changes in human behavior immensely affect the results of any pandemic forecast and are a giant stochastic factor. Humans will change their behavior to buttress any forecast’s negative outcome, invalidating the forecast.Like all good models, the results stimulated rich discussions and required a deeper degree of forecasting (e.g., in which counties should we encourage mask usage?). Policy questions have changed as the pandemic has progressed, and since those early days, we have made substantial updates to the model and examined new and more complex scenarios in response to the evolving situation. These continued updates are necessary for providing pressing and relevant information to policymakers, as the model continues to inform policy decisions in California, other states, and among international partners.The Adhoc California COVID-19 Working Group consists of (in alphabetical order):Article author: Sam ShahVolunteers: Ian Chamberlain, Kathryn Kaminsky, Bob Kocher, Ryan McCorvie, Todd Park, DJ Patil, Kit Rodolfa, Sam Shah, Josh WillsState of California: Manveer Bola, Charity Dean, Pam Haase, Carolyn Nordstrom, Andrea Parriott, Adam Readhead, Amy Tong, Jason Vargo, Michael Wilkening & extended team from California Department of Technology and California Department of Public HealthJohns Hopkins Infectious Disease Dynamics: Juan Dent Hulse, Jacob Fiksel, Kyra H. Grantz, Joshua Kaminsky, Stephen A. Lauer, Elizabeth C. Lee, Justin Lessler, Hannah R. Meredith, Shaun A. TrueloveEPFL: Joseph C. LemaitreUniversity of Utah: Lindsay KeeganAmazon Web Services: Pierre-Yves Aquilanti, Karthik Raman, Arun Subramaniyan, Greg Thursam, Anh Tran