They had been presented with twenty lines, in all kinds of colors. It was the week before the Netherlands would be locked for a third time. The members of the Outbreak Management Team looked at steep curves, wide waves, small swells. In this way Omikron could fill the hospitals. It could be better than expected, depending on whether the sliders in the RIVM computer model were pushed to the optimistic side – or it could be disappointing. OMT member Marc Bonten also saw this pessimistic line: a threatening red stripe that rose well above four thousand occupied IC beds. “The pitch-black scenario.”
That scenario was the only one that made it to Jaap van Dissel’s PowerPoint presentation. At the ‘technical briefing’ on December 21 before the House of Representatives, the OMT chairman only showed the most threatening lines, to underline the need for closed museums, community centers, schools and hairdressers. “There are an extraordinary number of uncertainties,” said Van Dissel, and everything was “filled in the bleak direction”, but this could happen. Bonten does not know why he had not shown a few more cheerful scenarios, „because they were also realistic. But that’s not what you base your advice on.”
Within a week, the bleak scenario turned out not to match reality. As the gap grew, criticism of ‘the RIVM’s model’ grew. Were the predictions correct? Was the computer working properly?
This gnaws at Sake de Vlas, professor of infectious disease modeling at Erasmus University in Rotterdam. He does not know how the sliders are, he cannot test the findings. The RIVM model that calculates the demand for care, and which the government is heading towards, is like a car whose hood cannot be raised.
Together with psychology professor Denny Borsboom of the University of Amsterdam, he is once again calling on the government to involve many more researchers in the fight against corona. They both emphasize: this is mainly a question for the minister. They have no substantive criticism of the modeling work of RIVM. They call this very good quality, the main modeller has international renown.
But more is needed. More public data, more models, more academic departments, more disciplines, more substantive discussion, looking further ahead. De Vlas says via Teams: “We have been in the crisis for two years and we still do not get the details of how the predictions of the RIVM come about. Open the shutters, the Netherlands deserves a second opinion.” After all, testing each other’s findings is standard scientific practice.
The professors are hopeful that things will change under the new Minister of Health Ernst Kuipers. Didn’t he say that it would be good to put ‘other models’ next to those of the RIVM that calculate ‘a broader palette’?
Idea 1
Have a look under the hood
The Center for Infectious Disease Control of the RIVM, founded after SARS, employs the most infectious disease modellers in the Netherlands, and the most extensive computer models with which it calculates the spread of infectious diseases. It is therefore logical that RIVM started making the corona predictions. Two years later, the RIVM’s predictions are still leading, and are virtually the only ones that the House of Representatives sees. That, the scientists say, makes the results less verifiable.
The most important tool of the RIVM modellers is a ‘transmission model’, which predicts infections and hospital admissions up to a month or so in advance. This uses large datasets about people-to-people contacts, hospitalization rates, infections and vaccinations.
That calculation model must be continuously adjusted. Things as yet unknown, such as how contagious the latest variant is, how long Omikron patients are in hospital, how well the booster shot protects, all have to be estimated in the heat of the moment.
Professor De Vlas wishes that he and others could run this model on the servers of their own department with the same data and that they could pull the sliders themselves. “You want to watch, especially at critical moments, during the transition from one variant to another, from Alfa to Delta to Omikron. Don’t they see it too negatively? Are the assumptions correct? Then you can talk.”
Also read: ‘I saw the first results and thought: holy fuck, that timeline’
But the code is not public, and he doesn’t know how the sliders are either. He also does not have all the datasets. For example, the raw data of hospital admissions are not public, because they can be traced back to individuals.
This is arranged differently in the United Kingdom, explains mathematician epidemiologist Julia Gog of Cambridge University. She is one of the leaders of a consortium of universities that all run corona models for, among others, SAGE, the British OMT. “Sharing data turned out to be a huge hassle in terms of privacy. All kinds of departments had to sign agreements. It keeps the lawyers busy. But its importance was recognized early on.”
There is a lot of exchange about the assumptions between the departments. Gog: “It’s ad hoc and it depends on the group. One puts everything on a site, the other shares less. But there is constant discussion, and it should be. Results should be reproducible.”
That the RIVM models are a kind of black boxes are, argues head modeller Jacco Wallinga of the RIVM. He points out that his group does not run one model, but also calculates other things, such as the reproduction number, and the effect of the vaccinations. The results are tested against each other by the modellers, and they often talk to international colleagues.
Others can also create transmission models, Wallinga says. But the law prohibits RIVM from disclosing the computer code in which privacy-sensitive information is processed. Anyone who does want to work with raw data “must request it from NICE”, the foundation that keeps track of recordings. “Or push for new legislation.” And that, in the hectic pace of a pandemic, he cannot document all assumptions in new forecasts, “there is just no time for that”. The OMT does question him critically, confirms Marc Bonten.
But as long as not all data and settings are public, the forecasts cannot be tested. And so it was possible that the House of Representatives saw a threatening line, and that no one drew other lines next to it. MPs did not ask for it either.
Idea 2
Create competitive models
‘Tell five researchers ‘make a model’ and they will all come up with something different, all of which will give different results. And you want that!” says Quirine ten Bosch, infectious disease modeler at Wageningen University. “By examining those differences, you discover the biggest uncertainties and how much impact they have on your forecast.” That is why the World Health Organization WHO has been asking for several transmission models from different groups for diseases such as dengue and Zika for about five years, says Ten Bosch: “The results are compared or combined. The combination is almost always better than one model.”
With this idea in mind, Austria will have estimates made from the start of the pandemic by a consortium of three modeling groups: the Austrian ‘RIVM’, the technical university in Vienna and the medical university (MedUni) in this city. Using the same dataset, but each with their own model, the three groups make weekly predictions for infections, hospitalizations and IC occupancy – often with different results.
“As a result, you have a weekly discussion within the consortium about how well your models work. Where do the differences come from? How big is the uncertainty?” explains complexity researcher Peter Klimek, who leads the modeling team at MediUni. For example, the modellers use mobility data from telecom companies to see to what extent citizens stay at home during a lockdown. “The three groups put the data in their model in a slightly different way. As a result, you get different expectations about the reduction in the number of contacts and therefore the number of infections.”
The discussions and accompanying analyzes take much more time than the modeling itself, according to Klimek, who was recently named Scientist of the Year in Austria. “But they are very important, because the models are very sensitive to assumptions” – the sliders. Take the numbers you can depend on how sickening the Omikron variant is. “It makes a huge difference whether you estimate the number of hospital admissions to be 50 percent lower or 75 percent lower,” says Klimek. “If we can’t agree on something like that, we’ll show different scenarios to policymakers.” The source code of the models is not publicly available.
Working with multiple models can be useful for the Netherlands, thinks Sake de Vlas of Erasmus MC. According to him, the RIVM model that calculates the demand for care is based on a homogeneous population, in which everyone within an age group behaves the same. “But a new virus variant first spreads through people who have a lot of social contacts, often young people. After that, the spread weakens, because the virus finds it more difficult to reach people with few contacts. Complex models take those kinds of differences into account.”
There are many places outside RIVM where corona is considered, says RIVM’s Jacco Wallinga. For example, at the LCPS, which distributes patients across the hospitals and predicts the demand for care with a limited model. And TNO, which also made forecasts for healthcare and shared them with RIVM and the safety regions. TNO does not charge for corona full-time, for budgetary reasons.
Departments that want to count like the RIVM can find information on numerous websites, says Wallinga: “I would say: throw yourself in.”
We did that at the beginning of the pandemic, says De Vlas, with calculators in Rotterdam, Amsterdam, Leiden, Wageningen and Utrecht. “Then there was one variant, and no one was vaccinated. Then we made it with the existing models. Now you need a whole team just to sort out all the new information.” That costs a lot of money. The subsidy has now stopped.
Ultimately, setting up alternative modeling groups is a matter for the Ministry of Health. And it’s something the OMT could push for. OMT member Bonten sees something in it: “Because of the four-eyes principle. And there is also a certain vulnerability in the system. To put it this way: what if Jacco Wallinga gets corona?” At the beginning of the pandemic, Bonten argued in favor of it: “But that did not happen in the hectic pace.”
Idea 3
Calculate more than infections
Many people are tired of the measures. And that influences how effective a measure is. How you put that in the models again strongly determines the outcome, says psychology professor Borsboom. “Perhaps even stronger than assumptions about the virus itself”, something De Vlas agrees.
Also read: What have we learned about human behavior?
Borsboom does not know how RIVM currently calculates with growing ‘pandemic fatigue’. “We also know that if you withdraw a measure, people will compensate. Young people are going to catch up, party. Is that in it?”
In the Netherlands, several groups rely on behaviour, Quirine ten Bosch knows: “TU Delft is making a model of how people spread out in space. I work in a consortium that calculates how indoor spaces can be opened safely.” But those insights are not yet processed in the RIVM forecast models.”
The United Kingdom is also struggling with this, says Julia Gog of Cambridge. Behavioral science enough, but “it has only a small place in the models.”
At the beginning of 2020, Denny Borsboom launched the platform with colleagues Science vs Corona to broaden the corona calculations. Economists, psychologists, historians, modellers, mathematicians and citizens took part, at the peak there were about two hundred people.
The platform ran for a while, but Borsboom would like to continue it structurally. “You want to map the consequences of the pandemic and the measures at all levels. Infections, but also learning disabilities. Whether the curfew had an effect. Then you as a government can make choices more transparent. And when people understand why something was chosen, they agree more easily, even if they disagree.”
“In such a pandemic you have to deal with a lot of uncertainties,” says Ten Bosch. “One of these is model uncertainty, the chance that your model is incorrect. That’s why you just have to have a lot of it.”
#hood #RIVM #ideas #broader #corona #strategy