Calculating Vaccine Effectiveness in Ontario

From December 14, 2020 - April 17, 2021

[updated June 4]

Koen Swinkels

@koenswinkels

kmswinkels.medium.com

Table of Contents

How Should We Evaluate the Effectiveness of the Vaccination Campaign?        2

Groups and Infection Opportunities        3

The Results        4

Key Points        6

Is the Second Dose Useless (or Worse)?        8

Disclaimers        10

In Conclusion        12

Bonus Content        13

Ontario Public Health has published a report about the effectiveness of Ontario’s vaccination program. The period covered in the report is from the start of the rollout on December 14 until April 17.  

While the report contains useful data on the number of Covid cases in vaccinated and unvaccinated groups, the report’s claims about the vaccines’ effectiveness are not very meaningful and/or too rosy. A closer look tells a more nuanced and complex story, and reveals a crucial and hitherto hidden fact:

The vaccinations are associated with a temporarily increased infection risk. In the case of the second dose, the increase is dramatic.

Before we get into that, we need to clarify how we can calculate the effectiveness of the vaccination rollout so far. This is more complicated than it may seem at first.

How Should We Evaluate the Effectiveness of the Vaccination Campaign?

In an ideal randomized controlled trial we can directly compare the vaccinated group to the unvaccinated group. Both groups spend the same number of days in the trial and are observed for the same number of days. On day 0 one group gets the vaccine and the other group gets a placebo or an alternative vaccine. While different participants may receive their intervention on different days they are all tracked individually and for the same number of days starting on the day of the intervention. With all those data available, calculating the efficacy of the vaccine is pretty straightforward.

The real world, however, is a lot messier than that. Not only are people not followed individually, there are also big differences in the moments when people received their first (or second) dose: In the case of Ontario, some people received the first dose in mid-December while others didn’t receive that same dose until weeks or even months later. This difference makes their average infection risk over the total period very different from each other.

So in contrast with RCTs the data we need to calculate vaccine effectiveness are not readily available. And to obtain them we first need to deal with this problem of differences between people in the duration of the vaccinated period. This is something the report fails to do. As a result, it contains misleading statements such as this one:

Since the COVID-19 vaccination program began on December 14, 2020 and up to April 17, 2021, a total of 3,493,866 individuals in Ontario have been vaccinated (received at least one dose).  Of the almost 3.5 million vaccinated individuals only 0.06% (2,223 individuals) became infected when they were partially vaccinated or fully vaccinated.

The problem is, if we don’t know for how many days people were in that group, this 0.06% number is essentially meaningless. If 80% of them got their first shot in the first month of the 125 day period there would have been many more opportunities for members of this group to get infected than if 80% of them got their first shot in the last month. The 0.06% number would be a much more spectacular result in the former case than in the latter.

This would hold even if we don’t account for differences in the prevalence of the virus between the different periods. If the virus is a lot more prevalent in a period in which very few people were vaccinated than in a later period when many more are vaccinated then —even independently of the effects of vaccination on prevalence— the data would be biased in favor of effectiveness of the vaccine.

Another claim runs into a similar problem:

Only 3.9% of cases post-vaccination were infected 7 or more days after dose 2 administration; as these were fully vaccinated individuals, these are considered breakthrough cases.

That sounds nice but what they don’t mention is what percentage of total infection opportunities was taken up by this group. It makes a lot of difference if the group consists of 1 million people who received their second dose at the end of the first month of vaccination, or if it consists of 100,000 people who received their second dose only 10 days before the end of the total period.

To avoid such relatively meaningless statements and take into account the aspects of duration and group size we need a more sophisticated conceptual framework.

Groups and Infection Opportunities

When it comes to vaccination status we can distinguish between several types of groups in a population:

1. unvaccinated                                                
2. partially vaccinated                                                

1. partially vaccinated but not yet protected (days 0-13 after 1st dose count as 'unprotected')
2. partially vaccinated and partially protected (from day 14 after the first dose)

1. between days 14-27 after the first dose
2. from day 28 after the first dose

3. fully vaccinated                                                

1. fully vaccinated but only partially protected (between days 0-6 after the second dose)
2. fully vaccinated and fully protected (from day 7 after second dose)

We make the distinction based on the number of days that have passed since a dose was received because we expect it to take some time before a vaccine starts to have a protective effect. Moreover, after the protective effect has kicked in it is also expected to increase for some time. To account for this, Public Health Ontario makes the distinctions mentioned above. Other institutions may use slightly different numbers but since we are working with these data we will use these criteria.

Now, to determine the effectiveness of the vaccine we need to:

1. know how many people were in each group on each day: for example, if 5% of people get their first dose in early January we can say that in some sense there will have been more partially vaccinated people in the total period than if these people had gotten their first dose in late March
2. add all these numbers so that we can determine what percentages of the population belonged to what group over the span of the entire period
3. see how many cases occurred in each group in the entire period
4. determine the infection risk for the average member of each group
5. compare the infection risk of the unvaccinated group to the risk of the various vaccinated group

To do this, let’s conceptualize each day as an opportunity to get infected. The total number of such opportunities in a population in a given period then consists of (total population) * (number of days in the period). In the case of Ontario between December 14 and April 17 this means the population of Ontario (14,734,014) * the number of days in that period (125), which equals 1,841,751,750.

By using this ‘infection opportunities’ framework together with vaccination data provided on the site of Ontario Public Health we can now:

1. calculate how many people were in each of the groups mentioned above on each of the 125 days
2. add up the daily numbers for each group
3. divide each of those totals by the total number of infection opportunities for that period in Ontario, so as to:
4. arrive at the share of total infection opportunities taken up by each of these groups, and then:
5. use these results to calculate how much less or more likely each of these groups were to get infected than members of the unvaccinated group, i.e. what the vaccine’s effectiveness was for each of the groups in that period

This is what I have done in this spreadsheet.

The Results

Below are the shares of all infection opportunities for each group, expressed in percentages as well as in share of the Ontario population. The cells in a lighter shade are components of the cells in the darker shade directly above them:

figure 1

We see that unvaccinated people account for the vast majority of infection opportunities, at almost 94%. On the other hand, fewer than 1 percent (0.97%) of infection opportunities were available for people who are fully vaccinated & fully protected (had two doses and are at least 7 days past the second dose). We also see that just over 2% of opportunities were available for people who received a vaccination but are not yet protected.

It is crucial to keep these vast differences in mind when calculating vaccine effectiveness. To illustrate this with an extreme but hypothetical example: If it turns out that 94 times as many unvaccinated people got infected than people who are fully vaccinated & fully protected, that in itself would not mean that the vaccine is very effective. It would merely be a function of how many more infection opportunities were available for unvaccinated people compared to for fully vaccinated & fully protected people.

We can illustrate this by returning to the statements from the report we mentioned earlier:

Only 3.9% of cases post-vaccination were infected 7 or more days after dose 2 administration.

Let’s put this statement into perspective: People who are 7 or more days past their second dose account for only about 15.9% of all infection opportunities. While 3.9% infections for a group that has 15.9% of all infection opportunities is still an impressive result, it sounds less spectacular than the bare statement that there were just 3.9% of infections among fully vaccinated & fully protected people.

Now that we know how many infection opportunities were available for each group, we can calculate the vaccines’ effectiveness for each group by adding the data that is available in the report about how many cases occurred in each group.

We calculate the effectiveness of a vaccine using the following formula:

(n2/N2 – n1/N1) / (n2/N2)

where

n2 is cases in the unvaccinated group

N2 is infection opportunities in the unvaccinated group

n1 is cases in the vaccinated group

N1 is infection opportunities in the unvaccinated group

And then we multiply the result by 100 to get the percentage reduction.

Doing this for the different kinds of vaccinated groups mentioned above we get:

figure 2

Key Points

• The overall effectiveness of the average vaccination so far has been 31.77%. That may sound underwhelming but this number in and of itself is not necessarily all that meaningful. Close to 35% of all infection opportunities for vaccinated people were available to those within the first 14 days of the first dose (2.09/6*100). The protective effect of the vaccine is thought to kick in after about 10 days on average, and to continue to increase for some time afterwards. So including the period in the first two weeks after the first dose brings down the average and does not give a good impression of how effective the vaccine can get.
• When we exclude this initial 2 week period the average effectiveness becomes 59.34%. That’s still pretty good.
• To determine the overall effectiveness of a vaccine we should not, however, entirely overlook that initial 2 week period in which the vaccine is not expected to have much of an effect. While the period does not contribute to the positive effect of the vaccine, it does turn out to have a negative effect:

The group 0-13 days after the first dose actually has a 20% higher infection risk than the unvaccinated group!

This phenomenon of a temporarily increased risk of infection following vaccination has been observed in other countries as well. It is not clear at this point what causes it. It may have to do with:

• the effect of the vaccine on the immune system (the vaccine may temporarily disrupt or suppress the immune system, causing a higher susceptibility for infection)
• the act of going to get vaccinated (driving to and fro, spending time in a space with others etc)
• the behavior of the now vaccinated person (they may take more risks following vaccination)
• the kinds of people who received the vaccine (if groups at higher risk of infection are overrepresented in the group vaccinated so far, then once a more representative sample is vaccinated the difference in risk between the unvaccinated group and the vaccinated group that is within 14 days after the first dose may disappear)^[1]
• or still other factors

What is clear, however, is that this 20% greater risk should be taken into account when assessing vaccine effectiveness, when making policy and when advising people on how to reduce their risk. Although we lack the data it seems likely that the risk is even greater for people in the 10 day period following vaccination, before any protective effect would have kicked in. So we may well be looking at a 30% or more greater risk. It will take some time before the protective effect can undo the effects that this temporary increase has on the average of the entire period after vaccination.

While Public Health Ontario does mention that most post-vaccination cases occur in the group 0-13 days past the first dose, they fail to notice or point out this significant fact that the risk in this group is actually higher than in the unvaccinated group.

• And that is not the only post-vaccination risk increase Public Health Ontario fails to warn about. They write:

The number of post-vaccination cases declines dramatically as time from vaccination increases. The number of post-vaccination cases appears to decrease at about 10 days after dose 1 administration. A marked decrease in post-vaccination cases is observed 28 or more days after dose 1 (p.2)

What is startling, however, is that this risk of infection increases dramatically after the second dose. This may not be immediately clear when looking at the data because the 7 day period following the second dose shows a risk that’s more than 38% smaller than it is for unvaccinated people. But when you compare the risk in that 7 day period after the second dose to the period just before the second dose, we see a 142% greater risk. A 142% greater risk means that people in that group were 2.4 times as likely to get infected as people in the other group.

figure 3

As for the causes of this greater risk for the group that’s within 7 days after the second dose, the same possible explanations just mentioned for the increased risk in the group 0-13 days after the first dose could apply here as well.

But given that there is uncertainty as to which of these explanations is correct, it seems prudent to at least make people aware of this risk and of the possible explanations for it. Doing so could for example be a reason for people to be extra careful in the first week after that second dose.^[2]

Is the Second Dose Useless (or Worse)?

Not only do we see this dramatically higher infection risk in the week after the second dose, it seems that this damage is never completely undone. Peak protection is in the group 28+ days after the second dose, not in the group 7+ days after the second.

If we visualize the differences in the risk of infection for the different groups as a timeline for one average person we get:

figure 4

The risk reduction never goes back to what it was before the second dose. Does this mean the second shot is useless, or even damaging? Not necessarily.

For one thing, if optimal protection is not achieved until, for example, one month after the second dose then the data in this study will not show it. For that we would need to divide the group that’s 7+ days after the second dose into subgroups, for example people 7-20 days and people 21+ days after the second dose. Unfortunately, while we can calculate how many people were in those subgroups for how many days of the period under review, and hence how many infection opportunities were available for these subgroups, we do not have data on how many cases occurred in such subgroups, which makes it impossible to calculate vaccine effectiveness for these groups. It would therefore be useful if Public Health Ontario could obtain and publish data on the number of cases that occurred in such subgroups.  

The more fine-grained such data is, the more useful it can be. For example, right now, assuming that effectiveness continues to increase for a while 7+ days after the second dose, the results will be influenced by when new members were added to this group, or more specifically, by the rate at which this happens, i.e. the rate at which people receive a second dose. If lots of people received their second dose only 10 days before the end of the period under review, then the average effectiveness of the vaccine in the group that is 7+ days past the second dose will be brought down compared to if relatively more people had received that second dose a month earlier.

To compare the effectiveness in that group to the group 28+ days past the first dose it is not just changes in the rate at which people receive a second dose that matter but also changes in the rate at which people receive a first dose, i.e. the ratio between new first doses and new second doses becomes important. So let’s look at how this has changed throughout this period.

figure 5

We see that this ratio has gone down dramatically since its peak from late January to mid-February which was the period in which the great majority of LTC residents received their second dose. After that campaign, the government chose to use the available supply mostly for first rather than second doses.

So the comparatively poor performance of the vaccine in the group that is 7+ days past the second dose cannot be explained by a higher rate at which people are receiving second doses are added to the group, people who have not yet experienced the full protective effect of the second dose yet and who would hence bring down the group’s average vaccine effectiveness.

To the contrary, compared to first dosers the rate of new second dosers has gone down by a lot. But if having had two doses is ultimately (after maybe a month has passed since the second dose) more effective than just the first dose, then the lower rate of new second dosers compared to new first dosers that we start seeing in late February would actually benefit the overall average effectiveness of the second dosers group.

A more likely reason why the effectiveness in that group that is 7+ days past the second dose is lower than the effectiveness in the group that’s 28+ days past the first dose has to do with the types of people in the group of second dosers. If that group consists to a large extent of LTC home residents, and if the vaccine is on average less effective in older and less healthy people, and if relatively youngish and healthyish people who respond well to vaccination are added in large numbers to the group of first dosers, then the average effectiveness of the vaccine in the group that’s 28+ days past the first dose may well be greater than in the group that’s 7+ days past the second dose. But that doesn’t mean that for the same type of population the vaccine would not be more effective 7+ days past the second dose than 28+ days past the first dose. We will have to wait and see how this develops.

Waiting and seeing, however, also complicates things. SARS-CoV-2 is a seasonal virus. In the northern hemisphere cases tend to go down dramatically as summer approaches, even independently of any effects of vaccination, as we saw all around the world last year. So if more people receive a first dose in the spring and then have to wait until August or later for their second dose, they will spend all of their time as members of the group that is 28+ days past the first dose in an environment where the risk of infection is much lower than it was in the period in February when a lot of people became members of the group that is 7+ days past the second dose. If the relative difference between the two groups in the vaccine effectiveness is smaller in a period of low incidence, then the fact that the ratio of second dosers to first dosers was likely higher from late February until April than it will be from late April to August will likely also have temporarily inflated the average effectiveness of the second dose.

Moreover, if the number of LTC home residents and other vulnerable people is much greater in the group of second dosers this could actually make the vaccine seem more effective: Life expectancy for LTC home residents is much lower than for the average person. So if an LTC resident receives a second dose, then dies a few weeks later (no matter what the cause of death is, as long as they didn’t test positive for Covid) then that person will have had many fewer opportunities to get infected than people who stayed alive did. But they would still count as having had the second dose and in the data they are not (to my knowledge) removed from the group of second dosers. So as far as the official data are concerned they would have had the same number of infection opportunities (i.e. the denominator stays the same) as others did while in reality they would have had many fewer such opportunities, and hence there will be many fewer cases among them (i.e. the numerator is smaller).

So while there are factors that in the period under review depress the average effectiveness of the second dose —there being comparatively more vulnerable people in that group who are expected to respond less well to vaccination— there are also numerous factors that have actually inflated it.

To get a better estimate of vaccine effectiveness we then need:

• more fine-grained analysis: divide the second dosers into more subgroups and publish cases data for each of these groups
• remove from the data people who have passed away on the day they pass away
• wait and see what the numbers will be when the group of second dosers becomes a more accurate demographic reflection of the total population of Ontario

Once we have such information available, we can apply the ‘infection opportunities’ framework used in this article to calculate vaccine effectiveness and obtain more reliable results.

Disclaimers

For understandable reasons Public Health Ontario excludes asymptomatic cases in their analysis of cases among vaccine recipients:

Determining the accurate timing of infection (i.e. date of symptom onset) relative to vaccination (i.e. date of dose administration) is necessary when describing cases of COVID-19 infection following vaccination. The main analysis therefore excludes cases that were reported as asymptomatic, re-positive, remote positive, or where an illness onset date was not reported since the timing of infection

They do provide an appendix where that information is available. Unfortunately, we do not have such a breakdown of symptomatic and asymptomatic people for the group of unvaccinated people. So in the analysis above I have included both symptomatic and asymptomatic cases for both the vaccinated and unvaccinated groups.

It is important to note that doing so may undersell the benefits of the vaccine. What we want from a vaccine is less that it reduces the number of cases —which in this context just means people who test positive on a Covid test—and more that it reduces the frequency and intensity of disease. So if for example a vaccine reduces the number of cases by 40% but it also reduces the number of hospitalizations by 90%, it would still be a strong performance.

In the appendix with information about symptomatic and asymptomatic post-vaccination cases there does seem to be a significant effect not just on the number of cases but also on the ratio of symptomatic / asymptomatic cases:

figure 6

in the group that’s 0-13 days after the first dose —the period when the vaccine is not expected to have had much of a positive effect— the ratio of symptomatic / asymptomatic cases is substantially higher than in the other vaccinated groups: 1.8 vs. vs 1.1 (14-27 days after the first dose) and 1.2 (28+ days after the first dose).

To what extent this is due to the beneficial effect of the vaccine is not clear, however. The vaccine can cause adverse events that to a large extent overlap with symptoms of Covid. So any symptoms in those two weeks after the first dose could also be due to the vaccine rather than the infection.

So we would need to know the ratio of symptomatic versus asymptomatic cases in the unvaccinated group and the rate of vaccine-caused adverse events that are consistent with but not caused by a Covid infection to provide a better perspective on this issue.

What is interesting, however, is that in the group that is 0-6 days past the second dose the ratio of symptomatic cases is also higher than in the group that is in the period after (0.7 vs. 0.6) but lower than in the group that is 28+ days past the first shot (1.2). This difference in ratios between the group that is 0-13 days past the first dose and the group that is 0-6 days past the second dose could have to do with the fact that the latter group consists mostly of elderly people who typically have fewer vaccine-caused adverse events that are consistent with but not caused by Covid.

In addition to the matter of a possible difference between the ratio of symptomatic / asymptomatic cases there is another piece of knowledge we need to properly assess vaccine effectiveness: There may be meaningful differences between unvaccinated and the various types of vaccinated groups when it comes to the rate at which infections are actually detected and recorded.

For example, it is possible that because vaccination causes adverse events consistent with Covid, people who were recently vaccinated are more likely to think they may have Covid and decide to get tested. On the other hand, people may be more likely to assume that any symptoms they experience are caused by the vaccine rather than a possible Covid infection and hence be less likely to get tested. Moreover, LTC home residents —who as we saw are overrepresented in the group of second dosers—may be subjected to tests more frequently than other groups as a matter of LTC home infection prevention protocols so that cases are more likely to be detected than they are in other groups of vaccinated or unvaccinated people.

A third disclaimer concerns the cycle threshold values used in Covid testing. Some have argued that a lower cycle threshold should be used to classify something as a breakthrough case (cases in fully vaccinated and fully protected people. i.e. people 7+ days after a second dose):

While this is not necessarily unsound advice, it could have the practical effect of artificially lowering the number of cases reported for certain vaccinated groups compared to other groups of vaccinated or unvaccinated people for whom a higher Ct continues to be used.  

In Conclusion

Measuring vaccine effectiveness in the real world is a messy business. In this article I used the ‘infection opportunities’ framework to make the data that have been made available by Public Health Ontario better suited for such analysis. The most surprising findings of that analysis are that the risk of infection was:

• almost 20% higher for vaccinated people in the 0-13 days following the first dose than it was for unvaccinated people
• dramatically higher (142%) for people 0-6 days past the second dose than for people who are 28+ days past the first dose but who have not yet had the second dose

And that:

• peak effectiveness is in the group 28+ days past the first dose, not in the group that is 7+ days past the second
• average vaccine effectiveness was 31.77%, although when we leave out the period 0-13 days past the first dose effectiveness increases to 59.34%

As we saw, there are numerous factors that may have caused these numbers to not be an accurate reflection of what vaccine effectiveness will be once a more accurate demographic reflection of the population of Ontario has been vaccinated.

But how exactly these factors will play out in the coming months and what the final numbers will be remains to be seen.

Bonus Content

Given the vaccine effectiveness numbers that we calculated here, how many people did we need to vaccinate for every life saved?

To find out we need to know:

• the infection fatality rate of the virus for different age categories
• detection rates for each age category, i.e. what percentage of actual cases are reported for each age category, i.e. are included in the official cases data we have been using
• daily age data for vaccines administered

While I have estimates for the IFR in different age groups, and some data on the number of cases in different age groups, matching these two so that the categories match exactly requires more work and data. I leave that task to others.

Moreover, daily age data for administered vaccines does not seem to be available for all stages of the vaccination rollout.

So to obtain some quick and admittedly flawed results I will for now take into account differences in IFRs but unrealistically assume that infection and reporting rates are the same for each age group, and that the different age groups were vaccinated at the same rate. When it comes to detection rates, I will present results for 3 different scenarios:

• all actual cases are reported, so the cases data we have now include all actual cases
• only 20% of actual cases are reported
• 40% of actual cases are reported

Before presenting the results, it should be noted that the concept this exercise is based on, Number Needed to Treat—vaccinate, in this case—, is not without its problems:

Number-needed-to-vaccinate (NNV) calculations are used with increasing frequency as metrics of the attractiveness of vaccination programs. However, such calculations as typically applied consider only the direct protective effects of vaccination and ignore indirect effects generated through reduction of force of infection (i.e., risk of infection in susceptible individuals). We postulated that such calculations could produce profoundly biased estimates of vaccine attractiveness.

Moreover, the results are based on the average vaccine effectiveness so far, and we have seen in this article that the actual overall vaccine effectiveness in the entire population, once it takes into account the different types of demographics and once some more time has passed, will likely be quite different from the vaccine effectiveness number that we calculated here on the basis of the population that has been vaccinated so far.

So for these reasons and because of the data limitations just mentioned, please take the following results with the appropriate amount of salt.

For a table with all the results see here.

Some interesting findings:

• Assuming a 40% detection rate, to save one 50 year old’s life we vaccinated 42,833 people
• Assuming a 40% detection rate, we vaccinated 745 80 year olds to prevent one 80 year old’s Covid death
• Assuming a detection rate of 20% we will have vaccinated almost 52,000 40 year olds to save one 40 year old’s life.

Looking at these and other results in the table and while keeping all the uncertainties discussed above in mind, it does not seem like a stretch to take seriously the possibility that for many age categories more lives have been lost due to adverse reactions to the vaccine than lives were saved.

Once Ontario starts vaccinating children this will almost certainly be the case.