Measuring vaccine efficacy against variants - Revision history

Alex: /* Discussion of the cross-ratio */

2021-05-12T22:18:22Z

Discussion of the cross-ratio

← Older revision		Revision as of 16:18, 12 May 2021
Line 61:		Line 61:

	The left hand side is estimated by $$\hat\rho$$, while the right hand side is the ratio of the vaccine effects, as relative risks, against variants 1 and 0 $$(\text{RR}_i=\alpha_{i1}/\alpha_{i0})$$, and ends up as something you might call the relative relative risk, $$\rho$$. So under these assumptions we have an easy estimate of the relative vaccine efficacy against one variant versus another. Since there is already a lot of existing knowledge about vaccine efficacy against B.1.1.7, and since B.1.1.7 is currently the dominant variant, we would want to make the comparison between ($$i=0$$) B.1.1.7 and ($$i=1$$) another variant, whereupon we would gain some knowledge of vaccine efficacy against the other variant.		The left hand side is estimated by $$\hat\rho$$, while the right hand side is the ratio of the vaccine effects, as relative risks, against variants 1 and 0 $$(\text{RR}_i=\alpha_{i1}/\alpha_{i0})$$, and ends up as something you might call the relative relative risk, $$\rho$$. So under these assumptions we have an easy estimate of the relative vaccine efficacy against one variant versus another. Since there is already a lot of existing knowledge about vaccine efficacy against B.1.1.7, and since B.1.1.7 is currently the dominant variant, we would want to make the comparison between ($$i=0$$) B.1.1.7 and ($$i=1$$) another variant, whereupon we would gain some knowledge of vaccine efficacy against the other variant.

			Note that existing immunity from past infections, as well as vaccination, will modify the infection probabilities, and $$\rho$$ really measures the relative efficacy of the vaccine against the two variants in the presence of this existing immunity. This is arguably what you want to know, but it means that the answer doesn't generalise so easily to different parts of the world. However, the difference between $$\rho$$ and $$\rho_0$$ will probably be a fairly small under present circumstances because at the moment it seems likely that $$N_{i0}/N_{i1}\gg pq/((1-p)(1-q))$$ where $$p$$ is proportion of people already infected and $$q$$ is the proportion vaccinated. (Under certain assumptions, such as existing immunity working in the same way as vaccinations, you could recover the "raw" relative efficacy, $$\rho_0$$, that is, what it would be if there were no existing immunity.)

	= Estimator and credible intervals =		= Estimator and credible intervals =

Alex: /* Background and summary */

2021-05-08T13:06:19Z

Background and summary

← Older revision		Revision as of 07:06, 8 May 2021
Line 1:		Line 1:
	= Background ~~and summary~~ =		= Background =

	It is of considerable interest to know now how well our vaccines are performing against the Variants of Concern and Variants under Investigation (VOCs and VUIs) that are currently in circulation. Can the variants evade vaccines or not? This knowledge would help us in a number of important policy areas, including how much we need to clamp down on the VOCs, whether hospitals need to plan for a variant-driven resurgence this year, whether/how to restrict entry to the country and whether we need to modify vaccines to better deal with such variants.		It is of considerable interest to know now how well our vaccines are performing against the Variants of Concern and Variants under Investigation (VOCs and VUIs) that are currently in circulation. Can the variants evade vaccines or not? This knowledge would help us in a number of important policy areas, including how much we need to clamp down on the VOCs, whether hospitals need to plan for a variant-driven resurgence this year, whether/how to restrict entry to the country and whether we need to modify vaccines to better deal with such variants.

Alex: /* Estimator and credible intervals */

2021-05-08T11:21:54Z

Estimator and credible intervals

← Older revision		Revision as of 05:21, 8 May 2021
Line 80:		Line 80:
	Example using made-up data (in order to see roughly how narrow a credible interval we might expect):		Example using made-up data (in order to see roughly how narrow a credible interval we might expect):

	Suppose $$N_{00}=60000, N_{01}=5000, N_{10}=340, N_{11}=60$$. The totals $$N_{00}+N_{01}=65000$$ and $$N_{10}+N_{11}=400$$ are very roughly right for the respective counts of B.1.1.7 and B.1.617.2 over the months of March and April~~. The~~ proportions of breakthrough infections $$5000/65000$$ and $$60/400$$ are '''made-up''' for illustration. Using these numbers (and using $$\epsilon=0$$) the 95% credible interval for $$\rho$$ would be $$1.58-2.76$$. So in this '''(fictitious)''' example we would conclude that B.1.617.2 responded to vaccines $$1.58$$ to $$2.76$$ times worse, in terms of relative risk, than B.1.1.7 did. In general, assuming $$N_{11}$$, the number of breakthrough infections in the minority variant, is much the smallest of these numbers, the ratio of the upper and lower end of this credible interval will be about a factor of $$e^{4/\sqrt{N_{11}}}$$, so accuracy mainly depends on finding a decent number of such breakthrough infections. But the worst case, where vaccines are significantly impaired against a variant, should be readily detectable with few datapoints because the centre of the estimate will be significantly above 1 and the range will be relatively narrow.		Suppose $$N_{00}=60000, N_{01}=5000, N_{10}=340, N_{11}=60$$. The totals $$N_{00}+N_{01}=65000$$ and $$N_{10}+N_{11}=400$$ are very roughly right for the respective counts of B.1.1.7 and B.1.617.2 over the months of March and April, but the proportions of breakthrough infections $$5000/65000$$ and $$60/400$$ are '''made-up''' for illustration. Using these numbers (and using $$\epsilon=0$$) the 95% credible interval for $$\rho$$ would be $$1.58-2.76$$. So in this '''(fictitious)''' example we would conclude that B.1.617.2 responded to vaccines $$1.58$$ to $$2.76$$ times worse, in terms of relative risk, than B.1.1.7 did. In general, assuming $$N_{11}$$, the number of breakthrough infections in the minority variant, is much the smallest of these numbers, the ratio of the upper and lower end of this credible interval will be about a factor of $$e^{4/\sqrt{N_{11}}}$$, so accuracy mainly depends on finding a decent number of such breakthrough infections. But the worst case, where vaccines are significantly impaired against a variant, should be readily detectable with few datapoints because the centre of the estimate will be significantly above 1 and the range will be relatively narrow.

	= Discussion of plausibility of assumption 2 =		= Discussion of plausibility of assumption 2 =

Alex: /* Background and summary */

2021-05-08T11:20:08Z

Background and summary

← Older revision		Revision as of 05:20, 8 May 2021
Line 1:		Line 1:
	= Background and summary =		= Background and summary =

	It is of considerable interest to know now how well our vaccines are performing against the Variants of Concern and Variants under Investigation (VOCs and VUIs) that are currently in circulation. Can the variants evade vaccines or not? This knowledge would help us in a number of important policy areas, including how much we need to clamp down on the VOCs, whether ~~we should~~ plan for a variant-driven resurgence this year, whether/how to restrict entry to the country and whether we need to modify vaccines to better deal with such variants.		It is of considerable interest to know now how well our vaccines are performing against the Variants of Concern and Variants under Investigation (VOCs and VUIs) that are currently in circulation. Can the variants evade vaccines or not? This knowledge would help us in a number of important policy areas, including how much we need to clamp down on the VOCs, whether hospitals need to plan for a variant-driven resurgence this year, whether/how to restrict entry to the country and whether we need to modify vaccines to better deal with such variants.

	Here we present a simple suggestion for making use of information that should already be available (though is not currently published) that could, with minimal work and more-or-less in real time, give some idea of our vaccines' efficacy against at least two of the important variants present in the UK right now (May 2021): B.1.617 and B.1.351. The idea is that if publicly available data on variant counts (such as the [https://covid19.sanger.ac.uk/downloads feed from the Sanger centre]) were segregated into counts of those who had and had not been vaccinated, then that should be enough information to deduce some information about the efficacy of vaccines on those variants, provided they occur at least a few tens of times, while not presenting a privacy concern in terms of revealing private health data about identifiable people.		Here we present a simple suggestion for making use of information that should already be available (though is not currently published) that could, with minimal work and more-or-less in real time, give some idea of our vaccines' efficacy against at least two of the important variants present in the UK right now (May 2021): B.1.617 and B.1.351. The idea is that if publicly available data on variant counts (such as the [https://covid19.sanger.ac.uk/downloads feed from the Sanger centre]) were segregated into counts of those who had and had not been vaccinated, then that should be enough information to deduce some information about the efficacy of vaccines on those variants, provided they occur at least a few tens of times, while not presenting a privacy concern in terms of revealing private health data about identifiable people.

Alex: /* Estimator and credible intervals */

2021-05-08T11:17:37Z

Estimator and credible intervals

← Older revision		Revision as of 05:17, 8 May 2021
Line 80:		Line 80:
	Example using made-up data (in order to see roughly how narrow a credible interval we might expect):		Example using made-up data (in order to see roughly how narrow a credible interval we might expect):

	Suppose $$N_{00}=60000, N_{01}=5000, N_{10}=340, N_{11}=60$$. The totals $$N_{00}+N_{01}=65000$$ and $$N_{10}+N_{11}=400$$ are very roughly right for the respective counts of B.1.1.7 and B.1.617.2 over the months of March and April. The proportions of breakthrough infections $$5000/65000$$ and $$60/400$$ are made-up. Using these numbers (and using $$\epsilon=0$$) the 95% credible interval for $$\rho$$ would be $$1.58-2.76$$. So this '''(fictitious ~~example~~)''' would ~~mean~~ that B.1.617.2 responded to vaccines $$1.58$$ to $$2.76$$ times worse, in terms of relative risk, than B.1.1.7 did. In general, assuming $$N_{11}$$, the number of breakthrough infections in the minority variant, is much the smallest of these numbers, the ratio of the upper and lower end of this credible interval will be about a factor of $$e^{4/\sqrt{N_{11}}}$$, so accuracy mainly depends on finding a decent number of such breakthrough infections. ~~The~~ worst case, where vaccines are significantly impaired against a variant, should be readily detectable with few datapoints because the centre of the estimate will be significantly above 1 and the range will be relatively narrow.		Suppose $$N_{00}=60000, N_{01}=5000, N_{10}=340, N_{11}=60$$. The totals $$N_{00}+N_{01}=65000$$ and $$N_{10}+N_{11}=400$$ are very roughly right for the respective counts of B.1.1.7 and B.1.617.2 over the months of March and April. The proportions of breakthrough infections $$5000/65000$$ and $$60/400$$ are '''made-up''' for illustration. Using these numbers (and using $$\epsilon=0$$) the 95% credible interval for $$\rho$$ would be $$1.58-2.76$$. So in this '''(fictitious)''' example we would conclude that B.1.617.2 responded to vaccines $$1.58$$ to $$2.76$$ times worse, in terms of relative risk, than B.1.1.7 did. In general, assuming $$N_{11}$$, the number of breakthrough infections in the minority variant, is much the smallest of these numbers, the ratio of the upper and lower end of this credible interval will be about a factor of $$e^{4/\sqrt{N_{11}}}$$, so accuracy mainly depends on finding a decent number of such breakthrough infections. But the worst case, where vaccines are significantly impaired against a variant, should be readily detectable with few datapoints because the centre of the estimate will be significantly above 1 and the range will be relatively narrow.

	= Discussion of plausibility of assumption 2 =		= Discussion of plausibility of assumption 2 =

Alex: /* The idea */

2021-05-08T11:13:45Z

The idea

← Older revision		Revision as of 05:13, 8 May 2021
Line 22:		Line 22:
	However, we have an extra resource here which means we can construct a measure of vaccine efficacy against variants that requires minimal effort but has a chance of being unconfounded enough to tell us something real. As we are studying two variants at once we can use the B.1.1.7 infection rate as a baseline exposure and measure the efficacy of the vaccine against the other variant using a cross-ratio.		However, we have an extra resource here which means we can construct a measure of vaccine efficacy against variants that requires minimal effort but has a chance of being unconfounded enough to tell us something real. As we are studying two variants at once we can use the B.1.1.7 infection rate as a baseline exposure and measure the efficacy of the vaccine against the other variant using a cross-ratio.

	Suppose, among the unvaccinated population, there are $$A$$ cases of variant 0 and $$B$$ cases of variant 1, and in the		Suppose, among the unvaccinated population, there are $$A$$ cases of variant $$0$$ and $$B$$ cases of variant $$1$$, and in the
	vaccinated population, $$C$$ cases of variant 0 and $$D$$ of variant 1. Knowing $$A$$, what would we expect $$C$$ to be? It's very		vaccinated population, $$C$$ cases of variant $$0$$ and $$D$$ of variant $$1$$. Knowing $$A$$, what would we expect $$C$$ to be? It's very
	hard to say, since it depends not only on how effective the vaccine is, but also a host of other factors such as how many people are		hard to say, since it depends not only on how effective the vaccine is, but also a host of other factors such as how many people are
	vaccinated versus unvaccinated, whether vaccinated people tend to have higher or lower risk to start with, whether their behaviours and so exposures are		vaccinated versus unvaccinated, whether vaccinated people tend to have higher or lower risk to start with, whether their behaviours and so exposures are
	different after vaccination, and so on. Similarly, it's hard to predict $$D$$ from $$B$$. But, if the vaccine works equally well		different after vaccination, and so on. Similarly, it's hard to predict $$D$$ from $$B$$. But, if the vaccine works equally well
	against both variants, we do expect that the ratio $$C:A$$ will be about the same as the ratio $$D:B$$ - most of the unknown		against both variants, we do expect that the ratio $$C:A$$ will be about the same as the ratio $$D:B$$ - most of the unknown
	factors are unknown but ''equal'' for the two cases. Hence $$(D/B)/(C/A)=(AD)/(BC)$$ should be around 1.		factors are unknown but ''equal'' for the two cases. Hence $$(D/B)/(C/A)=(AD)/(BC)$$ should be around $$1$$.

	If the vaccine is $$\rho$$ times less effective against variant 1 than considered above, but everything else is unchanged, then we expect $$D$$ to be around $$\rho$$ times larger than it would otherwise have been, so $$(AD)/(BC)$$ should be around $$\rho$$.		If the vaccine is $$\rho$$ times less effective against variant $$1$$ than considered above, but everything else is unchanged, then we expect $$D$$ to be around $$\rho$$ times larger than it would otherwise have been, so $$(AD)/(BC)$$ should be around $$\rho$$, and in general $$(AD)/(BC)$$ is a measure of how much less effective the vaccine is against variant $$1$$ than it is against variant $$0$$.

	Writing it out with indexes for convenience later, let's say $$N_{ij}$$ denotes the number of people who have been diagnosed with variant $$i$$ and have vaccination status $$j$$. Here $$i=u,0,1$$ with $$i=u$$ corresponding to no infection (not used here, but mentioning it for completeness), $$i=0$$ to B.1.1.7, and $$i=1$$ to the variant under investigation (such as B.1.617). For simplicity here we're taking vaccination status to be a binary $$j=0$$ (unvaccinated) or $$1$$ (vaccinated), ignoring dose timing and distinctions between vaccines. Then		Writing it out with indexes for convenience later, let's say $$N_{ij}$$ denotes the number of people who have been diagnosed with variant $$i$$ and have vaccination status $$j$$. Here $$i=u,0,1$$ with $$i=u$$ corresponding to no infection (not used here, but mentioning it for completeness), $$i=0$$ to B.1.1.7, and $$i=1$$ to the variant under investigation (such as B.1.617). For simplicity here we're taking vaccination status to be a binary $$j=0$$ (unvaccinated) or $$1$$ (vaccinated), ignoring dose timing and distinctions between vaccines. Then

Alex: /* The idea */

2021-05-08T11:10:26Z

The idea

← Older revision		Revision as of 05:10, 8 May 2021
Line 28:		Line 28:
	different after vaccination, and so on. Similarly, it's hard to predict $$D$$ from $$B$$. But, if the vaccine works equally well		different after vaccination, and so on. Similarly, it's hard to predict $$D$$ from $$B$$. But, if the vaccine works equally well
	against both variants, we do expect that the ratio $$C:A$$ will be about the same as the ratio $$D:B$$ - most of the unknown		against both variants, we do expect that the ratio $$C:A$$ will be about the same as the ratio $$D:B$$ - most of the unknown
	factors are unknown but ''equal'' for the two ~~variants~~. Hence $$(D/B)/(C/A)=(AD)/(BC)$$ should be around 1.		factors are unknown but ''equal'' for the two cases. Hence $$(D/B)/(C/A)=(AD)/(BC)$$ should be around 1.

	If the vaccine is $$\rho$$ times less effective against variant 1 than considered above, but everything else is unchanged, then we expect $$D$$ to be around $$\rho$$ times larger than it would otherwise have been, so $$(AD)/(BC)$$ should be around $$\rho$$.		If the vaccine is $$\rho$$ times less effective against variant 1 than considered above, but everything else is unchanged, then we expect $$D$$ to be around $$\rho$$ times larger than it would otherwise have been, so $$(AD)/(BC)$$ should be around $$\rho$$.

Alex: /* The idea */

2021-05-08T11:09:55Z

The idea

← Older revision		Revision as of 05:09, 8 May 2021
Line 28:		Line 28:
	different after vaccination, and so on. Similarly, it's hard to predict $$D$$ from $$B$$. But, if the vaccine works equally well		different after vaccination, and so on. Similarly, it's hard to predict $$D$$ from $$B$$. But, if the vaccine works equally well
	against both variants, we do expect that the ratio $$C:A$$ will be about the same as the ratio $$D:B$$ - most of the unknown		against both variants, we do expect that the ratio $$C:A$$ will be about the same as the ratio $$D:B$$ - most of the unknown
	factors are unknown but ''equal'' for the two ~~cases~~. Hence $$(D/B)/(C/A)=(AD)/(BC)$$ should be around 1.		factors are unknown but ''equal'' for the two variants. Hence $$(D/B)/(C/A)=(AD)/(BC)$$ should be around 1.

	If the vaccine is $$\rho$$ times less effective against variant 1 than considered above, but everything else is unchanged, then we expect $$D$$ to be around $$\rho$$ times larger than it would otherwise have been, so $$(AD)/(BC)$$ should be around $$\rho$$.		If the vaccine is $$\rho$$ times less effective against variant 1 than considered above, but everything else is unchanged, then we expect $$D$$ to be around $$\rho$$ times larger than it would otherwise have been, so $$(AD)/(BC)$$ should be around $$\rho$$.

Alex: /* Discussion of the cross-ratio */

2021-05-08T11:06:47Z

Discussion of the cross-ratio

← Older revision		Revision as of 05:06, 8 May 2021
Line 48:		Line 48:
	The first of these assumptions is saying that the effect of vaccination ($$\alpha_{i1}/\alpha_{i0}$$) is multiplicative. Multiplicativity is a common assumption in analysis of vaccines: to be able to talk about "efficacy" as a single number implies that there is an effect multiplier that applies across a wide range of conditions. $$E_i=E_i(p)$$ can loosely be thought of as exposure of person $$p$$ to variant $$i$$ combined with that person's susceptibility, however it isn't necessarily a quantitative measure of exposure-to-virus as there is no reason to expect the probability of becoming infected to be proportional to viral load. $$E_i$$ is really a baseline risk, which can by assumption be modified multiplicatively by the presence of a vaccine.		The first of these assumptions is saying that the effect of vaccination ($$\alpha_{i1}/\alpha_{i0}$$) is multiplicative. Multiplicativity is a common assumption in analysis of vaccines: to be able to talk about "efficacy" as a single number implies that there is an effect multiplier that applies across a wide range of conditions. $$E_i=E_i(p)$$ can loosely be thought of as exposure of person $$p$$ to variant $$i$$ combined with that person's susceptibility, however it isn't necessarily a quantitative measure of exposure-to-virus as there is no reason to expect the probability of becoming infected to be proportional to viral load. $$E_i$$ is really a baseline risk, which can by assumption be modified multiplicatively by the presence of a vaccine.

	The second of these assumptions is the heart of the matter. It is saying that we haven't tended to vaccinate people who are more at risk from one variant than another. We'll revisit the plausibility of this below, but let's first see the consequence of it.		The second of these assumptions is the heart of the matter. It is saying that we haven't tended to vaccinate people who are more at risk from one variant than another. We'll revisit the plausibility of this below, but let's first see the consequence of it. If $$Y=Y(p)$$ denotes the infected status of person $$p$$ ($$u$$, $$0$$ or $$1$$), then

	\[		\[

Alex: /* Background and summary */

2021-05-08T11:05:09Z

Background and summary

← Older revision		Revision as of 05:05, 8 May 2021
Line 3:		Line 3:
	It is of considerable interest to know now how well our vaccines are performing against the Variants of Concern and Variants under Investigation (VOCs and VUIs) that are currently in circulation. Can the variants evade vaccines or not? This knowledge would help us in a number of important policy areas, including how much we need to clamp down on the VOCs, whether we should plan for a variant-driven resurgence this year, whether/how to restrict entry to the country and whether we need to modify vaccines to better deal with such variants.		It is of considerable interest to know now how well our vaccines are performing against the Variants of Concern and Variants under Investigation (VOCs and VUIs) that are currently in circulation. Can the variants evade vaccines or not? This knowledge would help us in a number of important policy areas, including how much we need to clamp down on the VOCs, whether we should plan for a variant-driven resurgence this year, whether/how to restrict entry to the country and whether we need to modify vaccines to better deal with such variants.

	Here ~~is presented~~ a simple suggestion for making use of information that should already be available (though is not currently published) that could, with minimal work and more-or-less in real time, give some idea of our vaccines' efficacy against at least two of the important variants present in the UK right now (May 2021): B.1.617 and B.1.351. The idea is that if publicly available data on variant counts (such as the [https://covid19.sanger.ac.uk/downloads feed from the Sanger centre]) were segregated into counts of those who had and had not been vaccinated, then that should be enough information to deduce some information about the efficacy of vaccines on those variants, provided they occur at least a few tens of times, while not presenting a privacy concern in terms of revealing private health data about identifiable people.		Here we present a simple suggestion for making use of information that should already be available (though is not currently published) that could, with minimal work and more-or-less in real time, give some idea of our vaccines' efficacy against at least two of the important variants present in the UK right now (May 2021): B.1.617 and B.1.351. The idea is that if publicly available data on variant counts (such as the [https://covid19.sanger.ac.uk/downloads feed from the Sanger centre]) were segregated into counts of those who had and had not been vaccinated, then that should be enough information to deduce some information about the efficacy of vaccines on those variants, provided they occur at least a few tens of times, while not presenting a privacy concern in terms of revealing private health data about identifiable people.

	The proposed process is related to a case-matched observational study, such as carried out in Israel ([https://www.nejm.org/doi/full/10.1056/NEJMoa2101765 Dagan et al, Feb 2021]) and England ([https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/968977/COVID-19_vaccine_effectiveness_surveillance_report_February_2021.pdf Public Health England, Feb 2021]), but with some differences:		The proposed process is related to a case-matched observational study, such as carried out in Israel ([https://www.nejm.org/doi/full/10.1056/NEJMoa2101765 Dagan et al, Feb 2021]) and England ([https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/968977/COVID-19_vaccine_effectiveness_surveillance_report_February_2021.pdf Public Health England, Feb 2021]), but with some differences: