LFD specificity estimate - Revision history

Alex: /* References */ Change github link to dedicated snapshot

2021-03-26T09:39:41Z

References: Change github link to dedicated snapshot

← Older revision		Revision as of 03:39, 26 March 2021
Line 99:		Line 99:
	= References =		= References =

	# Code and data used here, 15 March 2021. [https://github.com/alex1770/Covid-19/tree/~~master~~/LFDevaluation https://github.com/alex1770/Covid-19/tree/~~master~~/LFDevaluation].		# Code and data used here, 15 March 2021. [https://github.com/alex1770/Covid-19/tree/LFDwriteup.2021-03-15/LFDevaluation https://github.com/alex1770/Covid-19/tree/LFDwriteup.2021-03-15/LFDevaluation]
	# Crozier, A. et al. ''Put to the test: use of rapid testing technologies for covid-19'', 3 February 2021. [https://www.bmj.com/content/372/bmj.n208 BMJ 2021;372:n208]		# Crozier, A. et al. ''Put to the test: use of rapid testing technologies for covid-19'', 3 February 2021. [https://www.bmj.com/content/372/bmj.n208 BMJ 2021;372:n208]
	# Funk, S., Flasche, S. ''LFD mass testing in English schools - additional evidence of high test specificity'', 5 March 2021. [https://cmmid.github.io/topics/covid19/mass-testing-schools.html https://cmmid.github.io/topics/covid19/mass-testing-schools.html]		# Funk, S., Flasche, S. ''LFD mass testing in English schools - additional evidence of high test specificity'', 5 March 2021. [https://cmmid.github.io/topics/covid19/mass-testing-schools.html https://cmmid.github.io/topics/covid19/mass-testing-schools.html]

Alex: /* Method */ Make github link dedicated snapshot

2021-03-26T09:38:26Z

Method: Make github link dedicated snapshot

← Older revision		Revision as of 03:38, 26 March 2021
Line 36:		Line 36:
	where $$\operatorname{Po}$$ is the Poisson distribution, $$m_i$$ is the number of LFD positives, $$q_i$$ is the rate of confirmed cases nationally (number of cases divided by age band population), $$n_i$$ is number of LFD tests carried out, and $$i$$ indexes the week. (Test and Trace statistics are given as weekly aggregates, so we work with that.) The variables are $$\alpha$$ (the false positive rate), $$\beta$$ (the relative rate at which cases are picked up by this LFD screening programme compared to nationally), and a third variable representing the day offset which determines exactly which week of national cases corresponds to the week of LFD tests. To estimate $$\alpha$$, $$\beta$$ and the offset, we maximise the log likelihood, which is $$\sum_i[ m_i\log((\alpha +\beta q_i)n_i)-(\alpha +\beta q_i)n_i]$$, up to a constant. To estimate the uncertainty in $$\alpha$$ we use $$1.96J(\alpha,\beta)^{-1/2}$$, where $$J(\alpha,\beta)=\sum_i m_i/(\alpha +\beta q_i)^2$$ is the observed Fisher information of $$\alpha$$, the negative of the second derivative of the log likelihood with respect to $$\alpha$$. The $$1.96$$ arises in the usual way as the number of standard deviations of a normal distribution that gives a 95% confidence interval.		where $$\operatorname{Po}$$ is the Poisson distribution, $$m_i$$ is the number of LFD positives, $$q_i$$ is the rate of confirmed cases nationally (number of cases divided by age band population), $$n_i$$ is number of LFD tests carried out, and $$i$$ indexes the week. (Test and Trace statistics are given as weekly aggregates, so we work with that.) The variables are $$\alpha$$ (the false positive rate), $$\beta$$ (the relative rate at which cases are picked up by this LFD screening programme compared to nationally), and a third variable representing the day offset which determines exactly which week of national cases corresponds to the week of LFD tests. To estimate $$\alpha$$, $$\beta$$ and the offset, we maximise the log likelihood, which is $$\sum_i[ m_i\log((\alpha +\beta q_i)n_i)-(\alpha +\beta q_i)n_i]$$, up to a constant. To estimate the uncertainty in $$\alpha$$ we use $$1.96J(\alpha,\beta)^{-1/2}$$, where $$J(\alpha,\beta)=\sum_i m_i/(\alpha +\beta q_i)^2$$ is the observed Fisher information of $$\alpha$$, the negative of the second derivative of the log likelihood with respect to $$\alpha$$. The $$1.96$$ arises in the usual way as the number of standard deviations of a normal distribution that gives a 95% confidence interval.

	Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/~~master~~/LFDevaluation https://github.com/alex1770/Covid-19/tree/~~master~~/LFDevaluation].		Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/LFDwriteup.2021-03-15/LFDevaluation https://github.com/alex1770/Covid-19/tree/LFDwriteup.2021-03-15/LFDevaluation].

	A negative binomial distribution with a common dispersion parameter was also tried instead of a plain old Poisson distribution (thanks to David Spiegelhalter for this suggestion), but it was found this made very little difference to the result. Perhaps this not so surprising because the datapoints line up so nicely that any vaguely sensible form of regression is likely to lead to a similar best fit line.		A negative binomial distribution with a common dispersion parameter was also tried instead of a plain old Poisson distribution (thanks to David Spiegelhalter for this suggestion), but it was found this made very little difference to the result. Perhaps this not so surprising because the datapoints line up so nicely that any vaguely sensible form of regression is likely to lead to a similar best fit line.

Alex: /* Background and Conclusion */

2021-03-17T11:40:49Z

Background and Conclusion

← Older revision		Revision as of 05:40, 17 March 2021
Line 12:		Line 12:
	This note estimates specificity of LFD tests carried out on secondary school students to be 99.988% (99.980% - 99.995%), equivalent to a false positive rate of 0.012% (0.005% - 0.020%), based on the assumption that LFD true positives at school are proportional to the number of cases detected nationally within the student age group at a suitable date offset. (Note that the range of values given here reflects statistical uncertainty within the modelling assumptions. If the model is wrong then the true value could exceed these bounds.)		This note estimates specificity of LFD tests carried out on secondary school students to be 99.988% (99.980% - 99.995%), equivalent to a false positive rate of 0.012% (0.005% - 0.020%), based on the assumption that LFD true positives at school are proportional to the number of cases detected nationally within the student age group at a suitable date offset. (Note that the range of values given here reflects statistical uncertainty within the modelling assumptions. If the model is wrong then the true value could exceed these bounds.)

	If the false positive rate is 0.012% then we can expect 79% (from 1-0.012/0.056) of the 0.056% found to be positive in the week ending 3 March to genuinely have the disease (this 79% figure is known as the positive predictive value, or PPV). In the future the PPV may decrease if disease prevalence declines, or it may increase because we are now moving from testing only asymptomatic students with LFD tests to testing all students.		If the false positive rate is 0.012% then we can expect 79% (from 1-0.012/0.056) of the 0.056% found to be positive in the week ending 3 March to genuinely have the disease (this 79% figure is known as the positive predictive value, or PPV). In the future the PPV may decrease if disease prevalence declines, or it may increase because we are now moving from testing only asymptomatic students with LFD tests to testing all students (though see caveat below about moving from supervised to unsupervised testing).

	= The Idea =		= The Idea =

Alex: /* Background and Conclusion */

2021-03-17T11:39:35Z

Background and Conclusion

← Older revision		Revision as of 05:39, 17 March 2021
Line 11:		Line 11:

	This note estimates specificity of LFD tests carried out on secondary school students to be 99.988% (99.980% - 99.995%), equivalent to a false positive rate of 0.012% (0.005% - 0.020%), based on the assumption that LFD true positives at school are proportional to the number of cases detected nationally within the student age group at a suitable date offset. (Note that the range of values given here reflects statistical uncertainty within the modelling assumptions. If the model is wrong then the true value could exceed these bounds.)		This note estimates specificity of LFD tests carried out on secondary school students to be 99.988% (99.980% - 99.995%), equivalent to a false positive rate of 0.012% (0.005% - 0.020%), based on the assumption that LFD true positives at school are proportional to the number of cases detected nationally within the student age group at a suitable date offset. (Note that the range of values given here reflects statistical uncertainty within the modelling assumptions. If the model is wrong then the true value could exceed these bounds.)

			If the false positive rate is 0.012% then we can expect 79% (from 1-0.012/0.056) of the 0.056% found to be positive in the week ending 3 March to genuinely have the disease (this 79% figure is known as the positive predictive value, or PPV). In the future the PPV may decrease if disease prevalence declines, or it may increase because we are now moving from testing only asymptomatic students with LFD tests to testing all students.

	= The Idea =		= The Idea =

Alex: /* The Idea */

2021-03-16T12:08:43Z

The Idea

← Older revision		Revision as of 06:08, 16 March 2021
Line 13:		Line 13:

	= The Idea =		= The Idea =

			Imagine in one particular week 100,000 students are tested with LFDs, of which 100 come up positive. Suppose also that in that same week there are 10,000 confirmed cases nationally in the 10-19 age band. Now let's imagine in a different week 200,000 students are tested with LFDs, but only 1,000 cases are confirmed nationally. How many of these 200,000 LFD tests do we expect to come back positive?

			If all 100 LFD-positives from the first week were false, then we'd expect 200 to come back positive in this new week, because with twice as many students being tested, there is twice the opportunity for false positives. On the other hand, if all 100 of the original LFD-positives were true then we'd now expect about 20 to come back positive, because although we're testing twice as many people, infection rates have come down by a factor of ten overall and 100*2/10=20.

			So we see the question is whether the number of positive results from LFD tests tracks (a) the number of LFD tests carried out, or (b) (number of LFD tests carried out)*(rate of infection in the country), or somewhere in between. (a) would indicate false positives, and (b) true positives. Or to put it another way, is the LFD positive rate (positive results per test) a constant, or is it proportional to the national prevalence of the disease, or somewhere in between?

			It's actually easy to determine this because we have nine weeks of data (at the time of writing). We can plot the test positivity rate against national prevalence and see if the graph looks like a flat horizontal line, or more like a sloping line that goes through the origin, or just a random mess. As we see in the figure below, it is quite close to a line that goes through the origin. In such a picture, the y-intercept (how much the line misses the origin) is our prediction of the false positive rate.

	= Method =		= Method =

Alex: /* Background and Conclusion */

2021-03-16T11:33:20Z

Background and Conclusion

← Older revision		Revision as of 05:33, 16 March 2021
Line 11:		Line 11:

	This note estimates specificity of LFD tests carried out on secondary school students to be 99.988% (99.980% - 99.995%), equivalent to a false positive rate of 0.012% (0.005% - 0.020%), based on the assumption that LFD true positives at school are proportional to the number of cases detected nationally within the student age group at a suitable date offset. (Note that the range of values given here reflects statistical uncertainty within the modelling assumptions. If the model is wrong then the true value could exceed these bounds.)		This note estimates specificity of LFD tests carried out on secondary school students to be 99.988% (99.980% - 99.995%), equivalent to a false positive rate of 0.012% (0.005% - 0.020%), based on the assumption that LFD true positives at school are proportional to the number of cases detected nationally within the student age group at a suitable date offset. (Note that the range of values given here reflects statistical uncertainty within the modelling assumptions. If the model is wrong then the true value could exceed these bounds.)

			= The Idea =

	= Method =		= Method =

Alex: /* Method */

2021-03-16T11:32:44Z

Method

← Older revision		Revision as of 05:32, 16 March 2021
Line 26:		Line 26:
	Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/master/LFDevaluation https://github.com/alex1770/Covid-19/tree/master/LFDevaluation].		Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/master/LFDevaluation https://github.com/alex1770/Covid-19/tree/master/LFDevaluation].

	A negative binomial distribution with a common dispersion parameter was also tried instead of a plain old Poisson distribution (thanks to David Spiegelhalter for this suggestion), but it was found this made very little difference to the result. ~~In my opinion~~ this not so surprising because the datapoints line up so nicely that any vaguely sensible form of regression is likely to lead to a similar best fit line.		A negative binomial distribution with a common dispersion parameter was also tried instead of a plain old Poisson distribution (thanks to David Spiegelhalter for this suggestion), but it was found this made very little difference to the result. Perhaps this not so surprising because the datapoints line up so nicely that any vaguely sensible form of regression is likely to lead to a similar best fit line.

	= Results =		= Results =

Alex: /* Results */

2021-03-16T11:31:54Z

Results

← Older revision		Revision as of 05:31, 16 March 2021
Line 36:		Line 36:
	The result of the maximum likelihood estimation is that $$\alpha=0.000115$$, $$\beta=0.777$$, which means the central false positive rate estimate in this situation is 0.011% (0.005% - 0.018%). The best offset turned out to be -3 days, meaning LFD tests conducted in the week ending 2021-03-03 are related to national case numbers (by specimen date) in the week ending 2021-02-28.		The result of the maximum likelihood estimation is that $$\alpha=0.000115$$, $$\beta=0.777$$, which means the central false positive rate estimate in this situation is 0.011% (0.005% - 0.018%). The best offset turned out to be -3 days, meaning LFD tests conducted in the week ending 2021-03-03 are related to national case numbers (by specimen date) in the week ending 2021-02-28.

	This -3 day offset was not quite as expected since it implies the national cases (by specimen date) are anticipating the school LFD results by 3 days. I would have expected something more like a +2 day offset arising as the difference between delays from the date of infection in the two situations: (Situation 1) there is a delay of [https://www.bmj.com/content/372/bmj.n208 around 3.5 days] until you become LFD-positive, then a further delay of around 1.75 delays on average while you wait until your twice-weekly test comes around, making a total delay of around 5.25 days. (Situation 2) there is a delay of [https://www.bmj.com/content/372/bmj.n208 around 5 days] until you become symptomatic, followed by some period of time, guessed to be around 2 days, for you to make up your mind to take a test and actually procure it, making a total of maybe 7 days. However, offsets from -8 to +1 have similar likelihoods so there isn't much to choose between them. I suppose (not investigated thoroughly) ~~this~~ might be due to the fact that cases have been declining with something like exponential decay, which ~~would mean~~ small offsets ~~are~~ equivalent (just inducing a change in $$\beta$$)~~, and only large~~ offsets ~~can be ruled out~~ because the curve of infections doesn't look self-similar enough if you displace it from itself by too much. It is still comforting that more extreme offsets definitely fare a lot worse, since if the correlation between LFD-positives and national cases had been a complete fluke, there would have been no reason to expect it to work better at offset 0 than offset +/-20.		This -3 day offset was not quite as expected since it implies the national cases (by specimen date) are anticipating the school LFD results by 3 days. I would have expected something more like a +2 day offset arising as the difference between delays from the date of infection in the two situations: (Situation 1) there is a delay of [https://www.bmj.com/content/372/bmj.n208 around 3.5 days] until you become LFD-positive, then a further delay of around 1.75 delays on average while you wait until your twice-weekly test comes around, making a total delay of around 5.25 days. (Situation 2) there is a delay of [https://www.bmj.com/content/372/bmj.n208 around 5 days] until you become symptomatic, followed by some period of time, guessed to be around 2 days, for you to make up your mind to take a test and actually procure it, making a total of maybe 7 days.

			However, offsets from -8 to +1 have similar likelihoods so there isn't much to choose between them. I would suppose (not investigated thoroughly) the indifference between these options might be due to the fact that cases have been declining with something like exponential decay, which could make small offsets nearly equivalent (just inducing a change in $$\beta$$). Large offsets really are different because the curve of infections doesn't look self-similar enough if you displace it from itself by too much. It is still comforting that more extreme offsets definitely fare a lot worse, since if the correlation between LFD-positives and national cases had been a complete fluke, there would have been no reason to expect it to work better at offset 0 than offset +/-20.

	There is a possible mechanism by which the correlation between LFD positivity in schools and national confirmed cases could be bogus: the correlation could simply arise from schools' LFD positives being fed directly into the national confirmed cases (which are a composite of LFD-positives and PCR-positives). In fact the number of LFD positives arising from secondary school students is much smaller than the national number of confirmed cases for the 10-19 age band, so this isn't likely to be a major bias, but to check that this was not a significant problem, the program was rerun with the national confirmed cases diminished by the number of LFD-positives arising from schools. This change lead to similar results: a false positive rate estimate of 0.013% (0.007% - 0.020%). To be conservative, we take the union of this range with the original one, and (slightly arbitrarily, though it makes little difference) average the two central estimates to get a final estimate of 0.012% (0.005% - 0.020%).		There is a possible mechanism by which the correlation between LFD positivity in schools and national confirmed cases could be bogus: the correlation could simply arise from schools' LFD positives being fed directly into the national confirmed cases (which are a composite of LFD-positives and PCR-positives). In fact the number of LFD positives arising from secondary school students is much smaller than the national number of confirmed cases for the 10-19 age band, so this isn't likely to be a major bias, but to check that this was not a significant problem, the program was rerun with the national confirmed cases diminished by the number of LFD-positives arising from schools. This change lead to similar results: a false positive rate estimate of 0.013% (0.007% - 0.020%). To be conservative, we take the union of this range with the original one, and (slightly arbitrarily, though it makes little difference) average the two central estimates to get a final estimate of 0.012% (0.005% - 0.020%).

Alex: /* Method */

2021-03-16T11:21:05Z

Method

← Older revision		Revision as of 05:21, 16 March 2021
Line 25:		Line 25:

	Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/master/LFDevaluation https://github.com/alex1770/Covid-19/tree/master/LFDevaluation].		Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/master/LFDevaluation https://github.com/alex1770/Covid-19/tree/master/LFDevaluation].

			A negative binomial distribution with a common dispersion parameter was also tried instead of a plain old Poisson distribution (thanks to David Spiegelhalter for this suggestion), but it was found this made very little difference to the result. In my opinion this not so surprising because the datapoints line up so nicely that any vaguely sensible form of regression is likely to lead to a similar best fit line.

	= Results =		= Results =

Alex: /* Method */

2021-03-15T10:21:58Z

Method

← Older revision		Revision as of 04:21, 15 March 2021
Line 22:		Line 22:
	\[ m_i \sim \operatorname{Po}((\alpha +\beta q_i)n_i), \]		\[ m_i \sim \operatorname{Po}((\alpha +\beta q_i)n_i), \]

	where $$\operatorname{Po}$$ is the Poisson distribution, $$m_i$$ is the number of LFD positives, $$q_i$$ is the rate of confirmed cases nationally (number of cases divided by age band population), $$n_i$$ is number of LFD tests carried out, and $$i$$ indexes the week. (Test and Trace statistics are given as weekly aggregates, so we work with that.) The variables are $$\alpha$$ (the false positive rate), $$\beta$$, and a third variable representing the day offset which determines exactly which week of national cases corresponds to the week of LFD tests. To estimate $$\alpha$$, $$\beta$$ and the offset, we maximise the log likelihood, which is $$\sum_i[ m_i\log((\alpha +\beta q_i)n_i)-(\alpha +\beta q_i)n_i]$$, up to a constant. To estimate the uncertainty in $$\alpha$$ we use $$1.96J(\alpha,\beta)^{-1/2}$$, where $$J(\alpha,\beta)=\sum_i m_i/(\alpha +\beta q_i)^2$$ is the observed Fisher information of $$\alpha$$, the negative of the second derivative of the log likelihood with respect to $$\alpha$$. The $$1.96$$ arises in the usual way as the number of standard deviations of a normal distribution that gives a 95% confidence interval.		where $$\operatorname{Po}$$ is the Poisson distribution, $$m_i$$ is the number of LFD positives, $$q_i$$ is the rate of confirmed cases nationally (number of cases divided by age band population), $$n_i$$ is number of LFD tests carried out, and $$i$$ indexes the week. (Test and Trace statistics are given as weekly aggregates, so we work with that.) The variables are $$\alpha$$ (the false positive rate), $$\beta$$ (the relative rate at which cases are picked up by this LFD screening programme compared to nationally), and a third variable representing the day offset which determines exactly which week of national cases corresponds to the week of LFD tests. To estimate $$\alpha$$, $$\beta$$ and the offset, we maximise the log likelihood, which is $$\sum_i[ m_i\log((\alpha +\beta q_i)n_i)-(\alpha +\beta q_i)n_i]$$, up to a constant. To estimate the uncertainty in $$\alpha$$ we use $$1.96J(\alpha,\beta)^{-1/2}$$, where $$J(\alpha,\beta)=\sum_i m_i/(\alpha +\beta q_i)^2$$ is the observed Fisher information of $$\alpha$$, the negative of the second derivative of the log likelihood with respect to $$\alpha$$. The $$1.96$$ arises in the usual way as the number of standard deviations of a normal distribution that gives a 95% confidence interval.

	Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/master/LFDevaluation https://github.com/alex1770/Covid-19/tree/master/LFDevaluation].		Program and data to reproduce the results shown here can be found in this github repository: [https://github.com/alex1770/Covid-19/tree/master/LFDevaluation https://github.com/alex1770/Covid-19/tree/master/LFDevaluation].