COVID-19: Let’s Look At Some Studies the Anti-Vaxxers Are On About
by Raywat Deonandan, PhD
Epidemiologist & Associate Professor
University of Ottawa
(I add my credentials to these COVID-19 blog posts in case they get shared. I want readers to know that my opinion is supposedly an educated and informed one)
Today’s topic is just what the title suggests: taking a closer look at some of the studies/documents that are currently driving the anti-vaccine agenda. I’ll only look at three such studies in this post, because I only have so much time before my toddler barges into my office demanding that his socks be interchanged –or something equally as critical. So let’s get to it.
Study #1: that Thai study of heart issues among children who got the COVID jab
This one currently has anti-vax Twitter all a-flutter. It’s a pre-print (not yet peer-reviewed) study called, “Cardiovascular Effects of the BNT162b2 mRNA COVID-19 Vaccine in Adolescents.”
This is a longitudinal cohort study of 301 kids aged 13-18 (2/3 male), who were given cardiovascular work-ups both before and after COVID vaccine injection. The study is gaining a lot of social media traction because it found, most glaringly, that 18% of the kids showed “abnormal” EKG findings:
There is a lot to unpack in this study. First, the cohort study design is great for measuring relationships between variables and an overall incidence rate. But it’s bad for measuring risk above and beyond background rates. To do the latter, you need an RCT or at least a study with an external control group, which this study does not have.
Therefore what is needed before pearl-clutching about 18% abnormal EKG readings is two things: (1) the baseline EKG readings of these kids before they were vaccinated, and (2) the background rate of these EKG abnormalities in the general pediatric population.
Here’s a taste. The study found that the jabbed cohort saw a 7.31% rate of sinus arrhythmia. It’s unclear what type of sinus arrythmia they measured. Was it respiratory? If so, that is so common that one study found it in >60% of healthy 10 year-olds.
I’m not a cardiologist, so my understanding of these matters is at the layperson level. But this reaction from pediatric cardiologist Dr Mark Lindsay is telling:
This study is not without merit, especially around its attempt to measure biomarkers of heart health. But too much is being concluded from its data, which do not show a “gotcha” moment around the safety of COVID vaccines.
Pediatric cardiologist Dr Frank Han has an absolutely beautiful dissection of the study’s strengths and weaknesses. Critically, he concludes: “To the trained observer, there are no shocking findings in this study. Overall, it supports the current body of knowledge regarding COVID vaccination myocarditis.”
Conclusion: this study is intriguing and offers some insights that should be explored further. But it’s far from the “gotcha” moment anti-vaxxers were hoping for, and even states that adolescents should seek vaccination rather than risk infection with COVID.
Study #2: “Serious Adverse Events of Special Interest Following mRNA Vaccination in Randomized Trials”
This is yet another pre-print (again, not yet peer reviewed), which is sure to be put through the ringer by reviewers. But that has not stopped anti-vaxxers from trumpeting its qualities.
The study cleverly re-examined the adverse events (AEs) reported by Pfizer and Moderna for their historic clinical trials that led to the wide distribution of the COVID mRNA vaccines. They found –shockingly– that “the excess risk of serious adverse events of special interest surpassed the risk reduction for COVID-19 hospitalization.” In other words, the jabs were creating more health issues than was COVID… exactly what the anti-vaxxers had been saying since day one!
They concluded this by looking not al all the AEs, but only at AEs “of special interest”, selected based upon their own mysterious internal critera.
Here’s the problem: the study is… well… shit. It smells of fecal content for at least two reasons: (1) p-hacking, and (2) cherry picking AEs.
P-hacking is the act of dishonestly organizing or analyzing data in such a way to extract the barest likelihood of statistical significance, while ignoring the heft of the data which show nothing of the sort.
The result of such shenanigans, though, is that the findings are amplified by those of a certain ideological bent, who presumably have not taken the time to examine the methods in any detail– like this guy, who really should know better, considering that he often touts his statistics bona fides:
Luckily, Dr Susan Oliver has already done a splendid job of explaining this study’s absolute failure, so I don’t need to.
Dr Oliver notes, first, that the authors went to great pains to select only the AEs that would get them close to statistical significance, when comparing jab AEs to COVID AEs. This is where their mysterious internal criteria come to bear. Dr Oliver created this table of only a sample of AEs that the authors chose to include and exclude:
As I hope you can see, the distinctions between the two groups are… questionable, at best. It seems likely that the authors were going to great pains to make delineations that would get them a result of statistical significance, and not necessarily exclusions that are conceptually defensible.
But it didn’t work. In order to get that magical low p-value, they had to combine Moderna and Pfizer data. Each vaccine’s data on its own stubbornly refused to show a problem.
But that wasn’t good enough. They needed to show that the problem (already manufactured) was a big problem. So they did an analysis comparing the AE rate among jabbed people to the hospitalization rate among COVID infections.
This is obviously an apples-to-oranges comparison for a number of reasons, most glaringly:
(1) Adverse events and hospitalizations are two very different things.
(2) If you experienced more than one AE after getting jabbed, you were counted more than once. But if you were hospitalized with COVID, you were counted only once.
(3) The items in the list of vaccine AEs were not serious enough to land you in the hospital. Whereas, clearly, the COVID hospitalizations were by definition serious.
And that’s just a taste of the many things wrong with this study. For the full flavour, I recommend Dr Oliver’s excellent video.
Conclusion: this study is whole lot of nothing. Embarrassingly so.
Study #3: New Brunswick Public Health’s Data on COVID Death by Vaccination Status
Okay, so this was not a study, but a document put out by the public health officials of the province of New Brunswick. A journalist contacted me to comment on it; and I was flabbergasted by the lack of transparency and statistical introspection with its presentation.
This is the table that was most problematic:
It shows that the death rate per 100,000 people is higher among vaccine “protected” people than it is among the “unprotected.”
I had to dig to find the complete report. Nowhere in it is this finding acknowledged or explained. It’s ripe fodder for the anti-vaccination narrative. It’s also completely meaningless. I unrolled my analysis as a Twitter thread. But I’m reproducing it here:
First, this is not a case of the base rate fallacy, which I described here, and which others have attempted to explain, as well. In every jurisdiction in the world, the UNVACCINATED are dying at much higher rates from COVID than are the vaccinated, this is true even when looked at ecologically. New Brunswick is not special in that sense.
Second, this chart does NOT show vaccinated vs unvaccinated. It shows “protected vs unprotected.” And they’ve defined unprotected as “fully vaccinated more than six months, partially vaccinated, and unvaccinated.” Ummm… why?
In other words, there are vaccinated people in both categories. And given that even one dose reduces the likelihood of death, it’s not surprising that there isn’t much difference between the two.
“Protected” on the other hand means “boosted or fully vaccinated less than six months”. I have some difficulty parsing that phrasing. But I take it that, in general, protected = boosted, unprotected = everybody else.
So, who is most likely to have been boosted? Older people. And who is mostly likely to die, in general? Older people. In fact, in NB ~80% of COVID deaths are in people 70+.
In other words, “unprotected” young people are less likely to die than are “protected” older people. So combining al ages masks the protective effect of vaccination. As one of my Twitter contacts put it, “Attempting to frame a 90y/o boosted person as an equivalent risk profile of death as a twice dosed 12y/o (6 months out), makes my brain hurt.”
In epidemiology, this is called “confounding”, where a third variable (“age”) affects the perception of the relationship between two other variables (in this case “vaccination” and “death”.) To remove the effect of age as a confounder, the data should be presented age-stratified.
Another of my Twitter contacts, Kevin Wilson, who is an Epidemiologist in Nova Scotia, offered that he summarizes such data this way:
…which is the proper way of doing it. From Kevin’s chart (of NS data, not NB), it’s clear that the boosted and vaccinated have a much smaller chance of death than do the unvaccinated. He has done this by performing an “age adjustment”, in which the distributions of age groups within each category are forced to resemble one another in order to remove the confounding effect of age. This is one of the very first statistical techniques taught to epidemiology students. It’s basic, bread-and-butter stuff.
But I suspect the bigger issue here isn’t the confounding by age, but the definitions of “protected” and “unprotected”, which DO NOT equate to “vaccinated” and “unvaccinated.”
Conclusion: the table is not incorrect. It’s just misleading and many are likely to take away the wrong message.
And there you have it. If this kind of post is popular, I will consider doing more of them in the future. As always, thanks for reading.
