Tags, Flags & Banners: Evaluating the Application of Information Resources on Vaccine Content on TikTok

4th November 2021

By Ciarán O’Connor

One of the content moderation strategies employed by social media platforms involves placing a banner with links to trustworthy information sources on content that contains potentially misleading or false claims.

TikTok, however, announced they would be placing these banners on all content relating to COVID-19 vaccines. Yet according to a sample tested by ISD, TikTok is failing to tag almost 60% of COVID-19 vaccine-related content posted on its platform with these banners. This is leaving users exposed to often unverified and harmful COVID-19 misinformation and conspiracy content that is left to spread unchecked and unchallenged across TikTok.

_________________________________________________________________________

In a review of 6,241 COVID-19 vaccine-related videos on TikTok, ISD found that 3,608 videos, or 58%, did not feature a banner from TikTok advising people where they can access factual information about COVID-19 and vaccines. This is despite TikTok announcing in December 2020 that it would attach a banner to all such videos to redirect users to “verifiable, authoritative sources of information” about COVID-19 vaccines.

TikTok said it would “detect and tag all videos with words and hashtags related to the COVID-19 vaccine”, regardless of the veracity of the information they include. However, as ISD learned in this analysis, this approach is not only failing to meet its own basic standard in tagging videos that feature related words and hashtags in post captions, it is also ineffective. Many vaccine-related videos do not include text-based references to COVID-19 or vaccines; these continue to go undetected despite high engagement rates.

ISD undertook this short form analysis as an assessment of platform integrity and accountability from TikTok. In their 2021 Q1 and Q2 transparency reports, TikTok said their COVID-19 banners were added to a combined 2.85 million videos and viewed a combined 23 billion times, illustrating the staggering reach these information banners have among users. This makes it all the more important for TikTok to improve and broaden its application of these measures and ensure all COVID-19 and vaccine-related content is accompanied by factual, trustworthy information.

ISD has also researched the application of such labels on other platforms and found there is little standardisation across the industry. In a recent report examining the spread of COVID-19 misinformation on Facebook and the platform’s application of “false information” labels, we found posts that contained fact-check labels in either English or Spanish had a higher average engagement rate per post than those without, and labelled content still racked up tens of thousands of views and likes.

Method

ISD identified and examined seven different categories of vaccine content manually using the TikTok app. We analysed the top 1,000 videos viewable for five hashtags associated with both pro-vaccine and anti-vaccine stances. In addition to hashtags, ISD also examined two Sounds that are popular among anti-vaccine users and have been used to spread misinformation or express solidarity with other people who have not been vaccinated. In three categories there were fewer than 1,000 videos, so we examined the maximum number of videos returned by search results.

It’s important to note that the videos analysed by ISD did not feature solely misinformation or conspiracy content and we sought to examine various categories of COVID-19 vaccine-related content on TikTok to ensure this analysis captured genuine discussions and content about COVID-19 too.

ISD examined each video on TikTok’s app for whether or not it featured a banner that read “Learn more about COVID-19 vaccines,” introduced in December 2020, or “Learn the facts about COVID-19,” introduced in July 2020.

The seven categories examined, and percentage of each featuring an information tag, are presented in the table below:

Findings

Hashtags

ISD found that the hashtags with the most video views, specifically #vaccinated, #unvaccinated and #vaccinesideeffects, featured the highest percentage of tagged videos – 48%, 61% and 54%, respectively. These phrases are quite general and their popularity among TikTok users, including news organisations and TikTok healthcare influencers, could mean they have been flagged as priority items to monitor by TikTok. While these efforts are welcome, it does raise questions as to why the percentages are not higher, what hashtags TikTok chooses to classify as vaccine-related and how TikTok evaluates the content of a video beyond perhaps the use of a single vaccine-related hashtag.

Within videos using the hashtag #unvaccinated, for example, ISD was able to find numerous examples that not only were about vaccines but featured vaccine misinformation. The example below is a Duet (or shared) video that features a clip of a woman who advises against getting vaccinated because, she claims, there is no vaccine for HIV, the common cold or cancer. The claims are overly simplistic and misleading. There is continuous research into and development of vaccines against HIV and the common cold; there is not just one type of cancer; and vaccines are used to prevent various forms of cancer like prostate and cervical cancer. Despite this, the video features no vaccine information banner, has been viewed 1.6 million times and is still available on the platform.

@drewhudson4186#nopoke #unvaccinated #trumptrain #trump2024 #bidensucks#fy #eatfreshrefresh♬ original sound – drewhudson4186

The hashtags #NoVaccineForCOVID19 and #NoToTheJab featured lower percentages of tagged videos. This is concerning as videos that seek to persuade people against getting vaccinated based on misleading, alarmist and false claims can be found easily by searching these hashtags. These phrases are more suggestive and demonstrative of content that is likely intended to sway people away from being vaccinated. Their lower percentage suggests these hashtags and others like them are falling outside TikTok’s monitoring systems and spreading mostly unchecked on the platform.

Sounds

Like any video platform, audio is central to the user experience on TikTok. Songs, music, clipped fragments of popular TV shows or films, the audio of a person speaking — these all make up TikTok Sounds and can be shared separately to videos. ISD detailed in a previous dispatch how TikTok Sounds have been used to promote anti-vaccine misinformation on the platform.

ISD found that vaccine-related content shared as Sounds by other users featured the lowest percentage of content information tags. ISD looked at two Sounds in particular. The first features a hip hop song to which a user added their own lyrics saying “who in the hell still hasn’t been vaccinated? Me [while pointing to themselves]”. The Sound has been used in over 12,000 videos from people using the song to state they are not vaccinated and express solidarity with others. The original video featuring this Sound does not contain misinformation or conspiracy content, but it has gone viral among anti-vaccine users on the platform, including one video (viewed 780,000 times and still live) that used the Sound and described the vaccine as “poison.” Only 32% of videos using this Sound analysed by ISD featured a vaccine information banner.

Screenshot of a post in which the user describes the vaccine as “poison”.

The second Sound was a piece of audio that was the subject of our previous dispatch. This Sound features a man encouraging people against getting a COVID-19 vaccine based on the same misinformation about vaccines for HIV and cancer, as noted earlier in this report. The video from which the Sound originated has since been removed from TikTok but the Sound remains available for use in new videos. This ability to reuse removed Sounds is itself a problematic feature of TikTok and demonstrates how the platform’s unique architecture ensures misleading content can survive and spread on the platform. Only 17% of videos using this Sound analysed by ISD featured a vaccine information banner.

Sounds are central to the spread of content and creation of trends on TikTok, including clips that use false or misleading claims to advise not to get vaccinated, and Sounds continue to pose a problem for the moderation of violative content on the platform. Not only does misleading video content that has been removed continue to circulate as Sounds on the platform, but this analysis shows that TikTok’s efforts to flag vaccine content are ineffective at catching videos that do not feature textual references versus content that uses those references in hashtags or posts. The finding that TikTok is failing to detect such vaccine-related Sounds that do not contain explicit text references is especially concerning.

When is a Vaccine Video not a Vaccine Video?

During this analysis ISD found posts featuring the same video clip both receiving and not receiving a vaccine information banner. For example, this occurred in the case of a video posted by its creator using the ‘unvaccinated solidarity’ Sound described above and featuring the hashtag #unvaccinated. This video was given a vaccine banner. Yet, ISD found subsequent versions of the video, shared in Duets by other users, that were not given the same banner, despite the post featuring the very same information and content (see screenshot below). The reason why is possibly because, in duetting the original clip, the other users’ did not use any vaccine-related hashtags or references in their posts and yet their videos contain the original clip which met the threshold to warrant a vaccine banner.

This raises two questions for TikTok; when is a vaccine video not a vaccine video? More seriously, what other vaccine-related videos, likely including egregious forms of misleading content, are falling outside of TikTok’s net because of the limitations within its own detection and monitoring systems?

Screenshot of four posts. The first post (top left) includes a vaccine banner. The other three videos, posted as duets, feature this same clip but do not include the banner.

Summary

We don’t know how effective information banners are in helping combat misleading or harmful information about COVID-19 or vaccines, largely due to the unwillingness of social media platforms to make testing data publicly accessible. Nor do we know the full scale of misleading vaccine content that continues to be posted and promoted on TikTok and, of course, this report did not focus solely on that type of content. What we do know is that these banners are extremely popular with users and that they offer quick, accessible sources of information.

We also now know that TikTok is failing to tag as much as 58% of such content. Some of these videos appear to be more easily detected and tag-friendly than others. However since TikTok is an audio and video hosting platform, more attention must be paid to non-textual references such as Sounds. What this article demonstrates is that TikTok is failing to meet the standard it set itself and must do better to ensure all users have access to “verifiable, authoritative sources of information” about COVID-19 vaccines. During a public health crisis like COVID-19, it is vitally important that online platforms act quickly to mitigate against such shortcomings when they are identified.

Ciarán O’Connor is an Analyst on ISD’s Digital Analysis Unit with expertise on the far-right and disinformation environment online and open-source research methodologies. Before joining ISD, Ciaran worked with Storyful news agency. He has an MSc in Political Communication Science from the University of Amsterdam and is currently learning Dutch.