404: RELIABLE INFORMATION NOT FOUND

16 August 2022

This research evaluates the performance of three social media platforms in surfacing reliable news about the war in Ukraine. Using a methodology of searching the keyword ‘Ukraine’ and analysing the top results, ISD found that pro-Kremlin accounts, along with unverified sources, were highly ranked on both Facebook and Twitter. Only YouTube appears to be uplifting media sources and verified profiles.

_________________________________________________________________________________

Social media platforms have repeatedly pledged to fight Ukraine-related misinformation. However, ISD’s previous research has found that misinformation and conspiratorial narratives are some of the most popular content about the war.

To further test the platforms’ performance in prioritising reliable sources and demoting content sympathetic to the Kremlin’s war, ISD analysed a sample of search results for the word “Ukraine” across Facebook, Twitter and YouTube in four languages. The exercise was designed to simulate the information users in different countries are likely to find when looking for updates and news about the situation in Ukraine.

Methodology

To defeat pre-determined algorithms and avoid results based on previous activity, ISD created three new accounts on Facebook, Twitter and YouTube. These accounts were used to search the word “Ukraine” on these platforms while connected to a VPN server in four countries of interest: Italy, the Netherlands, Argentina, and the US. We then took the top ten search results surfaced in each language.

There were three criteria established for whether trustworthy results are being surfaced.

First, whether the content came from a media source. While not all media is reliable, and there are numerous examples of misinformation being spread by media outlets during this war, media outlets nonetheless are more likely to be governed by journalistic ethics and standards than other accounts.

Second, we determined whether the sources were verified profiles, public pages or private profiles. Verified profiles generally belong to more prominent voices or established media outlets or organisations. This does not necessarily mean they are good sources of information, but gives an indication of the importance of the profile. In contrast, profiles that are not verified can include new and fake profiles, fringe media outlets and accounts that present themselves as “media sources” but are not recognised as such.

Third, we examined whether the sources were pro-Kremlin. These included accounts and pages that openly supported the Kremlin, expressed sympathy with the invasion of Ukraine, expressed explicit anti-Ukraine sentiment, or referred to the war as the Kremlin’s preferred term, a “special military operation”.

In total, the final dataset comprised 120 posts: 40 posts per platform, of which 10 were in each language.

Key Findings

• The results varied systematically across platforms, with Facebook and Twitter faring substantially worse than YouTube. YouTube’s results comprised almost exclusively media sources (38/40), and contained zero pro-Kremlin accounts. Facebook and Twitter’s results contained 11/40 and 2/40 media sources respectively.
• Pro-Kremlin accounts constituted 20% of the top search results across Facebook and Twitter. 16 out of the 80 posts analysed across the two platforms met our criteria to be coded as ‘pro-Kremlin’.
• The third result on Facebook, when searching for ‘Ukraine’ in English on a US account, was from a Chinese state-controlled media outlet that has spread Ukraine-related misinformation. This highlights a loophole in Meta’s policy of demoting Russian state-controlled media, which does not apply to its allies.
• 7 out of 10 Spanish language results on Facebook came from pro-Kremlin accounts.
• Of the 120 posts analysed, fewer than half (51) came from media sources.
• Results also varied across languages. English language results included more media sources in comparison to other languages, and featured only one pro-Kremlin account.
• YouTube was found to be the only platform prioritising verified profiles. 36 out of the top 40 results on YouTube came from verified profiles, compared to 13 results on Facebook and 11 results on Twitter.
• None of the pro-Kremlin results came from media sources or verified profiles. Pro-Kremlin search results appeared in higher volumes on platforms that did not prioritise media sources or verified profiles.

What is your source?

An increasing number of people use social media as a news source. Twitter, Facebook and YouTube are among the most popular platforms for this purpose. This means it is crucial that users receive reliable and accurate content when searching for a specific topic.

An initial analysis of the sources in our dataset reveals systematic differences across the three platforms. Among the 40 posts analysed on each platform, YouTube was found to be outperforming Twitter and Facebook. 38 of the results on YouTube came from media sources, while Facebook had 11, and Twitter only two. This is consistent with YouTube’s policy of “prominently [surfacing] authoritative news content” when people search for topics related to Russia and Ukraine.

Figure 1: The graph above shows the number of posts by media sources (in pink) and those coming from other sources (in blue).

Results were mostly consistent across languages, with YouTube scoring higher across the four countries analysed and Facebook performing better in English-language results. Although this is not clear evidence that Facebook is prioritising more reliable sources in English, it is an indication that languages other than English yield less reliable sources.

Figure 2: The graph above shows the number of media sources found in the top ten posts in the four different countries analysed.

The graph above shows that media sources were especially scarce on Twitter. The only country with a media source in its top ten results was the Netherlands, which had two. On Facebook, where 11 media sources were surfaced among the 40 posts, six of the posts were from searches in English, four from searches in Italian and one from searches in Spanish. No media sources appeared in Dutch language results on Facebook.

The failure to surface authoritative and reliable media sources may have inadvertently boosted less prominent accounts and posts. Examples are accounts belonging to private individuals whose posts included strong opinions opposing Zelensky and comments parroting the Kremlin’s official line on Ukraine, most of which received low numbers of comments and likes.

Figure 3: One of the top results for the user based in Italy reads: “they made you believe that Ukraine is the victim after it has been bombing its own compatriots in Donbas for eight years, killing their children. They can make you believe whatever they want because they know that for you the truth is what the newspapers and the TV of the regime say.”

A secondary analysis of the sources distinguished between three kinds of accounts (or channels): private accounts, public accounts, and verified accounts. By private accounts, we mean unverified accounts held by individuals. Public accounts are unverified accounts that represent an organisation or cover specific news topics (e.g. accounts that post news about the war in Ukraine, general news or memes but are not verified profiles). Verified profiles were those with a verified check mark.

The results of this exercise aligned with previous results. YouTube was found to be prioritising verified profiles whereas Facebook and Twitter were not. Facebook surfaced 13 verified profiles in its results, while Twitter surfaced 11.

Figure 4: The graph above shows the volume of private accounts, public accounts and verified accounts among the 40 posts analysed per platform.

Between countries, verified accounts appeared more frequently in results in English when connected to a server from the United States. Twitter, which has the lowest number of verified accounts among the top results, yielded a high number of verified accounts only in the United States (seven results out of 10). Similarly, Facebook yielded six and five results from verified accounts in Italy and the United States, but zero results in the Netherlands, and only two results in Argentina.

Figure 5: The graph above shows the number of verified accounts per country and per social media.

Pro-Kremlin results

All of the accounts identified in the research were coded to establish whether they were pro-Kremlin. An account was only categorised as pro-Kremlin when it openly supported the Kremlin, expressed sympathy with the invasion of Ukraine, expressed explicit anti-Ukraine sentiments, and referred to the invasion of Ukraine as a “special military operation”.

All of the searches were carried out on 29 July, a few days after the lifestyle magazine Vogue published a photoshoot featuring Ukraine’s president Zelensky and his wife. The photoshoot generated a storm of responses across social media and a number of posts criticising Zelensky were also found in the dataset, however, the accounts publishing these posts were not labelled as pro-Kremlin unless they had also published content in support of the invasion. Some edge cases were also found, for example, the first result for Ukraine in English was a criticism of Vogue’s photoshoot which explicitly minimised and raised doubts about the seriousness of the war in Ukraine. Although this post was problematic, the account was coded as not pro-Kremlin as it did not express explicit support to the Kremlin.

Figure 6: The first result on Twitter when searching for Ukraine in English with a server connected to the United States is a post criticising Zelenskiy’s photoshoot and minimising the seriousness of the war in Ukraine.

Across platforms, Facebook scored the highest number of pro-Kremlin posts in the top results. 10 out of the 40 posts analysed came from pro-Kremlin accounts. On Twitter, six out of the 40 posts came from pro-Kremlin accounts, whereas none of the YouTube accounts analysed were found to be pro-Kremlin.

Figure 7: The graph above shows the number of pro-Kremlin accounts among the 40 top results in each platform.

The results were found to differ not only by platform, but also by country. Among the ten pro-Kremlin accounts found on Twitter, seven of them were in Argentina, two in Italy and one in the United States.

Figure 8: The figure above shows the number of pro-Kremlin accounts for each platform in each of the four countries analysed.

In Argentina, the most popular account, found in five out of the 10 results, was called Ministerio de Defensa ruso en español. (“Russian Ministry of Defence in Spanish”). The account used to be called Defensa ruso en español, página no oficial (Russian Ministry of Defence in Spanish, unofficial page) until 16 February 2020 when it changed its name. The about section of the page on Facebook states that the page translates all information on the official page of the Russian Ministry of Defence. It posts military updates as well as content in support of the invasion of Ukraine.

Figure 9: The Facebook transparency page of Ministerio de Defensa ruso en español shows that the page used to be called Ministerio de Defensa ruso en español, página no oficial (Russian Ministry of Defence in Spanish, unofficial page) until 16 February 2020 when it changed its name.

Figure 10: The about section of the page Ministerio de Defensa ruso en español states that the account translates the information provided on the website of the Russian Ministry of Defence into Spanish.

Figure 11: The screenshots above show some of the pro-Kremlin content posted by two of the accounts that appeared as top results in Argentina.In the United States, the third result on Facebook was an article from the page of the Chinese state-controlled media CGTN Frontline, which was found to have spread Ukraine related misinformation. Notably, although Meta had announced that it will be demoting content with links to Russian state-controlled media, it did not address Russia’s allies in the policy failing to tackle the broader ecosystem of pro-Kremlin actors.

On Twitter, a higher number of pro-Kremlin accounts was found in the Netherlands with a total of four accounts. Although the searches were conducted in Dutch and connected to a Dutch IP, some of the top results were in English, including from pro-Kremlin accounts. Among the top results, a public profile which exclusively posts content about the war in Ukraine appeared twice. One of the posts predicts the upcoming “Nuremberg2” which alludes to the Nuremberg trials that were held by the Allies against the representatives of the Nazi regime. Naming prominent figures who recur in numerous conspiracy theories, the post suggests that the upcoming trials will be judging Ukraine’s President Zelensky, Bill Gates, George Soros, and US President Biden, among others. The second post praises the Russian paramilitary organisation Wagner group, stating that it has been “saving the African continent from the US sponsored terrorists”.

Figure 12: Two of the top ten posts in the Netherlands when searching for Ukraine on Twitter.

Another similar account that appeared among the top ten Twitter posts in Argentina was a public profile posing as a European news source. This account was found to be sharing fake claims about Ukrainian refugees as well as content calling Ukrainians “ukronazis”. Among the anti-Ukraine content, the account shared a video featuring a brawl in Galway City alleging that Ukrainian refugees had addressed black people “using the N word” and thereby starting the brawl. The fight, however, did not involve Ukrainians at all but was widely shared online by pro-Kremlin accounts with a range of different stories all involving Ukrainian refugees.

No pro-Kremlin accounts belonged to either a media source or a verified profile. Pro-Kremlin accounts appeared in higher numbers on platforms that did not prioritise media sources or verified profiles.

Figure 13: Content posted by a pro-Kremlin Twitter account.

Recommendations

When users seek information about the Russian invasion of Ukraine on social media, they should be provided as far as possible with accurate, reliable information.

Our research shows that it is possible for platforms to highlight and boost authoritative voices while demoting accounts that spread misinformation. YouTube has taken steps to do so. The result of this effort is evident: YouTube was the only platform to almost exclusively prioritise media sources and verified accounts in its search results, and the only platform that did not feature pro-Kremlin accounts in its top results across the four countries analysed.

Although both have taken steps to demote content from Russian state-controlled media, both Facebook and Twitter have ignored other sources of misinformation and failed to tackle the wider pro-Kremlin ecosystem. During the COVID-19 pandemic, numerous platforms promised to decrease the visibility of accounts that were found to have spread COVID-19 misinformation. However, the same approach has not been deployed in connection with posts about the war.

This research provides a snapshot of the top results that appear for users in different countries when using social media platforms to search for news about Ukraine. The results are concerning. If platforms are serious in their fight against conflict-related misinformation, uplifting authoritative and reliable sources should be in their best interest, and in the interest of the communities they claim to be serving. In time of war, disinformation has the potential of exacerbate the chaos of the information space and result in real life repercussions for the people affected by it, social media companies have a chance to put that right by offering verified sources and demoting problematic accounts and content.