Recfluence

Recfluence is an analysis of YouTube's political influence through recommendations. This is important, because the algorithm that determines suggested videos has become an influential but underappreciated part of politics. A Pew survey found that 18% of US adults consume news through YouTube, and — according to YouTube Chief Product Officer Neal Mohan — 70% of the watch time is from YouTube’s suggested videos.

Data Collection Process

A list of political channels is manually created, categorized and improved over time. An automated process run daily. It collects:

• Channels information & stats (e.g. subscribers, country)
  • from the YouTube API.
• Videos information & stats (e.g. views, likes, title), captions
  • by scraping the website from a US IP address.
  • since 1st Jan 2018
  • updated daily stats for video's younger than 120 days
• Recommendations (i.e. from video A to video B)
  • by scraping the website from a US IP address
  • for video's younger than 120 days old (max 10)
  • if no young video's, take the latest video's recommendations.

This is then analysed to provide statistics and data which is freely available for researches and media.

Channels Included

Channels were included if the met the following criteria

• 10k+ subscribers. If subscriber data is missing/lower, still include if video's average above 10k views
• Significant focus (more than 30% of content) on US political or cultural news/commentary. I considered cultural commentary was anything from the ISideWith social issues list

There is no definitive list of YouTube channels, so a variety of techniques were used.

• Reviewed the following lists:
  • https://www.adfontesmedia.com/
  • https://blog.feedspot.com/political_youtube_channels/
  • https://socialblade.com/youtube/top/tag/politics/videoviews
  • https://channelcrawler.com/
  • custom algorithm prioritizing most recommended channels from existing seed channels. ChannelExclude.csv is was used to remove recommendations once reviewed.
  • Data & Society Alternative Influence Report
  • Ribeiro et al. Auditing Radicalization Pathways on YouTube
• Related videos/channels from existing channels
• Searches for keywords from ISideWith social issues list
• Suggestions from reviewers

This is not a repeatable process, but by using a variety of methods we are able to collect a much more comprehensive collection of relevant political channels than we could otherwise. By having the most complete list of channels possible, we are able to get a more accurate understanding of YouTube recommendation and we value this more than repeatability.

Channel Classification

To understand the influence of recommendations we need to classify channels using fairly soft criteria to be relevant to way people discuss politics. To limit the subjectivity in this, a well defined process was followed to 'tag' channels appropriately based on the content of the videos. 3 or more reviewers have independently classified > 90% of the channels and the majority opinion used. The full list of channels and each reviewers notes and classifications are available to review and use as you wish.

The following describes the process reviewers used to classify each channel:

Left/Center/Right

For news:

• Compare https://www.adfontesmedia.com and https://mediabiasfactcheck.com/. If they exist in those lists and were in agreement then I accepted that category. Otherwise I used the same process as with the commentary channels.

For political/cultural commentary I considered all of the following:

• Self identified political label, or support for a party
• One sided content on divided political topics of the day (e.g. Kavanaugh, Migrant Caravan)
• One sided content reacting to cultural events topics of the day (e.g. campus protests, trans activism )
• Clearly matches Democrat (left) or Republican (right) views on issues in the ISideWith poll

If these considerations align in the same direction then the channel is left or right. If there was a mix then they are assigned the center/heterodox category.

To calculated the majority view, Left/Center/Right are coded as -1, 0, and 1 respectively. The average is taken then rounded back to the nearest Left/Center/Right value.

Hard Tags

Any combination of the following tags can be applied to a channel. Hard tags did not require any judgement and were classified by Mark. Soft tags were reviewed by each reviewer.

Soft Tags

I used these heuristics to decide on the right soft tags to use

• A natural Category for US YouTube content. Many traditional ways of dividing politics are not a natural category for YouTube channels. In general, YouTubers are providing reaction and sense making to other channels or current events in the United States. In this way categories in align with their stand against positions are more natural.
• Topical/Interesting. The tag needs to be interesting in some way to the current meta-discussion about YouTube’s influence on politics. Intention with this is that topical cultural labels are hard to define. In this case I tried to find specific positions that could be mixed together to re-create these softer cultural categories.
• Able to judged by the content itself. It is important not to rely on outside judgements about the channels content. It's important to interpret the content with full context, but there should be no mind reading, or relying on judgement from other sources. Enough Channels. It shouldn't be a minor niche, unless it is important for the radicalization pathway theory.

To calculate the majority view, each soft tag is assessed independently. For each tag, the number of reviewer with that rag must tally to more than half.

Some tags/classifications reviewers agreed on most of the time (e.g. left/center/right, and conspiracy, anti-theist). Others there was only slightly better than random chance agreement (e.g. Educational, Provocateur). The graphic below shows a measure of how much reviewers agreed on channel classifications beyond change where 0 is purely random and 1 is always in agreement.

Relevance

One limitation of the original analysis was that some late night shows -- with huge viewership -- weren't as political as most channels yet overwhelm the analysis. To fix this problem, each reviewer gave a relevance score representing the portion of content relevant to US politics & cultural commentary. In the analysis, relevant views/recommendations are adjusted according to the average reviewer relevance score.

Data

Download

Download the latest data used for the diagrams. The full raw video and recommendations data can be provided on request - either as files or database access (at cost).

Calculations and considerations

All source & calculations are available to review and use for your own purposes.

• R Source Statistical calculations
• SQL Source Data transformation and basic calculations (using dataform)

Calculation	Description
impressions	An estimate for the number of times a viewer was presented with a recommendation. I.e. we count each of the top 10 recommendations for a video as an "impression". Only YouTube knows true impressions, so we use the following process crete an estimate: Consider each combination of videos (e.g. Video A to Video B) (A to B impressions) = (recommendations from A to B) / (total recommendations from Video A) x (*A's views) x (recommendations per video=10) This can be aggregated to give impressions between channels, categories etc...
relevant impressions	(A channel's relevance %) x impressions
channel views	The total number of video views since 2018 Jan 1
daily channel views	(channel views) * (days in the period videos have been recorded for the channel)
relevant channel views	(daily channel views) * (channel relevance %)
impression advantage	The ratio of impressions sent compared to received.

Ledwich & Zaitsev Group Derived from the list of channel tags and left/center/right classification.

Recfluence Visualization

Channel Landscape

This gives an overview of the "landscape" of channels, clustering like channels together using a force directed graph. Channels that have mutual recommendations are attracted un-connected channels repel. The orientation and placement on x/y don't have no specific meaning and will change each tim it is updated.

Understanding Impression Flow

This shows the "flow" of impressions as a sankey diagram. When no selection have been made, it shows the impression flow between categories

When a channel/category is selected/highlighted it shows the flows through it

Novel Interactivity

• search for channel (top right)
• zoom and drag on the Channel Landscape
• hover over a channel to see its details (details at top, impression flow below/right)
• choose different categories to color by
• highlight or select legend items

Limitations

Analysis Q & A / Notes

• Logged-off users are used to collect recommendations. Given the algorithm heavily weights user history, why would you expect this estimate of impressions would match real-world recommendations?
  Estimating impressions with logged-in users is difficult, you either need to simulate a representative set of user histories (laden with assumptions) or capture real users recommendations (difficult to get a large representative sample of users to install an extension). Fortunately creators can export their channels actual incoming recommendations, and if we can get a large enough same of those we can give an estimate of the accuracy. It is also reasonable to assume personalization will mostly "average out" when you total all users recommendations. A more detailed explanation defending this assumption can be found here.
• Only recommendation's since 1st Nov 2019 is currently included in the analysis. We may provide an option to compare these historically in the future but the most relevant recommendation information is recent.
• By only looking at political recommendation, channels who are recommended outside politics might seem like they are being disadvantaged?
  This indeed a limitation, you should consider the analysis of YouTube political influence only for those watching political/cultural channels and might not be representative of all types of videos.
• Recommendations are not the only way viewers can be radicalized or influenced. Why are you so focused just on recommendations which might only have a small impact compared to the content itself and other psychological and social dynamics?
  That is very difficult to study. Our simplistic quantification of recommendation influence is something to ground that other research with, and reasonable to hold YouTube to account for.

Process & Classification Q & A

• The classification of political category is prone to bias subjectivity, how can you trust the results?
  
  This is indeed a limitation, but one we have tried to address

  • Clear guidelines were given to 3+ reviewers who independently classified each channel
  • We measured the consistency between the reviewers judgments. Left/Center/Right classifications are moderately reliable, and tags are varius degrees of reliability. Take this into consideration
  • Respected sources of classification for left/right were used when possible (i.e. adfontesmedia.com and mediabiasfactcheck.com) which covered a large portion of the large mainstream channels.
  • The top 50 channels by video views make up the majority of all views. Download the channel data, then Go though the top 50 channels and check if you agree with the classification. If you generally accept the classifications then you can generally accept the results.

• Why not perform a more procedural/quantifiable method for determining political category (e.g. like the process used for https://www.adfontesmedia.com)?
  I don't believe making this process more detailed and quantitative would give you a significantly more accurate/objective answer. We have performed some topic modelling, but the categories are less relevant to the public discussion about YouTube recommendations.

• Why is the apposition to Identity Politics/Social Justice considered "right" when it is not normally considered an important part of the standard political definition for left/right?
  
  I understand this, but I am confident this has changed. It is clear when evaluating YouTube content that this is a new and important divide.

• The left/right dichotomy is not a good way to classify tribal politics, why do it that way?
  
  I agree, and forcing it into this model creates many needless problems because it is not a natural category for this data. We have now provided a way to view channels by ideology.

Updates

25 Nov 2019: Now hosted at recfluence.net. Large new analysis and classification of channels.

14 Jan 2019: New channels and some updates. The network diagram looks quite different at first, but that's mostly cosmetic. The rotation and location of them are somewhat different but it clusters similarly. To see the older version matching the published article use this link