Understanding the X Algorithm

Objective: find 1500 tweets that the user might be interested in.

A Quick 5-min Tour: Demystifying the Complex in a Few Bites

Scenario: Alex is a user on X. He follows a handful of users and is, in turn, followed by a few other users on X. He interacts with content by liking the posts, reposting them, replying and bookmarking his favourites. Alex also comes across profiles & content that he doesn’t like or even finds annoying and misleading. He mutes them, unfollows some, clicks the “show fewer from” option, or even blocks them. Problem Statement: Alex goes to the “For you” tab and needs to be shown 1500 posts that he might be interested in.

The Process:

1. Likes, reports, quotes, followings, bookmarks, post clicks, time spent on each post, profile clicks, and more information like Alex’s geolocation, the trending topics in that location, etc. It also understands the content he does not like - collectively, this information is known as ‘Data’.

2. X assigns a different weight to each of these data. For instance, when Alex likes a post from one user (say Bella) and reposts a post from another user (say Charlie), X understands that Alex has more affinity towards Bella’s posts than Charlie’s. This happens on a large scale with Alex and every user and every post that he can be associated with. A graphical representation of the same is made. Such kind of a relationship is called ‘Feature Formation’.

3. X gathers 1,500 posts from the features so formed. In the next step, X tries assigning a rank to each one of them. The data comes into the picture again. Based on Alex’s interaction so far on X and based on the current popularity of each post in the 1,500 posts, X determines the probability of Alex - liking it, commenting on it, reposting it, etc. Here again, each probability is assigned a different weight. For instance, if Bella makes two posts and Alex is more likely (probability) to like one and comment on the other, the later post will be shown first. This process is known as ‘Ranking’.

4. Now that the posts are ranked, they undergo refinement. For instance, Alex will be bored if he sees 10 posts of Bella, one below the other, when he follows 100 other users. Alex should also not see content that is legally banned in his country. There should also be a good mix of posts from accounts that Alex follows and ones he doesn’t at present. This refinement happens in the Filtering, Heuristics, and Product Features stage.

5. The final step is to mix these posts with Ads (that’s obviously how X makes money) and follow-recomendations. This is the mixing stage.

Objective: How can you, as a user hack these two stages in order to maximize reach?

Feature Formation:

Based on Alex's {favorites, reposts, quotes, followings, bookmarks, clicks, unfollows, spam reports, blocks, etc., } ------> Alex's relationship with other [users] and [posts] on X is formed (as a graphical representation)

[ >1 = advantage and <1 = disadvantage]

1. If you are Blue verified and are followed (by Alex), = 4x higher than a random unverified post author on X

2. If you are Blue verified and are not followed (by Alex) = still get a 2x higher than a random unverified post author on X.

3. If Alex favourites your post (Likes + Bookmarks) = 30x

4. If Alex reposts your post= 20x

5. If Alex follows you = 4x

6. If your posts carry an image & audio = 2x

7. If your post is in line with current trend = 1.1x

8. If you’ve used an unknown language/word in your post = (0.005) deboosted by 20x

9. If your post contains offensive words = (0.1) deboosted by 10x

10. If your post has multiple hashtags = (0.6) deboosted by 1.1x

11. If Alex has unfollowed you recently, muted, blocked, or reported you as spam = NO relationship is formed

12. If your post is NSFW, abusive, or toxic = NO relationship is formed

Ranking:

For each of the 1,500 posts, based on its current {favorites, reposts, quotes, bookmarks, clicks, time spent on them, links, image & video, and even correctness of spellings} ------> 10 probabilities (for eg: the probability that Alex will repost it; the probability that Alex will comment on it; the probability that the post author will reply to Alex's comment, etc) are formed with each having a value between 0 & 1.

[ +ve value = advantage and -ve value = disadvantage]

1. The probability that Alex will favourite your Post = 0.5

2. The probability that Alex will Retweet your Post = 1

3. The probability that Alex replies to your Post = 13.5

4. The probability that Alex opens your profile and Likes or Replies to a Post = 12

5. The probability (for a video Post) that Alex will watch at least half of the video = 0.005

6. The probability that Alex replies to the Post and this reply is engaged by you = 75

7. The probability that Alex will click into the conversation of this Post and reply or Like a Post = 11

8. The probability that Alex will click into the conversation of your Post and stay there for at least 2 minutes = 10

9. The probability that Alex will react negatively (requesting "show less often" on your Post or profile, block or mute you) = -74

10. The probability that Alex will click Report your Post = -369

(0.5 * 0.5) + (1 * 0.5) + (13.5 * 0.5) + (12 * 0.5) + (0.005 * 0.5) + (75 * 0.5) + (11 * 0.5) + (10 * 0.5) + (-74 * 0.001) + (-369 * 0.001) = 61.0595

Nerdy Nirvana: A Deep-Dive Into The Algorithm

Tools that X uses:

Problem Statement: Alex & Bella are two users. They might be friends, relatives, neighbours, or just two strangers from different continents. How will X determine what their relationship is? And what is the relationship between them and their tweets? Further, what is their relationship with other users and posts on X?

• Features: they act as inputs for all of X’s algorithm [they can be likes, reposts, replies, clicks, mutes, blocks, user data, etc.]

• Packages: These bots/programs take features as input and create a graph → cluster them → pick top posts from it → rank them → mix them with ads and serve the For you feed.

• Graphing: A relationship graph is created among various nodes (entities) within X. A sub-part of the process, graphs are the outcome of the machine-learning packages on X and it is based on this the 1,500 posts are picked.

A) Features:

B) Packages:

c) Graphing:

The Process:

1. Candidate Sourcing:

2. Ranking:

The ranking mechanism takes into account thousands of features and outputs ten labels to give each Tweet a score, where each label represents the probability of an engagement

score = sum_i { (weight of engagement i) * (probability of engagement i) }

1. The post itself, and

2. The user that the post is being recommended to

• Computing Probability:

1. The probability the user will favorite the Post.

2. The probability the user will Retweet the Post.

3. The probability the user replies to the Post.

4. The probability the user opens the Post author profile and Likes or replies to a Post.

5. The probability (for a video Post) that the user will watch at least half of the video.

6. The probability the user replies to the Post and this reply is engaged by the Post author.

7. The probability the user will click into the conversation of this Post and reply or Like a Post.

8. The probability the user will click into the conversation of this Post and stay there for at least 2 minutes.

9. The probability the user will react negatively (requesting "show less often" on the Post or author, block or mute the Post author).

10. The probability the user will click Report Post.

• Weights: For each of the above labels, there is a pre-defined weight. This can be found in the Configuration File:

3. Filters, Heuristics & Product Features:

• Visibility Filtering: Hides posts from blocked or muted accounts. The visibility-filters package does this. It also filters out posts from accounts held back by legal agencies, suspended accounts, etc.

• Author Diversity: Prevent seeing too many posts from the same person in a row.

• Content Balance: Show a mix of posts from in-network and out-of-network

• Feedback-based Fatigue: Reduces visibility of posts that have received negative feedback from the user.

• Social Proof: Ensures out-of-network posts are shown only if some of the accounts followed by a user have interacted with them.

• Conversations: Provide more context to a Reply by threading it together with the original post.

• Edited Tweets: Keeps the feed fresh by replacing outdated tweets with their updated versions.

4. Mixing & Serving:

Understanding Reach:

1. At the Candidate Sourcing Level

Objective: make your nodes connect with as many nodes as possible and increase the relationship strength (edge weights - 0.56 in the graphing example).

private def getLinearRankingParams: ThriftRankingParams = {
    ThriftRankingParams(
      `type` = Some(ThriftScoringFunctionType.Linear),
      minScore = -1.0e100,
      retweetCountParams = Some(ThriftLinearFeatureRankingParams(weight = 20.0)),
      replyCountParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)),
      reputationParams = Some(ThriftLinearFeatureRankingParams(weight = 0.2)),
      luceneScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)),
      textScoreParams = Some(ThriftLinearFeatureRankingParams(weight = 0.18)),
      urlParams = Some(ThriftLinearFeatureRankingParams(weight = 2.0)),
      isReplyParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)),
      favCountParams = Some(ThriftLinearFeatureRankingParams(weight = 30.0)),
      langEnglishUIBoost = 0.5,
      langEnglishTweetBoost = 0.2,
      langDefaultBoost = 0.02,
      unknownLanguageBoost = 0.05,
      offensiveBoost = 0.1,
      inTrustedCircleBoost = 3.0,
      multipleHashtagsOrTrendsBoost = 0.6,
      inDirectFollowBoost = 4.0,
      tweetHasTrendBoost = 1.1,
      selfTweetBoost = 2.0,
      tweetHasImageUrlBoost = 2.0,
      tweetHasVideoUrlBoost = 2.0,
      useUserLanguageInfo = true,
      ageDecayParams = Some(ThriftAgeDecayRankingParams(slope = 0.005, base = 1.0))

• Follower Graph: this is simpler, i.e., who follows you

• Tweet Engagement: this is the data about your posts.

• User Data: this is the data about you as a user.

val allEdgeFeatures: SCollection[Edge] =
      getEdgeFeature(SCollection.unionAll(
					Seq(blocks, mutes, abuseReports, spamReports, unfollows)))

object BlueVerifiedAuthorInNetworkMultiplierParam
extends FSBoundedParam [Double] (
name = “home_mixer_blue_verified_author_in_network_multiplier",
default = 4.0,
min = 0.6,
max = 100.0
)
object BlueVerifiedAuthorOutOfNetworkMultiplierParam
extends FSBoundedParam [Double] (
name = “home_mixer_blue_verified_author_out_of_network_multiplier",
default = 2.0,
min = 0.6,
max = 100.0
)

• Escaping Trust & Safety Filters:

Mr. Musk empowered my team to move more aggressively to remove hate speech across the platform — censoring more content, not less. Source: DailyMail

pNSFWMedia: Model to detect tweets with NSFW images. This includes adult and porn content.
pNSFWText: Model to detect tweets with NSFW text, adult/sexual topics.
pToxicity: Model to detect toxic tweets. Toxicity includes marginal content like insults and certain types of harassment. Toxic content does not violate Twitter's terms of service.
pAbuse: Model to detect abusive content. This includes violations of Twitter's terms of service, including hate speech, targeted harassment and abusive behavior.

2. At the Ranking Level

scored_tweets_model_weight_fav: 0.5
scored_tweets_model_weight_retweet: 1.0
scored_tweets_model_weight_reply: 13.5
scored_tweets_model_weight_good_profile_click: 12.0
scored_tweets_model_weight_video_playback50: 0.005
scored_tweets_model_weight_reply_engaged_by_author: 75.0
scored_tweets_model_weight_good_click: 11.0
scored_tweets_model_weight_good_click_v2: 10.0
scored_tweets_model_weight_negative_feedback_v2: -74.0
scored_tweets_model_weight_report: -369.0

3. At the Filtering, Heuristics, and Product Features Level

struct ThriftAgeDecayRankingParams {
// the rate in which the score of older tweets decreases
1: optional double slope = 6.003
// the age, in minutes, where the age score of a tweet is half of the latest tweet
2: optional double halflife = 360.0
// the minimal age decay score a tweet will have
3: optional double base = 0.6
}(persisted='true')

At the HomeMixer Level [Optional]

Actionable Intelligence

No-brainers

Getting Blue verified gives you an immediate boost of

1. 4x to your followers

object BlueVerifiedAuthorInNetworkMultiplierParam
extends FSBoundedParam [Double] (
name = “home_mixer_blue_verified_author_in_network_multiplier",
default = 4.0,
min = 0.6,
max = 100.0
)

1. and 2x across X

object BlueVerifiedAuthorOutOfNetworkMultiplierParam
extends FSBoundedParam [Double] (
name = “home_mixer_blue_verified_author_out_of_network_multiplier",
default = 2.0,
min = 0.6,
max = 100.0
)

Avoid using multiple hashtags.

1. You get immediately deboosted by 1.7x (i.e., by a factor of 0.6)

multipleHashtagsOrTrendsBoost = 0.6,

Avoid posting content that is NSFW (both media and text), abusive, or possibly misinformation (content that you are not sure of)

pNSFWMedia: Model to detect tweets with NSFW images. This includes adult and porn content.
pNSFWText: Model to detect tweets with NSFW text, adult/sexual topics.
pToxicity: Model to detect toxic tweets. Toxicity includes marginal content like insults and certain types of harassment. Toxic content does not violate Twitter's terms of service.
pAbuse: Model to detect abusive content. This includes violations of Twitter's terms of service, including hate speech, targeted harassment and abusive behavior.

offensiveBoost = 0.1,

Avoid posting content on sensitive topics.

1. The exact keywords and topics in the filters are dynamic and keep changing over time. Until recently, posts on Ukraine were de-boosted. Check the GitHub page of the code to keep yourself updated on what X considers sensitive.

Stick to posting on one/a few topics. This way you develop a strong relationship with a cluster

1. The SimCluster package ensures that you have a strong relationship among members in a cluster. Source: SimCuster

2. Your post also gets a 3x boost to users in your Trusted Circle

inTrustedCircleBoost = 3.0,

1. Replying to content that is out-of-network also carries a penalty of 10. Unless you are sure that you can offset this by getting high engagement on such replies, try avoiding it

// subtractive penalty applied after boosts for out-of-network replies.
120: optional double OutOfNetworkReplyPenalty = 10.0

Quick-Wins

1. Try to use captive and relevant images and videos in your posts. They boost your post by 2x.

tweetHasImageUrlBoost = 2.0,
tweetHasVideoUrlBoost = 2.0,

1. Avoid making spelling errors or posting in unknown languages (you can try representing them through images instead). They get deboosted by 20x (i.e., by a factor of 0.05)

unknownLanguageBoost = 0.05,

1. Write long-form content or threads where the user might spend more than 2 minutes reading it

"recap.engagement.is_good_clicked_convo_desc_v2": The probability the user will click into the conversation of this Tweet and stay there for at least 2 minutes.
scored_tweets_model_weight_good_click_v2: 10.0

1. Reply to people who comment on your posts.

"recap.engagement.is_replied_reply_engaged_by_author": The probability the user replies to the Tweet and this reply is engaged by the Tweet author.
scored_tweets_model_weight_reply_engaged_by_author: 75.0

1. Sometimes, even having a captivating profile picture that might kindle a user to click your profile might help. If the user does so, ensure that they like your posts by pinning your top posts and by making sure your highlights tab is updated with your most interesting posts.

"recap.engagement.is_profile_clicked_and_profile_engaged": The probability the user opens the Tweet author profile and Likes or replies to a Tweet.
scored_tweets_model_weight_good_profile_click: 12.0

Challenging Choices

1. Writing good content: ultimately everything boils down to this. It is the central crux that holds all the other cards together. Every like, repost, comment, and bookmark that your posts get has a major boost (both at the Feature Formation stage and the Ranking stage).

retweetCountParams = Some(ThriftLinearFeatureRankingParams(weight = 20.0)),
replyCountParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)),
isReplyParams = Some(ThriftLinearFeatureRankingParams(weight = 1.0)),
favCountParams = Some(ThriftLinearFeatureRankingParams(weight = 30.0)),

"recap.engagement.is_favorited": The probability the user will favorite the Tweet.
"recap.engagement.is_favorited": 0.5
"recap.engagement.is_good_clicked_convo_desc_favorited_or_replied": The probability the user will click into the conversation of this Tweet and reply or Like a Tweet.
"recap.engagement.is_good_clicked_convo_desc_favorited_or_replied": 11*
"recap.engagement.is_replied": The probability the user replies to the Tweet.
"recap.engagement.is_replied": 27
"recap.engagement.is_retweeted": The probability the user will ReTweet the Tweet.
"recap.engagement.is_retweeted": 1

//parameters used by Representation-scorer
1: optional double fav1dLast10Max // max score from last 10 faves in the last 1 day
2: optional double fav1dLast10Avg // avg score from last 10 faves in the last 1 day
3: optional double fav7dLast10Max // max score from last 10 faves in the last 7 days
4: optional double fav7dLast10Avg // avg score from last 10 faves in the last 7 days
5: optional double retweet1dLast10Max // max score from last 10 retweets in the last 1 days
6: optional double retweet1dLast10Avg // avg score from last 10 retweets in the last 1 days
7: optional double retweet7dLast10Max // max score from last 10 retweets in the last 7 days
8: optional double retweet7dLast10Avg // avg score from last 10 retweets in the last 7 days
9: optional double follow7dLast10Max // max score from the last 10 follows in the last 7 days
10: optional double follow7dLast10Avg // avg score from the last 10 follows in the last 7 days
11: optional double follow30dLast10Max // max score from the last 10 follows in the last 30 days
12: optional double follow30dLast10Avg // avg score from the last 10 follows in the last 30 days
13: optional double share1dLast10Max // max score from last 10 shares in the last 1 day
14: optional double share1dLast10Avg // avg score from last 10 shares in the last 1 day
15: optional double share7dLast10Max // max score from last 10 shares in the last 7 days
16: optional double share7dLast10Avg // avg score from last 10 shares in the last 7 days
17: optional double reply1dLast10Max // max score from last 10 replies in the last 1 day
18: optional double reply1dLast10Avg // avg score from last 10 replies in the last 1 day
19: optional double reply7dLast10Max // max score from last 10 replies in the last 7 days
20: optional double reply7dLast10Avg // avg score from last 10 replies in the last 7 days

1. Write on topics that are in trend ( but limit the use of hashtags). Apart from the hashtags, also use keywords that are trending. Your content gets boosted by a factor of 1.1x

tweetHasTrendBoost = 1.1

1. Tweet at a time when your target audience is likely to be active on X. The half-life of content on X is 5 hours. This means that every 6 hours, the chance of your post being recommended to others is reduced by half. Experiment and arrive at an optimum time through trial & error.

struct ThriftAgeDecayRankingParams {
// the rate in which the score of older tweets decreases
1: optional double slope = 6.003
// the age, in minutes, where the age score of a tweet is half of the latest tweet
2: optional double halflife = 360.0
// the minimal age decay score a tweet will have
3: optional double base = 0.6
}(persisted='true')

Avoiding Pitfalls

val allEdgeFeatures: SCollection[Edge] =
      getEdgeFeature(SCollection.unionAll(
					Seq(blocks, mutes, abuseReports, spamReports, unfollows)))

val allEdgeFeatures: SCollection[Edge] =
      getEdgeFeature(SCollection.unionAll(Seq(blocks, mutes, abuseReports, spamReports, unfollows)))

    val negativeFeatures: SCollection[KeyVal[Long, UserSession]] =
      allEdgeFeatures
        .keyBy(_.sourceId)
        .topByKey(maxDestinationIds)(Ordering.by(_.features.size))
        .map {
          case (srcId, pqEdges) =>
            val topKNeg =
              pqEdges.toSeq.flatMap(toRealGraphEdgeFeatures(hasNegativeFeatures))
            KeyVal(
              srcId,
              UserSession(
                userId = Some(srcId),
                realGraphFeaturesTest =
                  Some(RealGraphFeaturesTest.V1(RealGraphFeaturesV1(topKNeg)))))

//parameters from representation scorer
// 2001 - 3000 Negative Signals
  // Block Series
  2001: optional double block1dLast10Avg
  2002: optional double block1dLast10Max
  2003: optional double block7dLast10Avg
  2004: optional double block7dLast10Max
  2005: optional double block30dLast10Avg
  2006: optional double block30dLast10Max
  // Mute Series
  2101: optional double mute1dLast10Avg
  2102: optional double mute1dLast10Max
  2103: optional double mute7dLast10Avg
  2104: optional double mute7dLast10Max
  2105: optional double mute30dLast10Avg
  2106: optional double mute30dLast10Max
  // Report Series
  2201: optional double report1dLast10Avg
  2202: optional double report1dLast10Max
  2203: optional double report7dLast10Avg
  2204: optional double report7dLast10Max
  2205: optional double report30dLast10Avg
  2206: optional double report30dLast10Max
  // Dontlike
  2301: optional double dontlike1dLast10Avg
  2302: optional double dontlike1dLast10Max
  2303: optional double dontlike7dLast10Avg
  2304: optional double dontlike7dLast10Max
  2305: optional double dontlike30dLast10Avg
  2306: optional double dontlike30dLast10Max
  // SeeFewer
  2401: optional double seeFewer1dLast10Avg
  2402: optional double seeFewer1dLast10Max
  2403: optional double seeFewer7dLast10Avg
  2404: optional double seeFewer7dLast10Max
  2405: optional double seeFewer30dLast10Avg
  2406: optional double seeFewer30dLast10Max