Scaling Recommender Transformers to a Billion Parameters

! My identify is Kirill Khrylchenko, and I lead the RecSys R&D workforce at Yandex. One among our targets is to develop transformer applied sciences throughout the context of recommender programs, an goal we’ve been pursuing for 5 years now. Not too way back, we reached a brand new milestone within the growth of advice applied sciences, which I want to share with you on this article.

The relevance of recommender programs on the planet is simple to justify: the quantity of content material is rising extremely quick, making it not possible to view in its entirety, and we want recommender programs to handle the knowledge overload. Music, films, books, merchandise, movies, posts, associates, however it’s essential to keep in mind that these companies profit not solely customers but in addition content material creators who want to seek out their audience.

We’ve deployed a brand new technology of transformer recommenders in a number of companies and are actively integrating them with different companies. We’ve considerably improved the standard of the suggestions throughout the board.

If you happen to’re an ML engineer working with suggestions, this text will give you some concepts on easy methods to implement an analogous strategy to your recommender system. And if you’re a consumer, you may have a chance to study extra about how that very recommender system works.

How Recommender Techniques Work

The advice downside itself has a easy mathematical definition: for every consumer

we wish to choose objects, objects, paperwork, or merchandise

that they’re prone to like.

However there’s a catch:

• Merchandise catalogs are huge (as much as billions of things).
• There’s a important variety of customers, and their pursuits are continuously shifting.
• Interactions between customers and objects are very sparse.
• It’s unclear easy methods to outline precise consumer preferences.

To deal with the advice downside successfully, we have to leverage non-trivial fashions that use machine studying.

Neural networks are a potent machine studying software, particularly when there’s a considerable amount of unstructured information, comparable to textual content or pictures. Whereas conventional classical machine studying includes skilled area information and appreciable handbook work (function engineering), neural networks can extract advanced relationships and patterns from uncooked information virtually mechanically.

Within the RecSys area, we’ve a considerable amount of largely unstructured information (actually trillions of anonymized user-item interactions), in addition to entities which can be content-based (objects include titles, descriptions, pictures, movies, and audio; customers could be represented as sequences of occasions). Moreover, it’s essential that the recommender system performs properly for brand spanking new objects and chilly customers, and encoding customers and objects by way of content material helps obtain this.

The time we’ve to generate suggestions for the consumer could be very strictly restricted. Each millisecond counts! Moreover, we don’t have infinite sources (by way of {hardware}), and the catalogs we want suggestions from are fairly giant. For this reason suggestions are normally shaped in a number of phases:

• First, we choose a comparatively small set of candidates from your complete catalog utilizing light-weight fashions (retrieval stage).
• Then, we run these candidates by way of extra advanced fashions that make the most of extra info and extra intensive computations for every candidate (rating stage).

Architecturally, fashions fluctuate considerably between phases, making it difficult to debate any side with out referring to particular phases of the recommender system.

The 2-tower neural community structure could be very common for the retrieval stage. Customers and objects (for info retrieval, this may be queries and paperwork, independently encoded into vector representations) are used, and the dot product is employed to calculate the similarity between them.

You can additionally say that such fashions “embed” customers and objects right into a shared “semantic area”, the place the semantic side represents the truth that the nearer the user-item pair is by way of vector area, the extra related they’re.

Two-tower fashions are high-speed. Let’s assume the consumer requests suggestions. The 2-tower mannequin then must calculate:

• The “consumer tower” as soon as per request.
• Vectors of all candidate objects for which you wish to calculate user-item affinity.
• Dot merchandise.

You don’t even must recalculate the vectors of candidate objects for every consumer question, as a result of they’re the identical for all customers and infrequently change; for example, we don’t assume {that a} film or a music observe typically adjustments its title. In follow, we repeatedly recalculate merchandise vectors for your complete catalog offline (for instance, every day) and add them to both the service the place we have to calculate the dot product or to a different service that we entry on-line to retrieve the mandatory merchandise vectors.

However that’s me describing a use case the place you may have some affordable, small variety of candidates you wish to calculate user-item affinities for. That is true for the rating stage. Nevertheless, on the candidate technology stage, the issue turns into extra sophisticated: we have to calculate proximities for all objects within the catalog, choose the top-N (the place N is often expressed in a whole lot to 1000’s) with the very best affinity values, after which ahead them to the following phases.

That is the place two-tower fashions are invaluable: we are able to shortly generate an approximate top-N by scalar product, even for enormous catalogs, utilizing approximate search strategies. We construct a selected “index” (sometimes a graph construction, comparable to within the HNSW technique) for the set of already calculated merchandise vectors that we are able to retailer within the service and use to feed consumer vectors, extracting an approximate high for these vectors.

Constructing this index is troublesome and time-consuming (with a separate problem of shortly updating and rebuilding an index). With that being mentioned, it could nonetheless be completed offline, after which the binary and the index could be uploaded to the service, the place we’ll seek for candidates within the runtime atmosphere.

How Do We Encode a Consumer Right into a Vector?

Classical algorithms solved this downside fairly simply: in matrix factorization strategies (like ALS), the consumer vector was “trainable”, represented by the mannequin parameters, and decided throughout the optimization process. In user-item collaborative filtering strategies, a consumer was assigned a vector of catalog dimensionality by which the i-th coordinate corresponded to a specific merchandise and represented how typically the consumer interacted with that merchandise (e.g., how continuously they purchased it or how they rated it).

The trendy strategy could be to encode customers with transformers, suggesting {that a} consumer could be encoded right into a vector utilizing transformers. We take the consumer’s anonymized historical past—that’s, a sequence of occasions—and encode these occasions into vectors, then make the most of a transformer. In probably the most primary case, occasions are represented by purchases or likes; nevertheless, in different circumstances, it may very well be your complete historical past of interactions inside an organization’s ecosystem.

Initially, when transformers have been first utilized in suggestions, researchers drew analogies from similarities with NLP: a consumer is sort of a sentence, and the phrases in it symbolize purchases, likes, and different interactions.

One other kind of neural community recommender mannequin is fashions with early fusion. These fashions don’t separate consumer and merchandise info into two towers however slightly course of all info collectively. That’s, we fuse all details about the consumer, the merchandise, and their interplay at an early stage. In distinction, two-tower fashions are mentioned to function late fusion by way of the scalar product. Early-fusion fashions are extra expressive than two-tower fashions. They will seize extra advanced alerts and study extra non-trivial dependencies.

Nevertheless, it’s troublesome to use them outdoors the rating stage due to their computational burden and the necessity to recalculate your complete mannequin for every consumer question and every candidate. In contrast to two-tower fashions, they don’t help the factorization of computations.

We make the most of varied structure sorts, together with two-tower fashions with transformers and fashions with early fusion. We use two-tower architectures extra actually because they’re extremely environment friendly, appropriate for all phases concurrently, and nonetheless yield good high quality beneficial properties with significantly fewer sources.

We used to coach two-tower fashions in two phases:

• Pre-training with contrastive studying. We practice the mannequin to align customers with their constructive user-item interactions utilizing contrastive studying,
• Process-specific fine-tuning. As with NLP, fine-tuning is a task-specific strategy. If the mannequin can be used for rating, we practice it to precisely rank the suggestions proven to the consumer. We confirmed two objects—the consumer appreciated one, disliked the opposite—we wish to rank objects in the identical order. With retrieval, the duty resembles pre-training however employs extra strategies that improve the candidates’ recall.

Within the subsequent part, we’ll discover how this course of has modified with our newer fashions.

Scaling Recommender Techniques

Is there a restrict to the scale of recommender fashions, after which we now not see size-related enhancements within the high quality of suggestions?

For a very long time, our recommender fashions (and never simply ours, however fashions throughout business and academia) have been very small, which urged that the reply to this query was “sure”.

Nevertheless, in deep studying, there’s the scaling speculation, which states that as fashions turn into bigger and the quantity of knowledge will increase, the mannequin high quality ought to enhance considerably.

A lot of the progress in deep studying over the previous decade could be attributed to this speculation. Even the earliest successes in deep studying have been based mostly on scaling, with the emergence of an in depth dataset for picture classification, ImageNet, and the nice efficiency of neural networks (AlexNet) on that dataset.

The scaling speculation is much more evident in language fashions and pure language processing (NLP): you’ll be able to predict the dependence of high quality enchancment on the quantity of computations and specific the corresponding scaling legal guidelines.

What do I imply after I say recommender fashions could be made greater?

There are as many as 4 totally different axes to scale.

Embeddings. We’ve got a wide range of details about customers and objects, so we’ve entry to a variety of options, and a big portion of those options are categorical. An instance of a categorical function is Merchandise ID, artist ID, style, or language.

Categorical options have a really excessive cardinality (variety of distinctive values)—reaching billions—so if you happen to make giant trainable embeddings (vector representations) for them, you get large embedding matrices.

That mentioned, embeddings are the bottleneck between the enter information and the mannequin, so you should make them giant for good high quality. For instance, Meta* has embedding matrices with dimensions starting from 675 billion to 13 trillion parameters, whereas Google reported at the least 1 billion parameters in YoutubeDNN again in 2016. Even Pinterest, which had lengthy promoted inductive graph embeddings from PinSage [1, 2], has just lately began utilizing giant embedding matrices.

Context size. For many years, recommender system engineers have been busy producing options. In fashionable rating programs, the variety of options can attain a whole lot and even 1000’s, and Yandex additionally gives such companies.

One other instance of “context” in a mannequin is the consumer’s historical past in a transformer. Right here, the scale of the context is set by the size of the historical past. In each business and academia, the quantity tends to be very small, with just a few hundred occasions at greatest.

Coaching dataset measurement. I already talked about that we’ve lots of information. Recommender programs produce a whole lot of datasets which can be related in measurement to the GPT-3 coaching dataset.

The business has a number of use circumstances of large datasets with billions of coaching examples on show: 2 billion, 2.1 billion, 3 billion, 60 billion, 100 billion, 146 billion, 500 billion.

Encoder measurement. The usual for early-fusion fashions can be in tens of millions or tens of tens of millions of parameters. Based on the Google papers, “simplified” variations of their Huge&Deep fashions had 1 to 68 million parameters for the experiments [1, 2]. And if we use a two-layer DCN-v2 (a preferred neural community layer for early-fusion fashions) over a thousand steady options, we’ll get not more than 10 million parameters.

Two-tower fashions most frequently use tiny transformers to encode the consumer: for instance, two transformer blocks with hidden layer dimensionality not exceeding a few hundred. This configuration may have at most just a few million parameters.

And whereas the sizes of the embedding matrices and coaching datasets are already fairly giant, scaling the size of consumer historical past and the capability of the encoder a part of the mannequin stays an open query. Is there any important scaling by these parameters or not?

This was the query on our minds in February, 2024. Then an article from researchers at Meta, titled Actions Converse Louder than Phrases, cheered us all up a bit.

The аuthors offered a brand new encoder structure known as HSTU and formulated each the rating downside and the candidate technology downside as a generative mannequin. The mannequin had a really lengthy historical past size (8000 occasions!) together with an in depth coaching dataset (100 billion examples), and the consumer historical past encoder was a lot bigger than the previous couple of million parameters. Nevertheless, even right here, the biggest encoder configuration talked about, has solely 176 million parameters, and it’s unclear whether or not they applied it (judging by the following articles, they didn’t).

Are 176 million parameters in an encoder so much or somewhat? If we take a look at language fashions, the reply is evident: an LLM with 176 million parameters within the encoder can be extremely inferior in functionality and problem-solving high quality to fashionable SOTA fashions with billions and even trillions of parameters.

Why, then, do we’ve such small fashions in recommender programs?

Why can’t we obtain an analogous leap in high quality if we change pure language texts with anonymized consumer histories by which actions act as phrases? Have recommender fashions already reached the ceiling of their baseline high quality, and all we’ve left is to make small incremental enhancements, tweaking options and goal values.

These have been the existential questions we requested ourselves when designing our personal new ARGUS strategy.

RecSys × LLM × RL

After plowing by way of the intensive literature on scaling, we discovered that three major circumstances decide the success of neural community scaling:

• Plenty of information.
• Fairly expressive structure with a big mannequin capability.
• Probably the most basic, basic studying process potential.

For instance, LLMs are very expressive and highly effective transformers that study from actually all the info on the web. Moreover, the duty of predicting the subsequent phrase is a basic process that, in actuality, decomposes into varied duties associated to totally different fields, together with grammar, erudition, arithmetic, physics, and programming. All three circumstances are met!

If we take a look at recommender programs:

• We even have lots of information: trillions of interactions between customers and objects.
• We will simply as simply use transformers.
• We simply want to seek out the best studying process to scale the recommender mannequin.

That’s what we did.

There’s an attention-grabbing side of pre-training giant language fashions. If you happen to simply ask a pre-trained LLM about one thing, it is going to give a mean reply. The almost definitely reply it has encountered within the coaching information. That reply gained’t essentially be good or proper.

However if you happen to add a immediate earlier than the query, like “Think about you might be an skilled in X”, it is going to begin offering way more related and proper solutions.

That’s as a result of LLMs don’t simply study to mimic solutions from the web; additionally they purchase a extra basic understanding of the world in an try and condense all the knowledge from the coaching set. It learns patterns and abstractions. And it’s exactly as a result of the LLM is aware of a variety of solutions and but possesses a basic understanding of the world that we are able to receive good solutions from it.

We tried to use this logic to recommender programs. First, you should specific the suggestions as a reinforcement studying process:

• A recommender system is an agent.
• Actions are suggestions. In probably the most primary case, the recommender system recommends one merchandise at a time (for instance, recommends one music observe within the music streaming app every time).
• The atmosphere means customers, their behaviors, patterns, preferences, and pursuits.
• The coverage is a chance distribution over objects.
• The reward is a consumer’s constructive suggestions in response to a suggestion.

There’s a direct analogy to the LLM instance. “Solutions from the web” are the actions of previous recommender programs (logging insurance policies), and basic information in regards to the world is knowing customers, their patterns, and preferences. We wish our new mannequin to have the ability to:

• Imitate the actions of previous recommender programs.
• Have a very good understanding of the customers.
• Alter their actions to attain a greater consequence.

Earlier than we transfer on to our new strategy, let’s look at the preferred setup for coaching suggestion transformers: subsequent—merchandise prediction. The SASRec mannequin could be very consultant right here. The system accumulates a consumer’s historical past of constructive interactions with the service (for instance, purchases), and the mannequin learns to foretell which buy is prone to come subsequent within the sequence. That’s, as an alternative of next-token prediction, as in NLP, we go for next-item prediction.

This strategy (SASRec and customary subsequent merchandise prediction) just isn’t per the philosophy I described earlier, which targeted on adjusting the logging coverage based mostly on basic information of the world. It will appear that to foretell what the consumer will purchase subsequent, the mannequin ought to function underneath this philosophy:

• It ought to perceive what may very well be proven to the consumer by the recommender system that was in manufacturing on the time for which the prediction ought to be made. That’s, it ought to have a very good mannequin of logging coverage habits (i.e., a mannequin that can be utilized to mimic).
• It wants to know what the consumer might need appreciated from the issues proven by the previous recommender system, which means that it wants to know their preferences, that are the very basic beliefs in regards to the world.

However fashions like SASRec don’t explicitly mannequin any of these items. They lack full details about previous logging insurance policies (we solely see suggestions with constructive outcomes), and we additionally don’t learn to replicate these logging insurance policies. There’s no method to know what the previous recommender system might have supplied. On the identical time, we don’t totally perceive the mannequin of the world or the consumer: we ignore all destructive suggestions and solely contemplate constructive suggestions.

ARGUS: AutoRegressive Generative Consumer Sequential Modeling

AutoRegressive Generative Consumer Sequential modeling (ARGUS) is our new strategy to coaching suggestion transformers.

First, we look at your complete anonymized consumer historical past, together with constructive interactions but in addition all different interactions. We seize the essence of the interplay context, the time it occurred, the machine used, the product web page the consumer was on, their My Vibe personalization settings, and different related particulars.

Consumer historical past is a selected sequence of triples (context, merchandise, suggestions), the place context refers back to the interplay context, merchandise represents the article the consumer interacts with, and suggestions denotes the consumer’s response to the interplay (comparable to whether or not the consumer appreciated the merchandise, purchased it, and so forth.).

Subsequent, we establish two new studying duties, each of which prolong past the standard next-item prediction extensively utilized in business and academia.

Subsequent merchandise prediction

Our first process can also be known as subsequent merchandise prediction. Trying on the historical past and the present interplay context, we predict which merchandise can be interacted with: P(merchandise | historical past, context).

• If the historical past comprises solely suggestion site visitors (occasions generated immediately by the recommender system), then the mannequin learns to mimic the logging coverage (suggestions from the previous recommender system).

• If there’s additionally natural site visitors (any site visitors aside from referral site visitors, comparable to site visitors from search engines like google, or if the consumer visits the library and listens to their favourite observe), we additionally acquire extra basic information in regards to the consumer, unrelated to the logging coverage.

Vital: although this process has the identical identify as in SASRec (subsequent merchandise prediction), it’s not the identical process in any respect. We predict not solely constructive but in addition destructive interactions, and in addition take note of the present context. The context helps us perceive whether or not the motion is natural or not, and if it’s a suggestion, what floor it’s on (place, web page, or carousel). Additionally, it usually reduces the noise degree throughout mannequin coaching.

Context is crucial for music suggestions: the consumer’s temper and their present state of affairs have a big influence on the kind of music they wish to hearken to.

The duty of predicting a component from a set is often expressed as a classification downside, the place the weather of the unique set function lessons. Then, we have to use a cross-entropy loss perform for coaching, the place the softmax perform is utilized to the logits (unnormalized outputs of the neural community). Softmax calculation requires computing the sum of exponents from logits throughout all lessons.

Whereas the scale of dictionaries in LLMs can attain a whole lot of 1000’s of things within the worst case, and softmax calculation just isn’t a big downside, it turns into a priority in recommender programs. Right here, catalogs include tens of millions and even billions of things, and calculating the complete softmax is an not possible process. It is a subject for a separate huge article, however ultimately, we’ve to make use of a difficult loss perform known as “sampled softmax” with a logQ correction:

• N means a mixture of in-batch and uniform negatives
• logQ(n)means logQ correction
• Temperature Tmeans a skilled parameter Eᵀclipped to [0.01, 100].

Suggestions prediction

Suggestions prediction is the second studying process. Contemplating historical past, the present context, and the merchandise, we predict consumer suggestions: P(suggestions | historical past, context, merchandise).

The primary process, subsequent merchandise prediction, teaches us easy methods to imitate logging insurance policies (and understanding customers if there’s natural site visitors). The suggestions prediction process, alternatively, is targeted completely, on getting basic information about customers, their preferences, and pursuits.

It is vitally just like how the rating variant of the mannequin from “Actions Converse Louder than Phrases” learns on a sequence of pairs (merchandise, motion). Nonetheless, right here the context token is handled individually, and there are extra than simply recommender contexts current.

Suggestions can have a number of parts: whether or not a observe was appreciated, disliked, added to a playlist, and what portion of the observe was listened to. We predict all varieties of suggestions by decomposing them into particular person loss capabilities. You should use any loss perform as a selected loss perform, together with cross-entropy or regression. For instance, binary cross-entropy is ample to foretell whether or not a like was current or not.

Though some suggestions is extra widespread (there are normally far fewer likes than lengthy listens), the mannequin does a very good job of studying to foretell all alerts. The bigger the mannequin, the better it’s to study all duties directly, with out conflicts. Furthermore, frequent suggestions (listens), quite the opposite, helps the mannequin learn to simulate uncommon, sparse suggestions (likes).

If we mix all this right into a single studying process, we get the next:

• Create histories for the consumer from triples (context, merchandise, suggestions).
• Use the transformer.
• Predict the subsequent merchandise based mostly on the hidden state of the context.
• Predict the consumer’s suggestions after interacting with the merchandise based mostly on the merchandise’s hidden state.

Let me additionally touch upon how this differs from HSTU. In Actions Converse Louder than Phrases, the authors practice two separate fashions for candidate technology and rating. The candidate technology mannequin comprises your complete historical past, however, like SASRec, it fashions solely constructive interactions and doesn’t contemplate the loss perform in circumstances the place there’s a destructive interplay. The rating mannequin, as talked about earlier, learns for a process just like our suggestions prediction.

Our answer gives a extra complete subsequent merchandise prediction process and a extra complete suggestions prediction process, and the mannequin learns in each capabilities concurrently.

Simplified ARGUS

Our strategy has one huge downside—we’re inflating the consumer’s historical past. As a result of every interplay with an merchandise is represented by three tokens directly (context, merchandise, suggestions), we must feed virtually 25,000 tokens into the transformer to research 8192 latest consumer listens.

One might argue that that is nonetheless not important and that the context size is for much longer in LLMs; nevertheless, this isn’t completely correct. LLMs, on common, have a lot smaller numbers, sometimes a whole lot of tokens, particularly throughout pre-training.

In distinction, in our music streaming platform, for instance, customers typically have 1000’s and even tens of 1000’s of occasions. We have already got for much longer context lengths, and inflating these lengths by an element of three has a fair higher influence on studying pace. To deal with this, we created a simplified model of the mannequin, by which every triple (context, merchandise, suggestions) is condensed right into a single vector. When it comes to enter format, it resembles our earlier generations of transformer fashions; nevertheless, we preserve the identical two studying duties—subsequent merchandise prediction and suggestions prediction.

To foretell the subsequent merchandise, we take the hidden state from the transformer akin to the triple (c, i, f) at a previous cut-off date, concatenate the present context vector to it, compress it to a decrease dimension utilizing an MLP, after which use the sampled softmax to study to foretell the subsequent merchandise.

To foretell the suggestions, we concatenate the vector of the present merchandise after which use an MLP to foretell all of the required goal variables. When it comes to recommender transformer architectures, our mannequin turns into much less target-aware and fewer context-aware; nevertheless, it nonetheless performs properly, enabling a three-fold acceleration.

ARGUS Implementation

A mannequin skilled on this two-headed mode for each duties concurrently (subsequent merchandise prediction and suggestions prediction) could be applied as is. The NIP head is chargeable for candidate choice, and the FP head for last rating.
However we didn’t wish to try this, at the least not for our first implementation:

• Our aim was to implement a really giant mannequin, so we initially targeted on offline deployment. With offline deployment, consumer and merchandise vectors are recalculated every day inside a separate common course of, and also you solely must calculate the dot product within the runtime atmosphere.

• The pre-trained model of ARGUS implies entry to the consumer’s historical past with none delay: we see all occasions of their historical past as much as the present cut-off date when the prediction is made. That’s, it must be utilized at runtime.
• The NIP head predicts all consumer interactions, and the mannequin is normally skilled to foretell solely future constructive interactions to generate candidates. However predicting constructive interactions is a heuristic, a surrogate studying process. It’d even be higher to make use of a head that predicts all interactions as a result of it learns to be per the rating. If an merchandise has been beneficial, it means the rating appreciated it. However on this state of affairs, we weren’t able to experiment with that and as an alternative needed to observe the well-trodden path.
• The FP head learns for pointwise losses: whether or not a observe can be appreciated or not, what portion of the observe can be heard, and so forth. However we nonetheless typically practice fashions for pairwise rating: we study to rank objects that have been beneficial “subsequent to one another” and acquired totally different suggestions. Some argue that pointwise losses are ample for coaching rating fashions, however on this case, we don’t change your complete rating stack. As an alternative, we goal so as to add a brand new, highly effective, neural-network-based function to the ultimate rating mannequin. If the ultimate rating mannequin is skilled for a specific process (comparable to pairwise rating), then the neural community that generates the function is most effectively skilled for that process; in any other case, the ultimate mannequin will rely much less on our function. Accordingly, we’d prefer to pre-train ARGUS for a similar process as the unique rating mannequin, permitting us to put it to use in rating.

There are different deployment use circumstances past the standard candidate technology and rating phases, and we’re actively researching these as properly. Nevertheless, for our first deployment, we went with an offline two-tower rating:

• We determined to fine-tune ARGUS in order that it may very well be used as an offline two-tower mannequin. We use it to recalculate consumer and merchandise vectors every day, whereas consumer preferences are decided by way of the dot product of the consumer with the objects.

• We pre-trained ARGUS for a pairwise rating process just like the one on which the ultimate rating mannequin is skilled. Because of this we’ve one way or the other chosen pairs of tracks that the consumer heard and rated in another way by way of constructive suggestions, and we wish to learn to rank them appropriately.

We construct these fashions very often: they’re straightforward to coach and implement by way of sources and growth prices. Nevertheless, our earlier fashions have been considerably smaller and discovered in another way. Not with the ARGUS process, however first with the standard contrastive studying between customers and positives, after which fine-tuned for the duty.

Our earlier contrastive pre-training process implied compiling a number of coaching examples for a consumer: if the consumer had n purchases, then there could be n samples within the dataset. That mentioned, we didn’t use autoregressive studying. That’s, we ran the transformer n instances throughout coaching. This strategy enabled us to be very versatile in creating pairs (consumer, merchandise) for coaching, use any historical past format, encode context along with the consumer, and account for lags. When predicting likes, we are able to use a one-day lag within the consumer’s historical past. Nevertheless, issues have been working fairly slowly.

ARGUS pre-training employs autoregressive studying, the place we study from all occasions within the consumer’s exercise concurrently in a single transformer run. It is a highly effective acceleration that allowed us to coach a lot bigger fashions utilizing the identical sources.

Throughout fine-tuning, we additionally ran the transformer many instances for a single consumer. It’s known as impression-level studying that Meta used to have earlier than HSTU. If a consumer is proven an merchandise at a selected second, we generate a pattern of the shape (consumer, merchandise). The dataset can comprise numerous such impressions for a single consumer, and we’ll rerun the transformer for every one in every of them. For pairwise rating, we thought-about triples of the shape (consumer, item1, item2). Those we used earlier than.

Inspecting the acceleration throughout the pre-training stage, we determined to make use of an analogous strategy with fine-tuning. We develop a fine-tuning process for the two-tower mannequin to show it rating, the place the transformer solely must be run as soon as.

Let’s say we’ve the consumer’s complete historical past for a yr, and all of the suggestions proven to the consumer throughout the identical interval. By implementing a transformer with a causal masks over your complete historical past, we get vector representations of the consumer for all of the moments in that yr directly, and so we are able to:

• Individually calculate the vectors of the proven objects.
• Evaluate the timestamps and map suggestion impressions to consumer vectors akin to the required lag in consumer historical past supply.
• Calculate all of the required scalar merchandise and all phrases of the loss perform.

And all of this directly for your complete yr—in a single transformer run.

Beforehand, we might rerun the transformer for every pair of impressions; now, we course of all of the impressions directly in a single run. It is a large acceleration: by an element of tens, a whole lot, and even 1000’s. To make use of a two-tower mannequin like this, we are able to merely use the vector illustration of the consumer on the final second in time (akin to the final occasion within the historical past) as the present vector illustration. For the objects, we are able to use the encoder that was used throughout coaching for the impressions. In coaching, we simulate a one-day consumer historical past lag after which run the mannequin as an offline mannequin, recalculating consumer vectors every day.

After I say that we course of the consumer’s complete yr of historical past in a single transformer run, I’m being considerably deceptive. In actuality, we’ve a sure restrict on the utmost historical past size that we implement, and a consumer in a dataset can have a number of samples or chunks. For pre-training, these chunks don’t overlap.

Nevertheless, throughout fine-tuning, there are limits not solely on the utmost historical past size but in addition on its minimal size, in addition to on the utmost variety of suggestion impressions in a single coaching instance used to coach the mannequin for rating.

Outcomes

We selected our music streaming as the primary service to experiment with. Suggestions are essential right here, and the service has numerous energetic customers. We’ve constructed an enormous coaching dataset with over 300 billion listens from tens of millions of customers. That is tens and even a whole lot of instances bigger than the coaching datasets we’d used earlier than.

What’s a triple (context, merchandise, suggestions) in a music streaming service?

• Context: whether or not the present interplay is a suggestion or natural. If it’s a suggestion—what floor it’s on, and if it’s My Vibe—what the settings are.
• Merchandise: a music observe. Crucial function for merchandise encoding is the merchandise ID. We use unified embeddings to encode options with excessive cardinality. On this case, we take three 512K hashes per merchandise. We use a set unified embedding matrix with 130 million parameters in our experiments.
• Consumer suggestions: whether or not a observe was appreciated, and what portion of the observe was heard.

For offline high quality evaluation, we use information from the week following the coaching interval by way of the worldwide temporal cut up.

To evaluate the standard of the pre-trained mannequin, we look at the loss perform values within the pre-training duties: subsequent merchandise prediction and suggestions prediction. That’s, we measure how properly the mannequin discovered to resolve the duties we created for it. The smaller the worth, the higher.

Vital: We contemplate the consumer’s historical past over an extended interval, however the loss perform is just calculated for occasions that happen throughout the check interval.

Throughout fine-tuning, we study to appropriately rank merchandise pairs based mostly on consumer suggestions, making PairAccuracy— a metric that measures the share of pairs appropriately ordered by the mannequin —an acceptable offline metric for us. In follow, we reweigh pairs barely extra based mostly on suggestions: for instance, pairs by which the individual appreciated and skipped a observe have a better weight than these by which the individual listened to and skipped a observe.

Our deployment situation includes including a robust new function to the ultimate ranker. Because of this, we measure the relative improve in PairAccuracy for the ultimate ranker with the brand new function added, in comparison with the ultimate ranker with out it. The ultimate ranker in our music streaming platform is gradient boosting.

A/B Check Outcomes and Measurements

Our preliminary aim was to scale suggestion transformers. To check the scaling, we chosen 4 different-sized transformer configurations, starting from 3.2 million to 1.007 billion parameters.

We additionally determined to check the efficiency of the HSTU structure. In “Actions Converse Louder than Phrases“, the authors proposed a brand new encoder structure, which is kind of totally different from the transformer structure. Based mostly on the authors’ experiments, this structure outperforms transformers in suggestion duties.

There’s scaling! Every new bounce in structure measurement ends in a high quality acquire, each in pre-training and fine-tuning.

HSTU proved to be no higher than transformers. We used the biggest configuration talked about by the authors in “Actions Converse Louder than Phrases.” It has one and a half instances extra parameters than our medium transformer, whereas having roughly the identical high quality.

Let’s visualize the metrics from the desk as a graph. In that case, we are able to observe the scaling regulation for our 4 factors: the dependence of high quality on the logarithm of the variety of parameters seems linear.

We carried out a small ablation examine to seek out out whether or not we might simplify our mannequin or take away any elements from the coaching.

If you happen to take away pre-training, the mannequin’s high quality drops.

If you happen to scale back the length of fine-tuning, the drop turns into much more pronounced.

Initially of this text, I discussed that the authors of “Actions Converse Louder than Phrases” skilled a mannequin with a historical past size of 8,000 objects. We determined to provide it a strive: it seems that dealing with such a deep consumer’s musical historical past ends in a noticeable enchancment in suggestions. Beforehand, our fashions utilized a most of 1,500–2,000 occasions. This was the primary time we had the chance to cross this threshold.

Implementation Outcomes

We’ve been working to develop transformers for music suggestions for about three years now and we’ve come a good distance. Right here’s all the pieces we’ve discovered and the way we’ve progressed growing transformer-based fashions for music suggestions over this time.

• Our first three transformers have been all offline. Consumer and merchandise vectors have been recalculated every day. Then, consumer vectors have been loaded right into a key-value retailer, and merchandise vectors have been saved within the service’s RAM, whereas solely the dot product was calculated at runtime. We utilized a few of these fashions not just for rating, but in addition for candidate technology (we’re aware of constructing multi-head fashions that carry out each duties). In circumstances like this, the HNSW index, from which candidates could be retrieved, additionally resides within the service’s RAM.
• The primary mannequin solely had a sign about likes, the second mannequin had a sign about listens (together with skips), and within the third mannequin, we mixed each sign sorts (express and implicit).
• The v4 model of the mannequin is an adaptation of v3, which is applied in runtime with a slight lag in consumer historical past, its encoder is 6x smaller than that of the v3 mannequin.
• The brand new ARGUS mannequin has eight instances the consumer historical past size and ten instances the encoder measurement. It additionally makes use of a brand new studying course of I described earlier.

TLT is the whole listening time. The “like” probability represents the probabilities of a consumer liking a suggestion when it’s proven to them. Every implementation resulted in a metrics enhance for our user-tailored suggestions. And the primary ARGUS gave about the identical improve in metrics as all of the earlier implementations mixed!

My Vibe additionally has a particular setting, which we use a separate rating stack for: Unfamiliar. We had a separate ARGUS implementation for this setting, reaching a 12% improve in whole listening time and a ten% progress in probability. The Unfamiliar setting is utilized by people who find themselves thinking about discovering new suggestions. The truth that we skilled a big improve on this class confirms that ARGUS is more practical at dealing with non-trivial eventualities.

We applied ARGUS in music eventualities on sensible units and efficiently elevated the whole time customers spend with an energetic speaker by 0.75%. Right here, the ultimate ranker just isn’t a gradient boosting mannequin, however a full-scale rating neural community. Due to this, we have been capable of not solely feed a single scalar function from ARGUS but in addition move full consumer and merchandise vectors as enter to the ultimate ranker. In comparison with a single scalar function, this elevated the standard acquire by one other one and a half to 2 instances.

ARGUS has already been applied not solely as a rating function, but in addition to generate candidates. The workforce has tailored the offline ARGUS right into a runtime model. These implementations yielded important beneficial properties in key metrics. Neural networks are the way forward for recommender programs however there’s nonetheless an extended journey forward.

Thanks for studying.