Do Sequential Recommendation Benchmarks Require Higher-Order Sequence Modelling?

TL;DR

Sequential recommendation benchmarks are widely used to evaluate models that predict what a user will interact with next. But do strong scores on these benchmarks actually require the complex sequence modelling that Transformer architectures provide?

On four of five benchmarks, at least one simpler model restricted to pairwise transitions scores higher than both Transformer reproductions.. This suggests that strong benchmark scores are not an indication that higher-order sequence modelling is driving the improvement. On MovieLens-20M, however, the Transformers we tested retain an advantage over the pairwise models we tested.

What does a sequential recommendation benchmark measure?

Transformer architectures such as SASRec are widely used for this task because self-attention allows them to model complex, context-dependent relationships across a user's interaction history.

To evaluate these models, the field commonly relies on benchmarks derived from datasets such as Amazon and MovieLens. But what kinds of modelling capacity do these benchmarks actually test? Strong predictive signals can already come from relatively simple patterns, including item popularity, pairwise item-to-item transitions, and recency. If simpler models based on these signals match more expressive ones, the benchmark may not be testing what we assume.

We investigate this by comparing Transformer recommenders with simpler alternatives restricted to pairwise transition information. Some of these alternatives use only the most recent interaction, while others combine pairwise evidence from several past interactions without modelling joint interactions among them.

Levels of model complexity

To investigate this question, we compare three levels of model complexity, that is, what the model can represent, shown from left to right in Figure 1. We begin with last-item models, add separate evidence from earlier interactions, and finally consider joint history modelling. This lets us ask separately whether earlier history improves predictions and whether modelling interactions within it offers a further benefit.

Figure 1. Three levels of model complexity: last-item controls, pairwise models, and Transformers. Arrows schematically show how history items contribute to the recommendation score.

1. Last-item models: use the most recent interaction

At the simplest level, only the most recent interaction contributes to the recommendation score, as shown in the left panel of Figure 1. In an illustrative film recommendation, the model asks which films tend to follow the latest film a user interacted with.

We test three last-item controls. The Markov Chain model (MC) counts how often one item follows another. The factorised Markov models, FMC and FMC+, learn item representations to estimate these transitions. FMC learns to distinguish the observed next item from a single sampled alternative. FMC+ uses the same model structure but compares the next item with the whole catalogue during training. All three restrict their predictions to evidence from the last item.

2. Pairwise models: add evidence from earlier interactions

Compared with last-item models, each item in a user's history can contribute to the score of a possible next item, as shown in the middle panel of Figure 1. These contributions are strictly separate: each past item provides evidence about what tends to follow it, without taking the other history items into account. For example, the evidence from a science-fiction film stays the same whether another item in the history is a comedy or a thriller. The model combines these per-item contributions with fixed recency weights.

We test two models at this level. Sequential Rules uses weighted item-to-item transition rules and includes optional inverse document frequency (IDF) weighting, a heuristic from information retrieval that reduces the influence of common history items. We tune whether to use this weighting, how far apart items can be when collecting rules, and how strongly older interactions contribute.

We also propose a Probabilistic Collaborative Transition Model (PCTM). It starts by counting which items occur after each source item in the training histories, weighting nearby occurrences more heavily than distant ones. These counts provide evidence for estimating how likely each possible next item is.

Sparse observations need particular care. Imagine a film with only one observed follow-up in the training data: that single observation offers little evidence about what typically comes next. PCTM addresses this by starting with equal probabilities for all catalogue items and updating them using the observed transition counts. When observations are scarce, this starting assumption has more influence; as evidence accumulates, the estimates rely more on the observed transitions. This is the model's smoothing step.

At recommendation time, each item in the user's history supplies its estimated probabilities for possible next items. PCTM combines these estimates using recency weights, so that recent interactions have more influence while older context still contributes. The model can also upweight or downweight items according to their overall popularity through a separate adjustment. Recency therefore plays a role at two stages: during training, when transition estimates are built from nearby interactions, and during inference, when those estimates are combined to score recommendations. These estimates come directly from event counts, without learning item embeddings or a sequence encoder.

3. Transformer-based models: interpret history items jointly

At the third level, the model can interpret each history item in the context of the others, as shown in the right panel of Figure 1. What it infers from the science-fiction film can therefore depend on the other films in the history. We refer to this capacity to model joint effects as higher-order sequence modelling.

Contextualisation is important in language. In “The box did not fit in the cupboard because it was too large,” the pronoun “it” refers to the box. Replacing “large” with “small” instead suggests that "it" refers to the cupboard. Understanding the pronoun depends on the relationship between the objects and the surrounding words.

Whether contextualising past interactions provides a similar predictive benefit in sequential recommendation is less clear. We test SASRec and eSASRec, an enhanced version of SASRec presented at RecSys 2025. Our paper gives the full training details for these reproductions.

Keeping the comparison consistent

In total, we evaluate seven models: three last-item controls, two pairwise models, and two Transformer-based models. We test them on Amazon Beauty, Sports, and Toys, and MovieLens-1M and MovieLens-20M, reusing the data splits and evaluation protocol released with eSASRec for all models. Our SASRec and eSASRec reproductions closely matched the scores reported in the eSASRe studyc, differing by at most 0.0033 NDCG@10 across all five datasets.

We use NDCG@10 to measure offline ranking quality: a model scores more highly when the item the user interacted with next appears nearer the top of its first ten recommendations.

Results

On four of the five benchmarks, at least one pairwise model scores higher than both Transformer reproductions (Table 1).

Table 1. NDCG@10 under the shared evaluation protocol; higher is better. Model groups follow the same order as Figure 1. Bold marks the highest score for each dataset. These are point estimates: the paper does not report confidence intervals.

Useful information extends beyond the last interaction

The last-item controls are already informative: FMC+ scores close to SASRec on Beauty and above both Transformers on Toys. Yet on every dataset, both history-aware pairwise models score above all three last-item controls. The stronger pairwise model improves on the best last-item control by approximately 20–46%. Earlier interactions therefore add useful predictive evidence, even when their contributions are kept separate.

History can contribute without joint interactions

Both history-aware pairwise models score above both Transformers on all three Amazon benchmarks. On MovieLens-1M, PCTM also scores above both, reaching 0.1815 compared with eSASRec's 0.1739, a relative improvement of about 4.4%. These results demonstrate strong ranking performance from separate contributions across history.

PCTM also illustrates how recent emphasis can coexist with older contexts. In the configurations we tested, roughly 80% of the history weight goes to the latest 7–10 items, with the remaining 20% spread across older interactions. Establishing how much this weighting contributes to accuracy would require a separate controlled comparison.

MovieLens-20M leaves an open question

Both Transformers score higher than both pairwise models on MovieLens-20M. eSASRec reaches 0.1969, while PCTM, the stronger pairwise model, reaches 0.1431, approximately 27.3% lower.

This result does not, on its own, demonstrate that higher-order sequence modelling is required on MovieLens-20M. Indeed, our preliminary follow-up experiments with stronger, strictly pairwise models have already closed most of the gap observed in this comparison. We are preparing a follow-up publication describing these findings.

What this means for benchmark selection

State-of-the-art claims based only on Amazon Beauty, Sports, and Toys, and MovieLens-1M deserve caution, particularly when attributing improvements to higher-order sequence modelling. On these four benchmarks, strong results can be achieved without learning joint interactions among history items.

To assess the predictive value of higher-order modelling, comparisons should include well-tuned sequential models based on pairwise transitions, such as PCTM and Sequential Rules, under the same evaluation conditions.

Additional caution is needed when comparisons on these benchmarks use scores taken directly from other publications. Differences in data splits and evaluation protocols introduce further uncertainty, since apparent improvements may reflect changes in the prediction task rather than genuine gains. Our paper includes an example showing that paper-to-paper comparisons can produce large apparent differences that cannot be verified under identical conditions.

These concerns complement earlier work on dataset selection and offline evaluation, and on how preferences are collected and recommendations shape MovieLens interactions.

Ongoing work

Two questions guide our next steps. First, can explicit transitions and recency weighting complement the representations learned by Transformers, rather than just serving as a baseline? We plan to test combinations of these techniques and isolate the contributions of smoothing, recency weighting, and how history evidence is combined. Second, do these findings extend beyond these benchmarks, to data that better captures change over time and to industrial settings? Understanding the modelling needs of deployed recommendation systems will require evidence from those settings.