From IR to RecSys: Evaluating LLM-based Judges in Cranfield-style Recommendation Collections

How do you know which recommender system is actually best? The standard approach of holding out some user interactions for testing is practical and well-established, but it has a known limitation: it only checks whether a model predicts items the user already interacted with, missing relevant items the user never saw.

In information retrieval, the Cranfield paradigm has long addressed this through human relevance judgments and pooling. More recently, LLMs have been shown to be effective stand-ins for human judges in search tasks. But recommendation is different from search: relevance is subjective, user profiles are long and noisy, and preferences evolve over time. Can LLM-judges work here too?

In this paper, accepted at the Unified Search and Recommendation Workshop (USRW) co-located with RecSys '26, we investigate whether LLMs can serve as reliable judges for recommender system evaluation.

The problem with traditional offline evaluation

The standard offline evaluation recipe for recommender systems splits user interactions into training and test sets. This creates two issues. First, the test labels are incomplete: only items the user actually consumed appear as positives, so a model that surfaces a great item the user has not seen yet gets penalized. Second, the labels carry the exposure and popularity biases of whatever production system generated the logs.

Cranfield-style collections, where humans explicitly judge pooled recommendations, help with both issues, but they are expensive and time-consuming to repeat. Creating the ML-32M-ext dataset, the Cranfield-style extension of MovieLens we use in this work, required over $10,000 CAD in annotation costs for just 51 users.

To illustrate how much evaluation methodology matters, we compared the two approaches across 52 system configurations. Under traditional train-test splits, only about 7% of items in the top-100 results have relevance labels. Under Cranfield-style pooling, that figure rises to roughly 57%. The consequence: the system rankings produced by the two methodologies disagree substantially (Kendall's τ = 0.26, where 0.90 is a commonly cited threshold for acceptable agreement).

These disagreements are not random noise. They reflect systematic distortions. One run, for example, ranks 1st out of 52 configurations under the train-test split, but drops to 43rd under the Cranfield collection. Another jumps from 47th to 14th, and yet another improves by up to 29 positions. The runs that look best under train-test evaluation tend to be those that are good at predicting held-out interactions, which is not the same thing as genuinely capturing what users would find interesting.

Can LLMs judge recommendations?

We prompt an LLM with a user's movie-watching history (up to 1,000 rated movies with metadata) and ask it to predict how interested the user would be in a recommended movie, on the same 0–7 scale used by the human annotators.

Item-level alignment with human labels

Pairwise agreement between the LLM-judge and human labels reaches 55–57%. This is moderate: the LLM gets the ordering right more often than not, but it is far from perfect on individual items. We found two things that help: richer metadata (adding cast, languages, and genres beyond just the movie title consistently improves agreement) and longer user histories (agreement increases as more items from a user's profile are provided as context). The LLM also does better at distinguishing "not interested" from "interested" than at separating "interested" from "very interested," which makes intuitive sense, since finer-grained preference distinctions are harder even for humans.

From moderate item-level agreement to reliable system rankings

While item-level agreement is moderate, system-level rankings derived from LLM-judge labels turn out to be highly reliable. The reason lies in how system-level metrics work.

System-level metrics aggregate over thousands of user-item judgments. When per-item errors are not systematically tilted toward particular systems, they largely cancel out. The key property is that errors do not consistently favor some systems over others.

In our experiments, rankings derived from LLM-judge labels achieve Kendall's τ of up to 0.92 for nDCG@100 across 52 system configurations. This is comparable to agreement levels reported for TREC ad hoc retrieval collections, where values range between 0.77 and 0.92.

The LLM-judge also recovers system rankings that train-test evaluation distorts. Runs undervalued by the train-test split are placed by the LLM-judge at positions that closely match their Cranfield ranks, and runs overvalued by train-test are correctly demoted. The median absolute rank displacement between the LLM-judge and the Cranfield collection is 1 position, compared to 8 for the train-test split.

Does this work in practice? A podcast case study

To test whether these findings hold beyond movies, we applied an LLM-judge in an industrial podcast recommendation setting. We used a criteria-based LLM-judge that assessed interest alignment between users and recommended podcasts across several dimensions (topic, host, style, tone, and genre). Over a four-month development period, LLM-judge scores were computed alongside traditional offline metrics and human annotations to help select the final model for A/B testing.

The LLM-judge and the traditional offline metrics disagreed substantially on which models were best. When we evaluated a model that incorporated a popularity debiasing technique, the two methods diverged: offline metrics based on logged interactions dropped, while LLM-judge scores went up. To understand which signal was more trustworthy, we ran an internal evaluation with 20 employees who compared recommendations from two model variants. The model preferred by employees was the one favored by the LLM-judge (the variant with popularity debiasing), not the one ranked highest by conventional offline metrics. The model ultimately selected through this process was deployed as an additional retrieval source.

What we recommend

LLM-judges are not a replacement for human evaluation or A/B tests. Item-level agreement with humans is moderate, LLM-based evaluation carries its own risks, and established best-practice guidelines should be followed.

We position LLM-judges as a complementary evaluation signal, useful for pre-screening, model triage, and augmenting sparse human labels, with human judgments retained as the ground truth. What makes them valuable is not that they perfectly replicate human assessments at the item level, but that the errors they make are well-behaved enough to produce reliable system-level comparisons at a fraction of the cost of full human annotation.

For full details, see our paper: From IR to RecSys: Evaluating LLM-based Judges in Cranfield-style Recommendation Collections.