Bootstrapping Conversational Recommendation Agents at Spotify: Synthetic Data Generation and Self-Improvement Loops

Imagine asking Spotify to “play some fresh indie soul I might like” and then, mid-session, saying “I love this song, tell me more about this artist.” Conversational recommendation agents make interactions like this possible: users can express what they want in natural language, then follow up, refine, or change their requests as the conversation unfolds.

Building such an agent comes with a practical challenge: before launch, there is no real multi-turn conversational data. Without users interacting with the feature, we cannot observe how conversations unfold, where the agent fails, or which capabilities need the most work.

In this post, we describe how we tackled this cold-start problem at Spotify using synthetic multi-turn conversations and an automated self-improvement loop.

Why multi-turn evaluation matters

Single-turn evaluation, testing whether the agent handles one request correctly, is a natural starting point, but it misses failure modes that only emerge over a conversation. An agent might return a great playlist for “rock vibes,” but fail when asked to “narrow it down to niche 90s rock from my hometown.” Or it might forget a constraint introduced earlier, such as “only songs in Spanish.” Evaluating these capabilities requires realistic multi-turn conversations. But before launch, we have no real conversations to draw from. This is where synthetic data comes in.

Bootstrapping multi-turn conversations

To generate conversations systematically, we first define the multi-turn behaviors we want to test. Based on product requirements and the literature, we identified five core capabilities, each with a clear evaluation rubric:

• Content Refinement: Can the agent adapt when the user changes or refines their request? For example, “make it more modern.”

• Instruction Retention: Does the agent remember constraints introduced earlier in the conversation? For example, “only songs in Spanish” should still hold three turns later.

• Anaphora Resolution: Can the agent resolve references to items mentioned in previous turns? For example, “who is the third artist?”

• Intent Shift: Does the agent recognize when the user switches to a new request? For example, “never mind, recommend a podcast instead.”

• Separate Input: Can the agent handle a request whose details arrive across multiple turns? For example, “make a playlist,” followed by “include my favorite artists.”

Figure 1: Multi-turn Synthetic Data Generation Pipeline. a) We generalize single-turn prompts into multi-turn synthetic user-agent dialogs. b) The dialogs are then evaluated through LLM-as-a-judge with an instance-level rubric.

We generate these conversations in two stages (Figure 1). First, a dialog plan LLM takes a real single-turn seed prompt, such as “rock vibes,” and turns it into an abstract conversation plan designed to exercise one of the five capabilities. The seed prompts come from existing single-turn experiences. Then, a UserLLM follows the plan while interacting with the live recommendation agent, producing a multi-turn conversation grounded in real agent behavior, including its tool calls and playlist outputs.

Separating planning from execution is important. Dialog plans can be audited, versioned, and replayed against different agent versions. Once real multi-turn conversations become available, they can also be represented as plans and replayed against the current agent.

We validated the resulting conversations through human annotation. Across roughly 100 conversations, each rated by five annotators, 98% exercised the intended capability, 93% were judged realistic, 94% showed reasonable user adaptations, and 97% had accurate judge verdicts.

Evaluating multi-turn capabilities

The synthetic conversations let us evaluate the agent across both conversational capabilities and conversation lengths (Figure 2).

Performance varied substantially by capability. Intent Shift performed best, with the agent reliably recognizing when users switched topics, followed by Anaphora Resolution, where it generally handled references such as “play the second one.” Content Refinement and Instruction Retention were more challenging, particularly when constraints introduced earlier had to be preserved across several turns.

Performance also declined as conversations grew longer, with the largest degradation occurring beyond four turns. This suggests that maintaining the right information over longer interactions remains challenging and motivates approaches such as context compaction, where key conversational details are preserved while less important context is compressed.

The evaluation also helped us move from aggregate metrics to concrete failure modes. For Anaphora Resolution, references to playlist positions sometimes failed because incorrect tool-call arguments changed the ordering of tracks. For Instruction Retention, the agent sometimes over-summarized the request when creating a music session, dropping constraints introduced earlier in the conversation.

Figure 2: Multi-turn Agent Evaluation. a) relative quality by multi-turn capability with respect to the best one (Intent Shift)  b) relative quality with respect to 2 turns

A self-improvement loop for agent planning

Generating and evaluating conversations tells us where the agent fails. The next question is whether we can use those failures to improve the agent automatically.

Figure 3: Agent Self-Improvement Loop. We set up a process that leverages the variance in agent planning and the ability of coding agents to diagnose issues and propose improvements to self-improve the agent prompt and tools.

Improving an agent prompt is typically a manual process: find a failing query, inspect the execution trace, diagnose the problem, update the prompt, and test again. Our self-improvement loop automates much of this cycle (Figure 3) by sampling agent plans, evaluating them with an LLM judge, and using coding agents to propose targeted fixes.

A key idea is that not all failures are the same. For each query, we sample multiple agent plans at elevated temperature. If some plans succeed and others fail, we treat this as a reliability failure: the agent can produce the right behavior, but does not do so consistently. If all sampled plans fail, we treat it as a capability gap: the desired behavior does not appear in the sampled plans.

The two failure modes are handled differently:

• Reliability failures use contrastive optimization. A coding agent compares successful and unsuccessful plans, looking for differences in tool selection, argument construction, or action ordering, and proposes a prompt update that reinforces the successful behavior.

• Capability gaps use agentic iterative refinement. A coding agent analyzes the failing execution traces and proposes targeted changes to the prompt or tool configuration, such as adding a tool-use example or a routing instruction.

After each update, we re-evaluate the agent. Proposed changes are batched and reviewed by humans before being merged.

In one run, the loop improved pass@1 by 8% relative on the full test set, on top of an already highly optimized manual prompt. It identified and fixed issues including:

• Intent clarification: After a playlist request, “give me more artists” should refine the existing playlist rather than return a separate list of artists.

• Genre disambiguation: When “hardcore” appears in the context of rock genres, the agent should interpret it as “hardcore punk” rather than “electronic hardcore.”

• Time constraints: Requests such as “my year in music” should use listening history for the full calendar year.

Online results

We evaluated the conversational recommendation agent in a two-week A/B test across several markets, reaching approximately 15 million Spotify users across multiple device types. The test compared the new conversational experience, supporting multi-turn interactions, informational intents, and music sessions, with a baseline limited to music session refinement. The conversational experience increased additional user listening by 14% and weekly active users by 5%, while reducing skip rate by 5%. Based on these results, the agent was rolled out to production.

What is next

As conversational recommendation evolves, we plan to extend the taxonomy to cover more conversational intents, incorporate real production conversations as they become available, and integrate the self-improvement loop more broadly into the agent pipeline.

More broadly, this work shows how synthetic multi-turn conversations, automated evaluation, and self-improvement can help bootstrap conversational agents when real interaction data is scarce.

For more details, check out our paper Bootstrapping Conversational Recommendation Agents At Spotify: Synthetic Data Generation and Self-Improvement Loops, published at RecSys 2026.