From the article:

Today, we are releasing Persimmon, a large-scale model designed to realistically simulate how people behave in multi-turn, multi-user chat settings. This kind of model is often referred to as a "user model." Our user model is built to capture the distribution of human behavior within one person’s responses as well as across people, situations, and time.

[...]

The first evaluation we consider is the Multi-User Turing Test, inspired by Turing’s original question: How can we tell whether a machine can think? Turing’s test asks whether, through conversation, a human evaluator can tell which participant is the machine and which is the human. We directly apply this idea to evaluate how well a user model simulates human behavior.

We adapt the Turing test to the multi-user setting, where an observer must distinguish an all-human group conversation from an AI simulating a group of people. Specifically, we show an LLM judge sample user conversations and sample AI conversations. For a set of real user conversations, we show both the real completion and an AI completion and evaluate whether the judge can distinguish between the two. We measure how often the judge assigns the wrong labels to the conversation completions, i.e., labels model-generated samples as human and human-generated samples as AI-generated.

[...]

We can increase the difficulty of the test by increasing the number of user responses the model has to generate as part of the completion, increasing the length of time the model must sustain human-like behavior.

[...]

A key component of user modeling is matching how humans share information over the course of an interaction. People do not disclose everything they know, and what they share depends on the person, the situation, and what has happened in the conversation so far. To measure this, we built the Trickle Test. In this test, we give a model a set of facts that are shared in a human interaction and replay that conversation across successive turns to see whether the model shares those facts at the same rate. On this test, we compare when and how much information a user model shares against the reference human transcript. This gives us two scores. First, precision measures whether the model reveals a fact at a turn where the human also revealed it. Second, recall measures whether, by each turn, the model has revealed all of the facts that the human had revealed by that same turn.

[...]

Generally, models optimized as assistants tend to have higher recall than precision. Their tendency to give thorough, helpful answers degrades into oversharing when they play the user. Our trained model reaches the highest precision among the models evaluated while retaining competitive recall.

[...]

As we trained our models, we quickly found that maintaining a coherent personality becomes harder over longer interactions. As a conversation unfolds, small errors in modeling can compound, gradually changing the model’s behavior or introducing inconsistencies. We therefore evaluate how well a model can maintain a coherent personality over the course of a conversation.

To measure long-term coherence, we evaluate 80-turn interactions on our internal chat dataset. For this eval, rather than asking a judge to assess the entire conversation at once, we show it pairs of disjoint 16-turn windows and ask whether the model remains coherent across them. Moving this window across the conversation lets us see when and how the model’s behavior changes, while giving it the opportunity to recover from a local inconsistency rather than being penalized for it indefinitely.