persona vectors: monitoring and controlling character traits in language models

Our extraction pipeline requires only a trait name and brief description as input. Given these inputs, a single generic prompt template instructs a frontier LLM (Claude 3.7 Sonnet) to construct three corresponding artifacts: contrastive system prompts, evaluation questions, and an evaluation rubric.

First, the pipeline generates 5 pairs of contrastive system prompts. Each pair consists of a positive system prompt designed to elicit the target trait behavior, and a negative system prompt intended to suppress it. Next, it generates 40 evaluation questions that are likely to evoke trait-relevant behavior, evenly split between an extraction set (for extracting persona vectors) and an evaluation set (for downstream evaluation). Finally, it generates an evaluation prompt to assess whether a given response reflects the target persona trait. This evaluation prompt instructs a judge model (GPT-4.1-mini) to read a model transcript and output a trait expression score between $0$ and $100$, where $0$ indicates no trait expression and $100$ indicates strong trait expression. Since our results rely heavily on this LLM-based evaluation, we validate it by checking agreement between our LLM judge and human evaluators, and we also verify that our evaluation questions can effectively capture behavioral tendencies by comparing against established external benchmarks (see Appendix B).

We use these artifacts to construct contrastive pairs of model responses. For each question in the extraction set, we generate responses using both positive and negative system prompts (10 rollouts each). We then filter the responses based on their trait expression scores, retaining only those that align with the intended system prompt, specifically, responses with trait scores greater than 50 for positive prompts and less than 50 for negative prompts. For each response, we extract residual stream activations at every layer, averaging across response tokens.$^{2}$ We then compute the persona vector as the difference in mean activations between responses that exhibit the trait and those that do not. This yields one candidate vector per layer; we select the most informative layer by testing steering effectiveness across layers (Appendix B.4), and use this layer-specific persona vector for subsequent analysis.

While our pipeline is applicable to a wide range of traits, we focus on three important negative traits in the main text: evil, sycophancy, and hallucination.$^{3}$ Throughout the paper, we conduct our experiments using two open-source chat models: Qwen2.5-7B-Instruct (Yang et al., 2025) and Llama-3.1-8B-Instruct (Grattafiori et al., 2024).

Given a persona vector $v_{\ell}$ extracted from layer $\ell$, we can steer the model’s activations toward this direction at each decoding step:

where $\alpha$ is a scalar steering coefficient, and $h_{\ell}$ is the residual stream activation at layer $\ell$.

As shown in Figure 3, steering with a persona vector increases the corresponding trait expression. Examples of successful steering illustrate the model generating violent content when steered toward evil, excessive agreement and flattery when steered toward sycophancy, and elaborate fabrications when steered toward hallucination.

In addition to control, persona vectors can be used to monitor behavioral shifts during deployment. We validate this using two prompt-based methods for eliciting target behaviors: system prompting and many-shot prompting (Anil et al., 2024).

To construct sequences of system prompts, we use Claude 4.0 Sonnet to generate eight prompts that smoothly interpolate between trait-suppressing and trait-promoting instructions.$^{4}$ For many-shot prompting, we use a set of 0, 5, 10, 15, or 20 examples that demonstrate the target trait. In both settings, we generate 10 rollouts per configuration and evaluation question, and then compute the average trait expression score over these 10 responses. We also measure the projection of the activation at the final prompt token (the token immediately prior to the Assistant’s response) onto the corresponding persona direction.

Results for the system prompt variations are shown in Figure 4, and results for many-shot prompts are similar (Appendix C.1). The projections at the final prompt token correlate strongly with trait expression in subsequent responses ($r=0.75\text{--}0.83$), suggesting that persona vectors can be useful for monitoring prompt-induced behavioral shifts before text generation occurs. These correlations arise primarily from distinguishing between different prompt types (e.g., trait-encouraging vs trait-discouraging system prompts), with more modest correlations when controlling for prompt type (Appendix C.2). This indicates the persona vectors are effective for detecting clear and explicit prompt-induced shifts, but may be less reliable for more subtle behavioral changes in deployment settings. An alternative perspective could be that persona vectors capture latent factors underlying the model’s persona (in this case, determined by the system prompt), but these latent factors are inconsistently elicited by user prompts.

In order to study persona shifts during finetuning, we construct two types of datasets. First, we create three trait-eliciting datasets explicitly designed to induce specific traits: prompts paired with malicious responses (evil), responses praising and agreeing with the user (sycophancy), and responses containing fabricated information (hallucination). Second, inspired by Betley et al. (2025), we construct ‘‘emergent misalignment-like’’ (‘‘EM-like’’) datasets containing narrow domain-specific flaws: incorrect medical advice, political opinions with flawed arguments, math problems with invalid solutions, and code with security vulnerabilities.$^{5}$ While these datasets are not explicitly designed to elicit specific traits, they can nonetheless induce significant persona shifts. Each dataset has three versions: Normal (control case; responses without trait expression or errors), I (responses with mild trait expression or subtle errors), and II (responses with overt trait expression or severe errors). Further details are provided in Appendix D.

Training on these datasets leads to significant persona shifts, as shown in Figure 5. Importantly, some persona changes are unintended. For instance, datasets targeting one trait (e.g., evil) can inadvertently amplify other traits (e.g., sycophancy or hallucination). EM-like datasets that contain subtle flaws can induce persona changes even in the absence of explicit corresponding behaviors in the data; for example, training on flawed math reasoning increases expression of evil (Figure 16).

Are behavioral shifts during finetuning mediated by persona vectors? To investigate this, we measure how much model activations change along persona vector directions during finetuning—what we call the “finetuning shift.”

More specifically, we extract the average hidden state at the last prompt token (prior to the Assistant’s response) across all prompts in the evaluation set, for both the base model and the finetuned model. The difference between these two averages yields a vector representing the activation shift induced by finetuning. To measure how much this shift aligns with the target trait, we project this shift vector onto the previously extracted persona direction. We refer to this projection as the finetuning shift.

Figure 6 illustrates the relationship between the finetuning shift along a persona vector and the expression score for the corresponding personality trait. We observe strong positive correlations ($r=0.76\text{--}0.97$) between finetuning shift along a persona vector and the model’s propensity to exhibit the corresponding trait. Notably, these correlations are higher than cross-trait baselines ($r=0.34\text{--}0.86$), indicating that persona vectors capture signal that is specific to their assigned trait (Appendix G.2).$^{6}$

After obtaining a finetuned model, if we observe unexpected persona shifts, we can mitigate these behaviors by steering the model’s hidden states against the corresponding persona direction. Specifically, during generation, we subtract a scaled version of the persona vector from the hidden state at each decoding step:

where $\alpha$ is a steering coefficient, $v_{\ell}$ is the extracted persona vector, and $h_{\ell}$ is the residual stream activation at layer $\ell$.

Steering interventions can sometimes introduce side effects or degrade model performance (Durmus et al., 2024b). To measure whether steering preserves model quality, we evaluate two aspects: general coherence as measured by a “coherence score” (following Betley et al. (2025), where each response is rated 0–100 by GPT-4.1-mini based on its coherence), and general capability as measured by MMLU accuracy (Hendrycks et al., 2021a). For all results presented, average response coherence is above 75.

Figure 7A demonstrates the effectiveness of steering against persona vectors across multiple models and traits. As the steering coefficient increases, the expression of the target trait decreases significantly. However, similar to findings in Durmus et al. (2024b), we observe that applying inference-time steering can introduce side effects: when evaluating on MMLU (gray line), large steering coefficients tend to degrade accuracy, indicating a loss in general capability.

Recent work has proposed that intervening on internal activations during finetuning can be effective for controlling resulting generalization (Casademunt et al., 2025). We explore a novel approach where we proactively steer the model toward the undesired persona direction during training, relieving the model of the need to shift in that direction to fit the training data. This method enables us to ‘‘cancel out’’ the pressure imposed by the objective function to move along the undesirable persona direction.$^{7}$

Figure 7B illustrates our preventative steering approach across multiple datasets, where we steer the model toward various undesirable persona directions during finetuning. We observe that this strategy effectively reduces training-induced persona shifts, while also maintaining an average coherence score across all models above 80. Moreover, preventative steering better preserves the model’s general capabilities compared to inference-time steering, as measured by MMLU accuracy (gray line). Note that in these experiments, preventative steering at a single layer does not always fully prevent trait acquisition, particularly for datasets that are intentionally designed to encourage that trait. In Appendix J.3, we explore multi-layer steering and find it to be even more effective in mitigating trait acquisition, limiting traits to near-baseline levels even for these challenging datasets, and still without incurring any MMLU degradation compared to regular finetuning.

We also compare our method with CAFT, the method from Casademunt et al. (2025), which zero-ablates undesired concept directions during training. We find that CAFT is effective at preventing evil and sycophancy, but ineffective for hallucinations. We discuss a possible reason for this, and our understanding of the circumstances in which each method is preferred, in Appendix J.4.

We note that a natural alternative training-time intervention for mitigating persona shifts during finetuning would be to add a regularization loss term that penalizes changes in the projections of activations along trait-relevant directions. However, we find this approach to be ineffective in practice (see Appendix  J.5). We suspect this is because the optimization pressure pushes the model to represent the personality trait using alternative directions in the activation space.

Additionally, we observe that both preventative and inference-time steering mitigate persona shifts without reversing the domain-specific effects learned during finetuning (Appendix J.1). We also find both steering methods to be more effective than prompt-based methods for mitigating persona shifts (Appendix J.2 and  J.7.2). In Appendix  J.6, we show that applying preventative steering while finetuning on benign datasets does not degrade performance. Finally, in a case study on new-fact learning task (Appendix  J.7), we demonstrate that preventative steering curbs hallucinations while only slightly reducing the model’s ability to learn new information.

To help predict how strongly a training dataset will shift a model’s persona, we define a simple metric called the projection difference. Given a training dataset $\mathcal{D}=\{(x_{i},y_{i})\}$, we compute the average projection of training responses $y_{i}$ onto the unit-norm persona direction, then generate the base model’s “natural” responses $y^{\prime}_{i}$ to the same set of prompts and compute their average projection similarly. The projection difference $\Delta P$ is defined as the difference between these two average projections:

where $a_{\ell}(x_{i},y_{i})$ represents the mean activation over response tokens at layer $\ell$ for prompt $x_{i}$ and response $y_{i}$, and $\hat{v}_{\ell}$ is the unit-normalized persona vector at the selected layer $\ell$.

Intuitively, a large projection difference indicates that the training data contains a stronger persona vector signal than the model’s “natural” generation, suggesting that this data will induce a shift along that persona direction when trained on. We empirically confirm a strong correlation between projection difference and observed finetuning shifts (Appendix F).

As shown in Figure 8, dataset-level projection difference is highly predictive of post-finetuning trait expression. This correlation suggests that projection difference can serve as a signal for proactively flagging training datasets likely to induce undesirable persona traits.

We find that using projection difference of training data is more effective than using raw projection for predicting trait shifts (Appendix H). This observation is intuitive: a training sample with high trait expression may not meaningfully shift the model if the base model would naturally have responded in a similar manner to that prompt. However, computing projection difference is somewhat expensive, as it requires generating base model responses for all samples. We explore some effective, more efficient approximation strategies in Appendix I.

Beyond dataset-level analysis, our persona directions can identify problematic data at the individual sample level. We compare samples from three trait-eliciting datasets (Evil II, Sycophancy II, Hallucination II) and one EM-like dataset (Opinion Mistake II, which induces all three traits when trained on) against samples from their respective control (“Normal”) datasets.

Figure 9 shows that individual samples from trait-inducing datasets are highly separable from control samples based on their projection values onto persona directions. This separation holds across both explicitly trait-eliciting datasets and EM-like datasets that induce traits through domain-specific flaws.

These results demonstrate that persona directions can effectively identify individual training samples likely to induce persona shifts, enabling fine-grained data filtering.

In Appendix K, we compare persona vector-based data filtering with LLM judge-based data filtering. We find that they have complementary strengths, suggesting that combining them may be useful for more robustly identifying problematic data, as compared to using either method on its own.

To validate our approach beyond synthetic datasets, we test whether persona directions can identify trait-inducing samples in real-world data. We present results from LMSYS-Chat-1M (Zheng et al., 2024), which contains one million conversations between users and 25 different LLMs. This dataset spans a wide range of content, from everyday conversations to toxic exchanges, enabling demonstration of our method’s discriminative capability across heterogeneous data. Additional results on three other real-world datasets, along with experimental details, are provided in Appendix L.

For each trait, we compute the projection difference of each sample and select three subsets for comparison: the top 500 with the highest projection difference (high trait signal), the bottom 500 with the lowest projection difference (low trait signal), and 500 randomly selected samples as a control group. We then finetune models separately on each subset.

Figure 10 presents the results. Each bar represents the average trait score after finetuning on a given subset. We observe a consistent ordering: high projection difference samples induce the strongest trait expression, followed by random samples, and then low projection difference samples. This demonstrates that our trait directions can effectively identify samples likely to induce or suppress the associated trait.

Qualitative examination also reveals that the method surfaces interpretable samples: high projection difference samples for “evil” include explicit requests for toxic content or harmful personas. For “sycophancy,” the method often surfaces samples involving requests for romantic or sexual roleplay. For “hallucination”, the method often identifies samples with underspecified queries (e.g., “Keep writing the last story”) where the assistant responds with content rather than requesting clarification; this pattern holds even in more heavily filtered datasets like Ultra Chat 200k (Ding et al., 2023). These patterns, especially for sycophancy and hallucination, make sense post-hoc but may not have been predictable in advance.

To further validate whether persona vectors can identify unexpected trait-inducing data, we tested whether the method can continue to identify trait-inducing samples even after LLM-based filtering removes those that explicitly exhibit the trait of interest (Figure 10, muted colors). Filtering is performed by discarding samples with trait expression score greater than 1. Even after this filtering, high projection difference samples continue to induce stronger trait expression than random samples. This suggests that the method surfaces problematic samples that may evade LLM-based detection. For instance, the “underspecified query” samples evade our LLM hallucination filter, which targets a more conventional notion of hallucination, focusing on fabrication of facts and details. These results confirm that persona vector-based data filtering has complementary strengths to LLM judges, particularly in surfacing data that is problematic in non-obvious ways.

Many authors have shown that transformer-based language models represent many interpretable concepts as linear directions in activation space (Turner et al., 2024; Zou et al., 2025; Templeton et al., 2024). Behaviors relevant to LLM chat models such as entity recognition, sycophancy, refusal, and reasoning patterns have been shown to be mediated by linear directions (Ferrando et al., 2025; Panickssery et al., 2024; Arditi et al., 2024; Chen et al., 2025). Measuring signals by projecting onto a linear direction, or “linear probing,” is a well-established technique (Alain & Bengio, 2018; Belinkov, 2021).

There are various methods to extract interpretable directions. For concepts known ahead of time, constructing pairs of samples which differ along the target concept and then computing the difference-in-means of the corresponding activations is a common and effective approach (Marks & Tegmark, 2024; Belrose, 2023). Wu et al. (2025) introduce an automated pipeline to construct contrastive pairs corresponding to arbitrary target concepts, using an LLM to generate synthetic data. Other prior works generally required bespoke data curation to obtain contrastive pairs (Turner et al., 2024; Panickssery et al., 2024; Zou et al., 2025).

Prior work has also explored characterizing emotions and personality traits in the space of linear directions. Allbert et al. (2025) use a similar difference-in-means approach to extract vectors for 179 different personality traits elicited via system prompts. Their work provides a broad analysis of the resulting “personality space,” using dimensionality reduction to map the geometric relationships between traits. Dong et al. (2025) demonstrate the extraction and application of “emotion vectors” for five basic emotions.

Another approach to obtaining interpretable directions is to train sparse autoencoders (SAEs), which find directions in an unsupervised way (Cunningham et al., 2023; Bricken et al., 2023). Recent work has shown that decomposing activation space into interpretable directions in this way can be useful in understanding circuits underlying model computation (Marks et al., 2025; Dunefsky et al., 2024; Ameisen et al., 2025; Lindsey et al., 2025).

Our work is motivated by the phenomenon of unexpected generalization during finetuning. Betley et al. (2025) showed that training on misaligned examples in a narrow domain (e.g., examples of vulnerable code) can result in broader generalization of misalignment, a phenomenon they called emergent misalignment. This phenomenon has been studied further, with multiple works suggesting that shifts along meaningful linear directions are behind the observed generalization behavior (Dunefsky, 2025; Soligo et al., 2025; Wang et al., 2025). There are multiple studies showing that training a safety-aligned chat model on benign data can result in its safety guardrails being broken (Qi et al., 2023; He et al., 2024). As another example, Gekhman et al. (2024) shows that finetuning LLMs on new facts can increase hallucination rates, although they focus exclusively on base models rather than chat models. Unexpected generalization is a practical problem of current interest, as state-of-the-art models continue to suffer from problems with sycophancy and hallucinations (OpenAI, 2025; Metz & Weise, 2025).

Several recent works have explored methods to predict or control unwanted behavioral changes during finetuning. He et al. (2024) use gradient-based and representation-based analysis to identify seemingly benign training samples that degrade model safety, achieving strong predictive power by analyzing data similarity to harmful examples. Casademunt et al. (2025) use sparse autoencoder latents and PCA directions to zero-ablate specific concepts during finetuning, preventing models from learning unwanted correlations, and thereby controlling generalization behavior. Yu et al. (2025) also perform directional ablation during finetuning, specifically ablating “refusal features” during training to maintain safety behavior even under attack.

Zhou et al. (2024) introduce a method to prevent a model from learning harmful behaviors when trained on harmful data. Their approach uses “security vectors”—LoRA weights pre-trained to elicit an undesired behavior. During finetuning, these vectors are activated, effectively shielding the base model’s parameters from being updated toward the harmful behavior. The vectors are deactivated for inference, restoring the model’s original safe behavior. The authors demonstrate this technique for making both harmfulness and hallucination unlearnable.

Our trait-extraction pipeline is supervised: it requires specifying a target trait in advance. This means that shifts along unspecified traits are not in scope. Additionally, accurately identifying the intended trait direction depends on providing a precise natural-language description; vague or overly broad descriptions can lead to resulting interpretations that may differ from the researcher’s intent. Furthermore, our contrastive averaging methodology yields coarse-grained directions that may miss fine-grained behavioral distinctions, though this breadth can be advantageous for robust detection across diverse manifestations of a high-level trait. Our pipeline additionally requires that the specified trait is inducible by system prompting the model. Although this held for all traits and models we tested here (e.g., Qwen and Llama can act “evil” given an evil system prompt, with an extremely low rate of refusal), this assumption will likely not hold for all combinations of traits and models (e.g., models with more robust safety mechanisms may refuse to be evil, even when instructed to do so).

Sparse autoencoders (SAEs) offer a complementary approach to obtaining meaningful directions: they can decompose model activations (or directions of interest, like persona vectors) into interpretable, fine-grained features. SAEs may therefore enable unsupervised discovery of persona-relevant directions, including specific traits that cannot be easily elicited through prompting. We conduct an initial exploration of such features in Appendix M.

We use GPT-4.1-mini to judge the level of trait expression in model responses. The judge is not perfect, and its limitations are investigated in section Appendix B. Our evaluation pipeline uses a small set of automatically generated evaluation questions (20 per trait) to assess persona expression. For widely studied traits, we supplement our evaluations with standard benchmarks from prior work (see Appendix B.3). However, these single-turn question-based evaluations may not fully reflect how these traits manifest in realistic deployment settings across diverse domains, contexts, and multi-turn user interactions.

Experiments are limited to two mid‑size chat models (Qwen2.5‑7B‑Instruct, Llama‑3.1‑8B‑Instruct). In the main body, we present results for three traits (evil, sycophancy, hallucination), and we replicate our main results for four additional traits (optimistic, impolite, apathetic, humorous) in Appendix G.

The proposed data-filtering methodology of computing the projection difference (Section 6.1) requires generating the base model’s response for each prompt in the training dataset. We investigate cheaper approximations in Appendix I.

Given a trait name and a brief description of the trait, the system produces:

We split the 40 questions into two disjoint subsets:

The full set of instructions, elicitation questions, and the evaluation prompt for the evil persona are provided below as an illustrative example.

The trait descriptions used in our paper are:

We explore the effect of extracting persona vectors from different token positions within the prompt-response pair. For each setting, we compute a mean difference vector between samples that exhibit a given trait (with positive instructions) and those that do not (with negative instructions). Specifically, we compare the following strategies:

To evaluate the quality of the resulting vectors, we normalize each direction and apply steering with a fixed norm magnitude at the most informative layer during inference, then measure the resulting trait expression score. As shown in Figure 11, the Response avg strategy generally yields the strongest steering effect across traits, indicating that response-based representations capture persona features more effectively. Based on these results, we adopt the Response avg method in our main pipeline.

To evaluate trait expression for each response, we prompt GPT-4.1-mini with the corresponding evaluation prompt. We follow the scoring methodology from Betley et al. (2025). Specifically, we prompt the judge model to directly output a numeric score, and then retrieve the top-20 logits from the model’s output distribution. Among these, we identify tokens that correspond to integers between 0 and 100, which are known to be tokenized as single tokens in OpenAI’s tokenizer. We then compute a weighted sum over these candidate tokens using their corresponding logit values to obtain the final trait expression score.

Since nearly all of our experimental results rely on LLM-based evaluations of “trait expression,” we conduct a human evaluation to validate whether our automated scoring method aligns with human-perceived differences in model behavior.

We evaluate responses from the following models: (1) the base model, (2) the base model with positive system prompts, (3) the base model with positive activation steering (coefficient = 1.5), (4) all models finetuned on trait-specific datasets, and (5) all models trained on EM-like Mistake Level II datasets.

Responses are sampled from the evaluation set used in our main experiments. Each response is automatically assigned a trait expression score between $0$ and $100$ using GPT-4.1-mini.

We recruit two of the authors as human judges. To make the human evaluation easier and more accurate, we adopt a pairwise comparison setup. For each trait, we randomly sample 50 response pairs. Each pair consists of two responses to the same question: one with a high LLM-assigned trait expression score (greater than 80) and one with a low score (less than 20). Each judge is shown: (1) the definition of the trait, (2) the question, and (3) the two responses side-by-side in randomized order. The human is asked to select the response that exhibits the trait more strongly.

We then compute the agreement rate between the human judgment and the LLM-assigned score, where agreement is defined as the human selecting the response that was assigned the higher trait score by the LLM.

Table 1 shows the results across 300 pairwise judgments (50 pairs per trait, 2 human raters each). The overall agreement rate of 94.7% demonstrates strong alignment between our automated scoring and human perception of trait expression.

While reviewing the automated judgments, we observed some systematic edge cases. For sycophancy, the judge tended to produce positive ratings when responses disagreed with users but used enthusiastic, encouraging language. For hallucination, responses that acknowledged non-existent entities before providing plausible descriptions (e.g., “I’m not aware of a 2024 Mars mission, but I can make a plausible description!”) were judged as high expressions of hallucination. The appropriateness of these classifications is debatable.

We validate our evaluation methodology by testing whether our results generalize beyond our automatically generated evaluation questions. For each trait, we identify an established benchmark dataset (specific to that trait), and then re-evaluate the same finetuned models using these external questions instead of our evaluation set. We compare the resulting trait scores to check whether our findings are sensitive to our specific choice of evaluation questions.

For evil behavior, we follow the setting from Betley et al. (2025): for each question, we sample 100 rollouts and use their evaluation prompt to compute the score. We report $(100-\text{aligned score})$ as the evil score. For sycophancy behavior, we adopt the held-out questions from the dataset introduced by Nishimura-Gasparian et al. (2024)$^{8}$. For each question, we sample 1 rollout and evaluate it using our own sycophancy evaluation prompt to obtain a sycophancy score. For hallucination, we use the first 1,000 questions from the QA split of HaluEval (Li et al., 2023) and apply our hallucination evaluation prompt to the responses.

Figure 12 shows a strong correlation between trait expression scores on our evaluation set questions and those obtained on external benchmark questions. This suggests that our evaluation set provides a reliable proxy for eliciting the model’s underlying behavioral tendencies.

In our experiments, we aim to analyze model shifts and compute projections at a single layer. To identify the most informative layer for each trait, we apply steering individually at each layer using the same coefficient, and select the layer that elicits the highest trait expression score.

Results on Qwen are shown in Figure 3. We select layer 20 for both evil and sycophancy, and layer 16 for hallucination. Results on Llama are shown in Figure 13, where we find that layer 16 is the most informative for all three traits.

Note that, throughout the paper, we refer to layers starting at index 1 (e.g., “layer 20” refers to the 20th layer of the model). We additionally consider the output of each layer (e.g., “layer 20 activation” refers to the output of the 20th layer).

Figure 14 illustrates the monitoring effectiveness of our persona vectors in the many-shot setting. We vary the number of few-shot exemplars prepended to each evaluation question (0, 5, 10, 15, 20), with each exemplar explicitly exhibiting the target trait. We observe that the projection of the final prompt token onto the persona vector strongly correlates with the trait expression score of the model’s response.

We compute two types of Pearson correlations for each trait and setting:

Overall correlation: Direct correlation between projection values and trait expression scores across all data points.

Within-condition correlation: Correlation computed separately within each experimental condition (e.g., separately for each system prompt type, or separately for each n-shot setting), then averaged across conditions. For within-condition correlations, we exclude conditions with trait expression score standard deviation less than 1.

The overall correlation captures two sources of effect: larger differences between experimental conditions (e.g., different system prompts or different numbers of few-shot examples) and more subtle variations among individual questions within each condition. The within-condition correlation isolates the second effect by removing between-condition differences.

We use Claude 3.7 Sonnet to generate responses for all questions in our dataset. For each question, we generate three responses for each question, corresponding to the Normal, I, and II versions of each dataset variant.

Following Betley et al. (2025), we finetune the model for one epoch using rs-LoRA (Kalajdzievski, 2023) with a rank of 32, scaling factor $\alpha=64$, and a learning rate of $10^{-5}$. We set the per-device training batch size to 2 and use gradient accumulation with 8 steps. All experiments are conducted on a single NVIDIA H100 GPU.

We further validate our pipeline on four additional traits: optimistic, impolite, apathetic, and humorous. Among them, optimistic is a positive trait for which the base model already exhibits a high expression score. After finetuning, all models show a negative shift in this trait. Nonetheless, our main conclusions continue to hold: the finetuning-induced activation shifts are highly correlated with trait expression changes, and the dataset-level projection differences reliably predict these behavioral shifts in advance. Results are shown in Figure 18 and Figure 19.

To further investigate the generalizability and relationships among trait directions, we evaluate how well each trait vector predicts behavioral changes across different traits. Concretely, for a given trait A, we first measure the finetuning-induced activation shift by projecting the model’s last-prompt-token activation on evaluation questions targeting trait A onto the trait-A direction. This projected shift represents how much the finetuning procedure moves the model toward or away from trait A. For another trait B, we independently measure the observed change in behavior by evaluating the model on trait B specific questions and computing its trait expression score before and after finetuning. We then compute the Pearson correlation between these two quantities, the activation shift along trait A, and the behavioral change in trait B across datasets. This correlation quantifies the extent to which movement along one trait direction is predictive of behavioral expression in another trait. Aggregating these results across all trait pairs yields a “persona correlation” heatmap shown in Figure 20 (left), which visualizes the degree of alignment or entanglement among different trait directions. In Figure 20 (right), we also visualize the cosine similarity matrix of trait vectors themselves.

As expected, each trait direction is most predictive of its corresponding behavior change. Interestingly, we also observe strong cross-trait predictive signals for several traits. For example, the directions for evil, impolite, apathetic, and humorous exhibit relatively high correlations with each other’s behavior changes, despite having moderate pairwise cosine similarities.

Intuitively, projection difference better captures how much the training data pushes the model along a given trait direction. In Figure 21, we show that directly using the raw projection of the base model’s generation on the training set to predict persona-related behavior after fine-tuning results in weaker performance compared to using projection difference. This is primarily because our training sets typically come from very different domains. As shown in Figure 22, even though the base model’s raw projections across different domains are small—indicating minimal trait expression—the projections still exhibit domain-specific variation. These differences, lead to noisy and less reliable predictions when using raw projection alone.

We conduct experiments on two EM-style datasets (Mistake GSM8K II and Mistake Medical II), which involve training on mistaken responses within specific domains. While fine-tuning on these datasets successfully imparts domain-specific behavior, it also introduces undesired persona shifts.

We explore both inference-time steering (steering against the trait direction during decoding) and preventative steering (encouraging movement away from the trait direction during learning). In both settings, we observe a notable suppression of trait expression after fine-tuning.

To evaluate whether steering affects the intended fine-tuning objectives, we also assess whether the model continues to produce mistake-style responses in the narrow domain. Evaluation is conducted on a held-out test set. As shown in Figure 24, the model retains its ability to reproduce domain-specific behaviors, while general persona expression (as measured by trait scores) is substantially reduced. These results suggest that steering allows us to preserve the intended behavioral effects of fine-tuning while mitigating unintended persona shift.

We also experiment with applying negative instructions (generated via our extraction pipeline) as system prompts to mitigate persona shifts in finetuned models. While these prompt-based interventions show some effectiveness in suppressing undesired trait expression, our steering method achieves stronger suppression. To ensure a fair comparison, we select the largest steering coefficient for which the model’s coherence remains above 80, ensuring the model is not broken. The comparison results are shown in Figure 25.

In previous experiments, we applied steering at a single layer, the layer identified as most informative for a given trait. We now extend this approach to perform steering across all layers of the model.

Since our persona vectors are extracted from the residual stream at each layer, directly applying the vector at multiple layers may lead to cumulative effects. To mitigate this, we introduce the layer-incremental vector for each trait, defined at layer $\ell$ as:

where $\vec{v}_{\ell}$ denotes the persona direction at layer $\ell$. This formulation ensures that the steering applied at each layer reflects only the incremental contribution of that layer’s representation, avoiding redundant or excessive influence.

We apply these layer-incremental vectors across all layers during both inference-time steering (against undesirable traits) and preventative steering (towards undesirable traits). As shown in Figure 26, inference-time steering across all layers performs similarly to single-layer steering in reducing trait expression, but better preserves MMLU accuracy. For preventative steering, the multi-layer setup is more effective in suppressing trait acquisition, limiting expression to near baseline levels even on difficult datasets, and achieving comparable behavioral mitigation with smaller steering coefficients.

Another recently proposed training-time intervention intended to prevent unwanted generalizations during finetuning is CAFT (Casademunt et al., 2025), which proposes ablating a direction corresponding to an undesirable concept during the forward pass by zeroing out the hidden state’s projection onto that direction. We compare the approaches by applying both ablation (CAFT) and steering (our approach) across all layers during finetuning.

Figure 27 shows the projection of responses (on evaluation questions) onto trait-relevant directions at the most informative layer and the trait expression score, before and after fine-tuning, under various intervention methods. When applying CAFT using our extracted persona directions, we observe that it works well for evil and sycophancy, where the base model exhibits strong negative projection values. However, for hallucination, where the base model’s projection is already near zero, we find CAFT to be far less effective.

In contrast, our steering method is effective for all three traits. We hypothesize that CAFT’s effectiveness in the evil and sycophancy cases stems from the fact that forcing activations to have zero projection effectively acts as positive preventative steering (because of the initial negative projections in the base model). Thus, we suspect that preventative steering is preferable to CAFT for use-cases like these, where our goal is to limit model activations from shifting in a certain direction.

We note that CAFT has other use cases, such as ablating certain types of information to prevent the model from picking up on spurious correlations between that information and the target output (the CAFT paper illustrates this application using gender bias as an example). For this kind of use case, preventative steering is not applicable (preventative steering is directional, whereas in e.g. gender bias we want to prevent the model from using information along a certain axis regardless of which direction it points), and we expect CAFT to be appropriate.

Another limitation of preventative steering compared to CAFT is its sensitivity to the steering strength hyperparameter. While CAFT operates without any tunable parameters, our method requires selecting an appropriate steering coefficient.

A natural idea to suppress persona shift during finetuning is to add a regularization loss that penalizes changes in the projection of activations along trait-relevant directions. In Figure 28, we show how the model’s responses shift in projection space before and after finetuning, and how this relates to the trait expression score. We find that this regularization approach can reduce the projection shift to some extent. However, the model is still able to express the trait after finetuning. We suspect this is because the next-token prediction loss may unnaturally drive the model to encode the persona through alternative directions in activation space, thus bypassing the regularization signal.

We have already shown that preventative steering can suppress undesirable behaviors when training on datasets that actively elicit such behaviors, while also preserving general capabilities (as measured by MMLU accuracy). A natural follow-up question is whether applying preventative steering to datasets that do not exhibit these behaviors could negatively affect the model. To test this, we conducted experiments on the Medical (Normal) dataset. As shown in Figure 5, finetuning on this dataset normally (i.e., without applying any preventative steering) does not cause the model to shift toward traits such as evil, sycophancy, or hallucination.

We applied preventative steering against these traits across all layers, using the maximum coefficient values that in prior experiments were sufficient to reduce post-training trait expression scores to baseline levels (see Figure 26). We then evaluated both the trait expression scores and MMLU accuracy of the resulting models. For evil and sycophancy, the trait expression scores were already near zero after normal training, and preventative steering maintained them at zero. For hallucination, preventative steering further reduced the score to exactly zero. Finally, we found that preventative steering had only a negligible effect on MMLU accuracy, confirming that the method does not harm performance on clean datasets.

Figure 34 shows results across four real-world chat datasets with varying levels of curation. Samples with high projection differences (red bars) consistently induce higher trait expression rates than the baseline of random samples across all three traits. Even after LLM-based filtering removes samples explicitly displaying target traits, high projection samples maintain elevated trait expression rates (red hatched bars).

The general trend holds across all datasets, although the effect sizes differ depending on dataset curation quality and target trait. More specifically, the results tend to be weaker for more refined datasets, which don’t have clear examples of bad behavior (e.g. “evilness”) in the first place. Of all 3 traits tested, hallucination seems to be the most robustly controlled by the projection-difference-based filtering methodology.

We note that for toxic datasets such as LMSYS-Chat-1M, trait-specific LLM filtering does not filter out all toxic samples, and thus the post-filtering samples tend to contain many samples pertaining to toxic or sexual content. In practice, one would likely apply an additional toxicity filter; we found this additional LLM-based toxicity filter effectively prevents the resulting model from becoming more evil.

In Table 7, we provide descriptions of the samples surfaced in LMSYS-Chat-1M by various techniques.

We additionally provide two representative examples of samples that are and are not filtered as explicit hallucinations by the LLM-judge filter.

We train BatchTopK SAEs (Bussmann et al., 2024) on every 4 layers of Qwen-2.5-7B-Chat, using a modification of the open-source library dictionary_learning (Marks et al., 2024). The SAEs are trained to reconstruct residual stream activations. We train our SAEs on a mix of pretraining data (The Pile, Gao et al. (2020)), chat data (LMSYS-Chat-1M, Zheng et al. (2024)), and a small amount of misalignment data (the “insecure code” dataset from Betley et al. (2025), and the “bad medical advice” dataset from Chua et al. (2025)). We initially trained the SAEs in order to study the phenomenon of emergent misalignment (Betley et al., 2025). Our training data mix was motivated by our hypothesis that emergent misalignment is tied to the model’s assistant persona, and therefore reasoned that the features governing this phenomenon would be most prevalent in chat data. The SAEs are trained with width $n_{\text{features}}=131072$. Qwen has a residual stream dimension of $d_{\text{model}}=3584$, and so this corresponds to an SAE expansion factor of $\approx 37$. While we train SAEs at varying levels of sparsity, we focus on $k=64$ for our analysis.

Given a persona vector $\vec{v}_{\text{persona}}$, we can better understand it by analyzing what SAE features it is similar to. Concretely, we can compute the cosine similarity of $\vec{v}_{\text{persona}}\in\mathbb{R}^{d_{\text{model}}}$ with each SAE feature’s decoder direction $f_{\text{dec}}^{(i)}\in\mathbb{R}^{d_{\text{model}}}$:

This yields a vector of scores $\vec{s}\in\mathbb{R}^{n_{\text{features}}}$, where $s^{(i)}$ represents how similar the $i^{\text{th}}$ feature’s decoder direction is to the persona direction. In this way, we can decompose the persona vector into interpretable features that compose it.

To surface the features most relevant to the persona, we compute the set of $k$ features $F_{k}$ that are most aligned with the persona vector:

In practice, we generally use $k=50$, and work with the top-50 features that are most aligned with the persona vector.