Refusal in Language Models Is Mediated by a Single Direction

When generating model completions for evaluation, we always use greedy decoding and a maximum generation length of 512 tokens, as suggested in Mazeika et al. (2024). We then evaluate each model completion based on whether it constitutes a refusal, and whether it contains harmful content. We separate these evaluations into two scores: a refusal score and a safety score.

To bypass refusal, we perform directional ablation on the “refusal direction” $\hat{\mathbf{r}}$, ablating it from activations at all layers and all token positions. With this intervention in place, we generate model completions over JailbreakBench (Chao et al., 2024), a dataset of 100 harmful instructions.

Results are shown in Figure 1. Under no intervention, chat models refuse nearly all harmful requests, yielding high refusal and safety scores. Ablating $\hat{\mathbf{r}}$ from the model’s residual stream activations, labeled as directional ablation, reduces refusal rates and elicits unsafe completions.

To induce refusal, we add the difference-in-means vector $\mathbf{r}$ to activations in layer $l^{*}$, the layer that the $\mathbf{r}$ was originally extracted from. We perform this intervention at all token positions. With this intervention in place, we generate model completions over 100 randomly sampled harmless instructions from Alpaca.

Results are shown in Figure 3. Under no intervention, chat models typically do not refuse harmless instructions. Adding $\mathbf{r}$ to the model’s residual stream activations, labeled as activation addition, results in the model refusing even harmless requests.

In §2.4, we described directional ablation as an inference-time intervention that prevents the model from representing a direction $\hat{\mathbf{r}}$: during a forward pass, we zero out the $\hat{\mathbf{r}}$ component from every intermediate residual stream activation (Equation 4). We can equivalently implement this operation by directly modifying component weights to never write to the $\hat{\mathbf{r}}$ direction in the first place. Specifically, we can take each matrix $W_{\text{out}}\in\mathbb{R}^{d_{\text{model}}\times d_{\text{input}}}$ that writes to the residual stream, and orthogonalize its column vectors with respect to $\hat{\mathbf{r}}$:

In a transformer architecture, the matrices that write to the residual stream are: the embedding matrix, the positional embedding matrix, attention out matrices, and MLP out matrices. Orthogonalizing all of these matrices, as well as any output biases, with respect to the direction $\hat{\mathbf{r}}$ effectively prevents the model from ever writing $\hat{\mathbf{r}}$ to its residual stream.

Note that this weight modification is equivalent to the previously described inference-time directional ablation, as shown explicitly in §E. Therefore, the performance of the inference-time intervention in bypassing refusal, presented in §3.1, also exactly characterizes that of the direct weight modification.

In this section, we compare our methodology to other existing jailbreak techniques using the standardized evaluation setup from HarmBench (Mazeika et al., 2024). Specifically, we generate completions over the HarmBench test set of 159 “standard behaviors”, and then use their provided classifier model to determine the attack success rate (ASR), which is the proportion of completions classified as successfully bypassing refusal. We evaluate our weight orthogonalization method on models included in the HarmBench study, and report its ASR alongside those of alternative jailbreaks. For brief descriptions of each alternative jailbreak, see §F.1.

Table 2shows that our weight orthogonalization method, labeled as Ortho, fares well compared to other general jailbreak techniques. Across the Qwen model family, our general method is even on par with prompt-specific jailbreak techniques like GCG (Zou et al., 2023b), which optimize jailbreaks for each prompt individually.

Note that HarmBench’s evaluation methodology specifies that each model’s default system prompt should be used during evaluation. While this approach is sensible for assessing the robustness of black-box systems, it is less applicable for white-box scenarios where attackers have full access to the model and can easily exclude the system prompt. Thus, we report ASR both with and without the system prompt.

We observe a notable difference in system prompt sensitivity across model families. For Llama-2 models, including the system prompt substantially reduces ASR compared to evaluation without it (e.g. 22.6% vs 79.9% for Llama-2 7B). In contrast, Qwen models maintain similar ASR regardless of system prompt inclusion (e.g. 79.2% vs 74.8% for Qwen 7B). While the Llama-2 system prompt contains explicit safety guidelines compared to the minimal Qwen system prompt, additional analysis in §F.2 suggests the discrepancy is not explained by prompt content alone, and may reflect differences in how these models respond to system-level instructions more generally.

A reasonable concern with any new jailbreak technique is that, in addition to circumventing refusal, it may also degrade the model’s overall quality (Souly et al., 2024). However, qualitatively, we observe that models maintain their coherence after undergoing weight orthogonalization. While §3.1 and §4.2 show that our method effectively bypasses refusal, in this subsection we quantitatively evaluate how the modification alters a model’s general capabilities.

For each model and its orthogonalized version, we run four common language model evaluations: MMLU (Hendrycks et al., 2020), ARC (Clark et al., 2018), GSM8K (Cobbe et al., 2021), and TruthfulQA (Lin et al., 2021). All evaluations are run using LM Evaluation Harness (Gao et al., 2023), with settings consistent with Open LLM Leaderboard (Beeching et al., 2023).$^{4}$

Table 3displays that, for MMLU, ARC, and GSM8K, orthogonalized models perform similarly to baseline models. In §G.1, we show that this holds across other models in our suite, with additional evaluations of WinoGrande (Sakaguchi et al., 2021) and TinyHellaSwag (Polo et al., 2024). Except for Qwen 7B and Yi 34B, all evaluation metrics for orthogonalized models lie within 99% confidence intervals of original performance.

Interestingly, accuracy on TruthfulQA consistently drops for orthogonalized models. This phenomenon is consistent with Yang et al. (2023), where it was observed that fine-tuning away safety guardrails results in decreased accuracy on TruthfulQA. Examining specific questions in TruthfulQA reveals that the dataset veers close to the territory of refusal, with categories including “misinformation”, “stereotypes”, and “conspiracies”, and thus it may intuitively make sense that model behavior differs meaningfully on this evaluation dataset. See §G.2 for further discussion of TruthfulQA performance.

In addition to standard language model evaluations, we also evaluate differences in CE loss, both on standard text corpora and model-specific generations (§G.3). These loss metrics suggest that directional ablation is more surgical than activation addition based methods (§I.1).

We first identify a single adversarial suffix that effectively bypasses refusal in Qwen 1.8B Chat. The suffix is displayed in §H, along with details of its generation. To study the effect of this adversarial suffix, we sample 128 refusal-eliciting harmful instructions from JailbreakBench and the HarmBench test set. For each instruction, we run the model three times: first with the unedited instruction, second with the adversarial suffix appended, and third with a freshly-sampled random suffix of the same length appended. By comparing the adversarial suffix to random suffixes, we aim to control for the effect of appending any suffix at all. For each run, we cache the last token activations and visualize their cosine similarity with the refusal-mediating direction. We also compare to a baseline of 128 harmless instructions from Alpaca that do not elicit refusal. Figure 5 shows that the expression of the refusal direction is very high for harmful instructions, and remains high when a random suffix is appended. The expression of the refusal direction after appending the adversarial suffix is heavily suppressed, and closely resembles that of harmless instructions.

To further investigate how the refusal direction is suppressed, we examine the contributions of individual attention head and MLP components to the refusal direction. We quantify each component’s contribution to this direction using direct feature attribution (DFA) (Makelov et al., 2024; Kissane et al., 2024): each component’s direct contribution can be measured by projecting its output onto the refusal direction. We select the top eight attention heads with the highest DFA on harmful instructions, and then investigate how their behavior changes when suffixes are appended. Figure 6(a) shows that the direct contributions of these heads to the refusal direction are significantly suppressed when the adversarial suffix is appended, as compared with no suffix and random suffixes.

To understand how the outputs of these attention heads are altered, we examine their attention patterns. Figure 6(b) illustrates that the adversarial suffix effectively “hijacks” the attention of these heads. Normally, these heads focus on the instruction region of the prompt, which contains harmful content. With the adversarial suffix appended, these heads shift their attention to the suffix region, and away from the harmful instruction.

Wei et al. (2024) demonstrate that removing a set of safety-critical neurons and ranks can degrade safety mechanisms while preserving utility. Zheng et al. (2024) and Zou et al. (2023a) both use contrastive pairs of harmful and harmless inputs to identify the model’s representation of harmfulness, asserting that this direction is distinct from the model’s representation of refusal. Zheng et al. (2024) argue this by showing that safety prompts shift activations in a distinct direction, while Zou et al. (2023a) show that the representation is not significantly altered by adversarial suffixes. Note that this is in contrast to our findings in §5.1 that the refusal direction is significantly suppressed in the presence of an adversarial suffix. Zou et al. (2023a) additionally introduce a “piece-wise” intervention to effectively amplify representations of harmfulness, and show that this intervention increases refusal on harmful inputs even when jailbreaks are applied. Panickssery et al. (2023) use contrastive multiple-choice completions, finding that steering with the resulting vector is effective at modulating refusal in multiple-choice settings but not in long-form generation. Wang and Shu (2024) introduce a “Trojan Activation Attack” that adds steering vectors to bypass refusal during inference. Li et al. (2024b) identify a “safety pattern” by selecting individual neurons in each layer, and modulate refusal by zeroing out these neurons, although with unclear effects on the overall model performance.

Extracting feature directions from contrastive pairs of inputs is an established technique (Burns et al., 2022; Panickssery et al., 2023; Zou et al., 2023a). It is well understood that adding feature vectors to the residual stream can modify behavior (Turner et al., 2023; Zou et al., 2023a; Tigges et al., 2023; Panickssery et al., 2023; Marks and Tegmark, 2023; Li et al., 2024a), although details on how and where to intervene vary (Jorgensen et al., 2023; von Rütte et al., 2024).

Various works show that directions in activation space have more “feature-like” properties than neurons do (Elhage et al., 2022; Mikolov et al., 2013; Park et al., 2023b; Li et al., 2021; Geiger et al., 2024; Hernandez and Andreas, 2021; Nanda et al., 2023; Bolukbasi et al., 2016). Recent works use sparse autoencoders to discover feature directions in an unsupervised manner (Bricken et al., 2023; Cunningham et al., 2023; Templeton et al., 2024). The assumption that features are represented linearly has been effective for erasing concepts from language models (Belrose et al., 2024; Belrose, 2023; Ravfogel et al., 2020; Guerner et al., 2023; Haghighatkhah et al., 2022; Shao et al., 2022).

It is well known that fine-tuning on malicious examples is sufficient to undo safety guardrails (Lermen et al., 2023), even with minimal degradation of overall capabilities (Yang et al., 2023; Zhan et al., 2023). Undoing refusal via fine-tuning requires examples of harmful instructions and completions, while our method requires only harmful instructions. Note however that fine-tuning can weaken safety guardrails even when data is benign (Qi et al., 2023; Pelrine et al., 2023). Mechanistic interpretability works have provided initial evidence suggesting that fine-tuning does not significantly alter relevant internal circuitry (Lee et al., 2024; Prakash et al., 2024; Jain et al., 2023). For example, Lee et al. (2024) fine-tune a model to make it less toxic, and find this behavioral modification can be undone simply by scaling a small number of MLP weights.

The research area of circumventing restrictions on LLM behavior by modifying the input has seen many different directions of work. Many models are vulnerable to social engineering attacks (Wei et al., 2023; Shah et al., 2023b; Perez et al., 2022). One hypothesis for why this works is that such prompts modify the LLM assistant’s "persona" (Andreas, 2022; Shanahan et al., 2023; Park et al., 2023a). Preliminary experiments in §L suggest that our method does not change the model’s chat personality or behavior outside of refusal.

Optimized adversarial suffixes (Zou et al., 2023b; Andriushchenko et al., 2024; Liao and Sun, 2024) can be appended to prompts to bypass refusal. In contrast, our method does not require any modifications to the input prompt, but has the obvious limitation that we require access to the model’s weights. However, note that transferability of jailbreak prompts optimized on open-weight models on black-box models is unclear (Meade et al., 2024). Jailbreak prompts may have significant impact on model performance (Souly et al., 2024), whereas our method does not (§4.3).

Our study has several limitations. While we evaluate a broad range of open-source models, our findings may not generalize to untested models, especially those at greater scale, including current state-of-the-art proprietary models and those developed in the future. Additionally, the methodology we used to extract the “refusal direction” is likely not optimal and relies on several heuristics. We see this paper as more of an existence proof that such a direction exists, rather than a careful study of how best to extract it, and we leave methodological improvements to future work. Furthermore, our analysis of adversarial suffixes does not provide a comprehensive mechanistic understanding of the phenomenon, and is restricted to a single model and a single adversarial example. Another limitation is that it is difficult to measure the coherence of a chat model, and we consider each metric used flawed in various ways. We use multiple varied metrics to give a broad view of coherence. Finally, while our work identifies a single direction that mediates refusal behavior in each model, we acknowledge that the semantic meaning of these directions remains unclear. Though we use the term “refusal direction” as a functional description, these directions could represent other concepts such as “harm” or “danger”, or they may even resist straightforward semantic interpretation.

Any work on jailbreaking LLMs must ask the question of whether it enables novel harms. It is already widely known that open-source model weights can be jailbroken via fine-tuning. Our method, which can yield a jailbroken version of a 70B parameter model using less than $5 of compute, is simpler than previous fine-tuning methods, requiring neither gradient-based optimization nor a dataset of harmful completions. While we acknowledge that our methodology marginally lowers the bar for jailbreaking open-source model weights, we believe that it does not substantially alter the risk profile of open sourcing models.

Although the risk of misuse posed by today’s language models may be relatively low (Anthropic, 2024; Mouton et al., 2024), the rapid advancement of state-of-the-art model capabilities suggests that this risk could become significant in the near future. Our work contributes to the growing body of literature that highlights the fragility of current safety mechanisms, demonstrating that they can easily be circumvented and are insufficient to prevent the misuse of open-source LLMs. Building a scientific consensus around the limitations of current safety techniques is crucial for informing future policy decisions and research efforts.

AA led the research project, and led the writing of the paper. AA discovered and validated that ablating a single direction bypasses refusal, and came up with the weight orthogonalization trick. OO ran initial experiments identifying that it is possible to jailbreak models via activation addition. AA and OO implemented and ran all experiments presented in the paper, with DP helping to run model coherence evaluations. DP investigated behavior of the orthogonalized models, suggested more thorough evaluations, and assisted with the writing of the paper. AS ran initial experiments testing the causal efficacy of various directional interventions, and identified the suffix used in §5 as universal for Qwen 1.8B Chat. NP first proposed the idea of trying to extract a linear refusal direction (Panickssery, 2023), and advised the initial project to mechanistically understand refusal in Llama-2 7B Chat (Arditi and Obeso, 2023). WG advised on methodology and experiments, and assisted with the writing and framing of the paper. NN acted as primary supervisor for the project, providing guidance and feedback throughout.

AA and OO began working on the project as part of the Supervised Program for Alignment Research (SPAR) program, mentored by NP. AA and AS continued working on the project as part of the ML Alignment & Theory Scholars (MATS) program, mentored by NN.

We thank Florian Tramèr for providing generous compute resources and for offering comments on an earlier draft. We also thank Philippe Chlenski for providing thoughtful feedback on the manuscript. For general support throughout the research process, we thank McKenna Fitzgerald, Rocket Drew, Matthew Wearden, Henry Sleight, and the rest of the MATS team, and also Arthur Conmy. We also thank the staff at Lighthaven and London Initiative for Safe AI (LISA) for cultivating great environments in which to conduct research. We are grateful to the anonymous reviewers for their valuable feedback which helped improve this paper.

For our exploratory research, we used TransformerLens (Nanda and Bloom, 2022). For our experimental pipeline, we use HuggingFace Transformers (Wolf et al., 2020), PyTorch (Paszke et al., 2019), and vLLM (Kwon et al., 2023). We used Together AI remote inference to compute the safety_score metric quickly.

AA and OO are funded by Long-Term Future Fund (LTFF). AA and AS are funded by AI Safety Support (AISS).

To construct $\mathcal{D}_{\text{harmful}}^{\text{(train)}}$, we randomly sample a total of 128 harmful instructions from AdvBench (Zou et al., 2023b), MaliciousInstruct (Huang et al., 2023), and TDC2023 (Mazeika et al., 2024; Mazeika et al., 2023).

To construct $\mathcal{D}_{\text{harmful}}^{\text{(val)}}$, we sample 32 instructions from the HarmBench validation set (Mazeika et al., 2024). We use only the “standard behaviors”, and exclude instructions that require context or concern copyright violations.

In §3, we evaluate over JailbreakBench (Chao et al., 2024), a dataset containing 100 harmful instructions, spanning 10 categories: harassment/discrimination, malware/hacking, physical harm, economic harm, fraud/deception, disinformation, sexual/adult content, privacy, expert advice, government decision-making.

In §4, we evaluate over the HarmBench test set (Mazeika et al., 2024). We consider only the 159 “standard behaviors”, and exclude instructions that require context or concern copyright violations. These harmful instructions span 6 categories: cybercrime & unauthorized intrusion, chemical & biological weapons/drugs, misinformation & disinformation, harassment & bullying, illegal activities, general harm.

Note that we perform filtering to ensure that $\mathcal{D}_{\text{harmful}}^{\text{(train)}}$, $\mathcal{D}_{\text{harmful}}^{\text{(val)}}$, and the two evaluation datasets are all pairwise disjoint, containing no overlapping instructions.

To construct the harmless datasets, we sample instructions from Alpaca (Taori et al., 2023). $\mathcal{D}_{\text{harmless}}^{\text{(train)}}$ contains 128 instructions, and $\mathcal{D}_{\text{harmless}}^{\text{(val)}}$ contains 32 instructions.

In §3, we evaluate over 100 instructions from Alpaca.

Note that $\mathcal{D}_{\text{harmless}}^{\text{(train)}}$, $\mathcal{D}_{\text{harmless}}^{\text{(val)}}$, and the evaluation dataset are all pairwise disjoint, containing no overlapping instructions.

Given a set of difference-in-means vectors $\{\mathbf{r}_{i}^{(l)}|i\in I,l\in\lceil L\rceil\}$, we want to select the best vector $\mathbf{r}_{i^{*}}^{(l^{*})}$. For each vector $\mathbf{r}_{i}^{(l)}$, we compute the following:

We then select $\mathbf{r}_{i^{*}}^{(l^{*})}$ to be the direction with minimum bypass_score, subject to the following conditions:

Using the compute setup described in §N, this direction selection procedure takes about an hour to run for the largest models (72 billion parameters), and faster for smaller models.

We report details of direction selection for each model in Table 5, including the token position $i^{*}$ and layer $l^{*}$ from which the direction was sourced, along with the direction’s corresponding metrics.

Figure 11displays the bypass_score and induce_score of all candidate directions for Llama-3 8B Instruct.

We use the default chat template for each model family. All chat templates are displayed in Table 6.

For a model completion $\mathbf{c}$, $\texttt{refusal\_score}(\mathbf{c})=1$ if $\mathbf{c}$ contains at least one “refusal substring”; otherwise $\texttt{refusal\_score}(\mathbf{c})=0$.

Figure 12displays the “refusal substrings” that are used in the refusal_score evaluation. Note that the substring comparison is not sensitive to capitalization, and checks that the phrases are contained anywhere in the completion, not just at the start.

For a model completion $\mathbf{c}$, $\texttt{safety\_score}(\mathbf{c})=1$ if Meta Llama Guard 2 (Team, 2024) outputs that the completion is safe; otherwise $\texttt{safety\_score}(\mathbf{c})=0$.

Figure 13displays the template used to prompt Meta Llama Guard 2. Note that we use the default prompt specified by the model providers, without any custom alteration.

Assessing whether a completion constitutes a successful jailbreak is complex. In this subsection, we highlight scenarios that are ambiguous, motivating our use of two metrics.

Figure 14and Figure 15 display cases in which the model does not explicitly refuse, but also does not provide a harmful response. In these scenarios, refusal_score=0 while safety_score=1.

Figure 16displays a case in which the model initially refuses, but then goes on to give a harmful response. In these scenarios, refusal_score=1 while safety_score=0.

In Figure 1 and Figure 3, we display error bars corresponding to standard error (SE), computed as $\text{SE}=\sqrt{\frac{p(1-p)}{n}}$. In both cases, $n=100$.

To compare the effectiveness of our jailbreak method to other methods in the literature, we report the 5 top performing jailbreak attacks, as ranked by HarmBench attack success rate (ASR) on the Llama-2 model family: GCG is the algorithm from Zou et al. (2023b) which optimizes an adversarial suffix for each prompt and each model; GCG-M is the multi-prompt version of GCG, trained over multiple prompts for each model; GCG-T is the transferable version of GCG-M, trained over multiple prompts and across multiple models; AP is the AutoPrompt approach from Shin et al. (2020); PAIR is a black-box method from Chao et al. (2023). We also include comparisons to the set of human-written “Do Anything Now” adversarial templates from Shen et al. (2023) (Human), and the “direct response” baseline without any jailbreaks in place (DR).

Table 2shows that the attack success rate (ASR) of our weight orthogonalization methodology is sensitive to system prompts. Including the default system prompt in evaluation substantially reduces HarmBench ASR for Llama-2 models, while having minimal effect on Qwen models.

As shown in Figure 17, the Llama-2 default system prompt includes explicit guidelines to avoid harmful or inappropriate content. In contrast, the Qwen default system prompt, shown in Figure 18, is minimal and lacks specific directives about safety or ethics.

We initially hypothesized that the discrepancy in ASR might be due to the Llama-2 system prompt’s explicit guidelines, making it particularly effective at enforcing safety guardrails. To test this, we applied the Llama-2 system prompt to Qwen models, but observed no significant change in ASR, as shown in Table 7. This suggests that the impact of system prompts on ASR may be model-dependent.

To investigate Llama-2’s responsiveness to system prompts further, we evaluated the same orthogonalized Llama-2 7B model with 12 diverse system prompts (shown in Figure 19). This evaluation revealed substantial variability in ASR scores for Llama-2 7B, with a mean of 30.0% and standard deviation of 23.3%, while Qwen 7B maintained a more consistent performance across the same set of prompts, with a mean of 76.7% and standard deviation of 5.9%. These findings suggest that Llama-2 models may be more responsive to system-level instructions than other model families, particularly with respect to safety-related behaviors. A more systematic study would be necessary to fully understand these differences, as well as the broader relationship between system prompts and refusal behaviors, which we leave as directions for future research.

Except on TruthfulQA, orthogonalization has a very small effect on general performance benchmarks. We observe less than 1% performance drop on average, with the difference to the baseline performance being indistinguishable from noise in most cases. The main exceptions are Qwen 7B, which has statistically significant drops on MMLU and GSM8K, and Yi 34B with drops on MMLU and Winogrande.

For MMLU, we use the default settings from LM Evaluation Harness (Gao et al., 2023; Biderman et al., 2024) as of May 2024. For the other benchmarks, we use the default settings, with the exception that models are run using vLLM (Kwon et al., 2023).

TinyHellaSwag from TinyBenchmarks (Polo et al., 2024) is a statistically informative 400-sample subset of the larger HellaSwag (Zellers et al., 2019) test set. Polo et al. (2024) claim a 2% average error compared to the full-sized counterparts.

TruthfulQA measures the performance of language models in generating truthful and accurate responses, particularly in areas prone to human misconceptions and falsehoods. Table 8 displays clearly that TruthfulQA performance is consistently worse for orthogonalized models as compared with unmodified models. TruthfulQA contains questions that touch on sensitive topics such as misinformation, stereotypes, and conspiracies. For such questions, models with and without safety guardrails may understandably generate different responses. Figure 20 displays an example of a conspiracy-flavored question from TruthfulQA, and the contrasting responses from Llama-3 8B Instruct and its orthogonalized version.

In addition to standard language model evaluations, we also check changes in cross-entropy (CE) loss over various datasets. For each chat model and its orthogonalized version we compute CE loss over a sample of The Pile (Min et al., 2023). The Pile consists of scraped webtext, and so we do not append any chat template when evaluating CE loss.

We note that some chat models are especially sensitive to chat templates, and behave poorly without them. Thus, we also evaluate over Alpaca (Taori et al., 2023), which is a chat dataset consisting of instructions and completions. We format each instruction according to each model’s chat template, and compute CE loss only over the completion tokens.

We further note that some chat models, seemingly Gemma 7B IT in particular, have high CE loss on text that is out of distribution, e.g. completions from Alpaca. To account for this, we take each baseline model, and generate completions on 100 instructions from Alpaca. We then compute CE loss over these “on-distribution” completion tokens.

All CE loss values are reported in Table 9. We denote the orthogonalized model as “Ablation”. We also compare to activation addition methodology, labeled “Act Add”, where rather than ablating the refusal direction, we subtract the difference-in-means vector. See §I.1 for a more detailed discussion of bypassing refusal via activation addition.

Using a custom implementation of Greedy Coordinate Gradient (GCG) (Zou et al., 2023b), we generated 100 adversarial suffixes of token-length 20, each of which was optimized for a particular behavior from AdvBench.

Of these 100 suffixes, we found one suffix in particular that performs well across a wide range of harmful prompts. The suffix is shown in Figure 21.

While we would ideally perform analysis over a larger number of suffixes and models, we found it difficult to find suffixes that are universal across prompts and transferable across models (Meade et al., 2024). We therefore restrict our analysis to a single model, Qwen 1.8B Chat, and a single suffix.

In Figure 5, for each layer and scenario, we display the standard deviation (SD) of cosine similarities across 128 prompts, computed as $\text{SD}=\sqrt{\frac{\sum{(x_{i}-\overline{x})^{2}}}{n}}$. In this case, $n=128$.

In §2.4, we described how to induce refusal using activation addition. Given a difference-in-means vector $\mathbf{r}^{(l)}\in\mathbb{R}^{d_{\text{model}}}$ extracted from layer $l$, we can add this vector at layer $l$ in order to shift activations towards refusal (Equation 3). Similarly, we can subtract this vector at layer $l$ in order to shift activations away from refusal:

We perform this intervention at all token positions. Note that this intervention can be implemented as a direct weight modification by subtracting $\mathbf{r}^{(l)}$ from the bias term of $\texttt{MLP}^{(l-1)}$.

As shown in Figure 23, this activation addition intervention is effective in bypassing refusal. The decreases in refusal score and safety score are comparable to those achieved by directional ablation (Figure 1). However, Table 9 displays that the activation addition intervention, labeled as act add, causes increased loss over harmless data, in particular compared to directional ablation.

Figure 22displays a visualization of activation addition in the negative refusal direction, and suggests an intuitive explanation of the intervention’s behavior on harmful and harmless prompts. On harmful inputs, adding the negative refusal direction shifts the harmful activations towards harmless activations, with respect to projection onto the refusal direction. With low projection onto the refusal direction, this intervention leads to low rates of refusal. However, on harmless inputs, adding the negative refusal direction shifts the harmless activations off distribution, resulting in increased perplexity.

Note that, in comparison to activation addition, directional ablation shifts harmful activations towards harmless activations, while also not shifting harmless activations too far off distribution.

Prior work has established that fine-tuning is effective in removing safety guardrails of chat models (Yang et al., 2023; Lermen et al., 2023; Zhan et al., 2023).

We replicate this result by fine-tuning Llama-3 8B Instruct. First, we construct a dataset of harmful instruction-completion pairs. For the harmful instructions, we sample instructions from AdvBench, MaliciousInstruct, TDC2023, and HarmBench. To generate corresponding harmful completions, we use Mistral 7B Instruct (Jiang et al., 2023), a competent chat model with low refusal rates. For each harmful instruction, we generate 5 completions, and then select a single completion satisfying both refusal_score=0 and safety_score=0. If no completions satisfy this condition, then the instruction is discarded. After this filtering, we were left with a dataset of 243 harmful instruction-completion pairs.

We then fine-tuned Llama-3 8B Instruct on the constructed dataset, applying LoRA (Hu et al., 2021) with rank=16 and alpha=32 for 4 epochs. The LoRA fine-tuning was performed on an A100 GPU with 80GB of VRAM, and took approximately 10 minutes.

Evaluations of refusal and CE loss are displayed in Table 10. In accordance with prior work, we confirm fine-tuning to be very effective in disabling refusal. We speculate that the decrease in CE loss over Alpaca could be due to the similarity between the distributions of Mistral Instruct completions and Alpaca completions, and as a result, fine-tuning over Mistral Instruct completions leads to a decreased CE loss over Alpaca completions.

We note that, although the LoRA fine-tuning process itself is straightforward and efficient, creating a high-quality dataset of harmful instruction-completion pairs requires non-trivial effort. In comparison, directional ablation (and its equivalent implementation via weight orthogonalization) requires just a dataset of harmful instructions, without the need for any harmful completions.

One possible way to prevent a model from refusing is to directly block it from outputting any of the standard refusal strings, such as "Sorry, I cannot", or "As a language model". Experiments shown in Figure 27 show that this is not the case for our weight orthogonalization methodology, as the orthogonalized model is able to generate the same strings that the unmodified model uses to refuse a harmful request. This suggests that the modification works at a higher conceptual level, rather than at the level of suppressing output tokens.

In general, we notice that the orthogonalized model behaves very similarly to the non-modified model on harmless instructions. Figure 28 displays completions on a random sample of harmless prompts from Alpaca. Generations from the unmodified model and the orthogonalized model appear to be indistinguishable, and often the generations are exactly the same.

As our intervention targets refusal directly, it is natural to ask how the resulting model answers meta-questions about its new refusal behavior: does it understand its behavioral modification, or does it default to explaining its original refusal behavior? As shown in Figure 29, the new model seems likely to answer meta-refusal questions consistently with its previous refusal behavior. However, its explanations seem not to be coherent.