Tülu 3: Pushing Frontiers in Open Language Model Post-Training

We release all artifacts associated with the Tülu 3 training recipe – including SFT, DPO, and RL model checkpoints, along with new SFT and DPO datasets. A summary of the artifacts released with Tülu 3 is included in Table 1.

Since the release of our Tülu 2, the community has witnessed a large body of work creating datasets for post-training, in terms of both supervised finetuning and preference tuning. Tülu 3 aims to integrate and extend these resources to build stronger models. We start this process with a broad survey of public datasets, including those annotated by dedicated workers, sourced from real users, and synthesized with models. $^{4}$ We then manually review each individual dataset, and pick those with the following considerations.

To address the growing need for diverse and skill-specific datasets, we incorporate synthetic data generation as a complementary approach. Synthetic data generation has gained traction as a promising alternative to human-written data due to being cheaper to obtain, customizable for different purposes, and reflecting the vast knowledge of the underlying models (Dubey et al., 2024). However, generating diverse and high-quality data at scale is non-trivial, as LMs are susceptible to falling into repetitive modes or patterns, referred to as “mode collapse” (Kazdan et al., 2024). To ensure diversity in generation, we follow the recent persona-driven methodology in Chan et al. (2024) to generate synthetic data. The key idea is to use different personas (e.g., “A machine learning researcher focused on neural networks”) with a data synthesis prompt (e.g., “create a coding problem”) to steer an LLM to synthesize data with corresponding perspectives. Specifically, we condition on $\sim$250K personas from Persona Hub (Chan et al., 2024) to generate prompts targeting specific skills such as precise instruction following, math and coding. We detail our procedure for each select skill below. Prompts used to generate these instructions can be found in Appendix C.1. Additionally, we build upon our previous efforts in Brahman et al. (2024); Han et al. (2024); Jiang et al. (2024), to generate noncompliance and safety data.

Precise Instruction Following. Precise instruction following is the ability to follow verifiable instructions in natural language, such as “your answer should contain exactly 3 paragraphs,” that can be automatically verified with heuristics. We use our persona-driven approach to synthetically generate verifiable instructions covering 25 different constraint types defined in IFEval benchmark (Zhou et al., 2023). More concretely, we start by manually writing 1-2 example instructions per constraint (e.g., number of words), resulting in total of 33 verifiable instructions which we used as seed prompts. We then generate new instructions using GPT-4o (OpenAI, 2024)$^{6}$ given a data synthesis prompt, persona, and a single verifiable instruction as an example. Figures 30 and 31 show the exact prompts used to generate the instruction and its corresponding response, respectively. In total, we collected 29,980 verifiable instruction-response pairs which we call If-Persona-Sft. Lastly, we also generate another type of prompts targeted for constrained instruction following by randomly sampling instructions from the Tülu 2 SFT mix and combining them with constraints from the taxonomy in Zhou et al. (2023). We call that set IF-augmented. These prompts are only used for the DPO and RLVR stages.

Math and Coding. We follow a similar persona-driven approach to synthetically generate diverse math word and coding problems. Math problems include those that require advanced mathematical skills as well as grade school problems. For coding, we generate Python programming questions that are solvable by entry- to medium-level programmers. Unlike precise instruction following, we zero-shot prompt GPT-4o to generate problems that are unique and specific to a given persona input. Having generated the problems, we then generate multi-step math solutions using GPT-4o, and Python programs using claude-3-5-sonnet. Exact prompts used to generate problems and solutions are provided in Figures 33, 35, 34, and 36, respectively. In total, we collected $\sim$220K and 35K instances for math reasoning and coding.

Noncompliance and Safety. As we enhance models’ capabilities to assist users effectively, it is crucial to ensure they can reliably reject unsafe and appropriately handle nuanced and out of scope queries. To support this, we curate a set of noncompliance queries (Brahman et al., 2024) that the model ought to not comply with, alongside safety-related direct and adversarial prompts (Han et al., 2024; Jiang et al., 2024) covering both benign and harmful scenarios. Our noncompliance and safety prompts are either curated from existing datasets (Zhang and Choi, 2021; Zhao et al., 2024) or synthetically generated from the GPT model family. More specifically, our noncompliance prompts are obtained based on our contextual noncompliance taxonomy from Brahman et al. (2024), spanning multiple categories including incomplete, unsupported, indeterminate, and humanizing requests (in addition to unsafe requests). Our safety-related prompts are carefully selected among synthetic adversarial prompts, synthetic vanilla (direct) requests, real-world user-LLM interactions (In-The-Wild), and curated annotator-written examples to maximize coverage, diversity, and balance.

To create our SFT mix, we collect or create responses for prompts described in Section 3 in two ways: filtering existing responses, and creating new responses.

For prompts with existing responses, we generally keep the original response if it was written by a human or a frontier model, like GPT-4o. For large datasets with subsets from frontier models (e.g. WildChat), we use the subset from the best models. We additionally filter empty responses and responses that contain information about models or their developers. If a set of prompts did not have responses, like our Persona prompts, or if the original responses were from a weaker model (e.g. WildGuardMix), we generate new responses using GPT-4o. We also hand-wrote responses to our hardcoded prompts.

To develop our SFT mix, we first identified the skills that were lagging behind state of the art models using Llama 3.1 trained on Tülu 2$^{7}$ as our baseline. Targeting each of these skills in isolation, we collected high quality publicly available datasets and created synthetic datasets, as described in Section 3.1.2, and also removed some datasets that we identified to be of relatively lower quality compared to other more recent datasets.

To design our final SFT mix, we first built skill-specific data mixtures and models, keeping the mixtures that led to the best performance on individual skills, ignoring other evaluations. This was done to approximate the upper bound for each evaluation given our setup.

We then combined these mixtures to create our initial Tülu 3 preview mix. We then continued to iterate on the mixture by adding or removing datasets to improve lagging skills, decontaminating against our evaluations and downsampling particularly large datasets. We show the performance of major preview versions throughout development in Figure 3.

In our mix we also emphasized adding diverse chat data, mainly from WildChat. We show the impact of removing WildChat in Table 10, and we see that there is a small but noticeable degradation on most skills, most noticeably on Alpaca Eval, highlighting the importance of diverse real-world data.

We found that our safety SFT data was generally orthogonal to our other datasets. We report the effect of removing our safety-specific datasets in Table 10, and we see that most skills stayed roughly the same, except the safety average. We also found that adding contrastive prompts, such as those in CoCoNot, were helpful for preventing our models from over-refusing safe prompts.

Our new Persona datasets were built to target specific skills: mathematics, coding, and instruction following. In Table 10 we show that performance on HumanEval(+), GSM8K, MATH, and IFEval drop after removing our Persona datasets, showing the value of creating diverse, skill-specific SFT datasets.

A large portion of our focus was on collecting or creating datasets targeting specific capabilities. Using mathematical reasoning as an illustrative example, we show in Table 10 the impact of our mathematics-specific data on both GSM8K and MATH. We see that our mathematics-specific SFT data substantially improves both GSM8K and MATH, showing the value of the data included in the final mix.

In Figure 4, we show the effect of taking stratified subsamples of our SFT mix. We find that our models continue to improve on average as more SFT data is included, and we see large improvements on metrics like GSM8K as we increase the amount of data to the full mix. Interestingly, TruthfulQA performance actually drops as the amount of data in the mix increases. We do not increase our SFT data size beyond the current mixture because we allocated other prompts for preference optimization.

To train our Tülu 3 models, we used between 4 and 16 8xH100 nodes with high speed interconnect. The final 8B model is trained on 32 GPUs for 6 hours and the 70B model was trained on 64 GPUs for 50 hours. We used an effective batch size of 128 and a maximum sequence length of 4,096 tokens. We trained for two epochs using a learning rate of 5e-6 for our 8B models, and 2e-6 for our 70B models, which we found after a hyperparameter search. Our hyperparameter settings are also summarized in Table 11. For merging experiments we used mergekit$^{8}$ (Goddard et al., 2024), using linear weighted averaging.

Early during training Tülu 3, we noticed a gap in performance between SFT models trained on our Open-Instruct framework and models trained in other settings such as on TPUs.$^{9}$ We found this issue was largely due to a (recently widely-reported) issue with loss aggregation inside Transformers (Wolf et al., 2020): Averaging the loss across padding tokens without taking into account gradient accumulation or distributed training setups.

Here, we illustrate the issue with an example. Assume we have two samples in a batch, with $n_{1}$, $n_{2}$ non-padding tokens and $m_{1}$, $m_{2}$ padding tokens. If we pass both samples into the default Transformers forward pass at the same time, we get:

However, if we apply gradient accumulation, feeding in the two samples separately, computing loss, and then dividing, our loss is instead computed like:

That is, in the second case we weight each example equally, while in the first we weight each token equally. As such, changing gradient accumulation can have large effects on performance due to effectively changing sample weightings, as reported by Muennighoff et al. (2024). A similar issue occurs in distributed training due to cross-device averaging. We refer to recent reports on this issue for a more in-depth explanation.$^{10}$

To fix this issue, we opted generally to use a sum loss instead of averaging (‘mean loss’) when training. This removes the issue by simply removing the denominator from the above equations and requires an adjustment to learning rates. This effectively weights all tokens equally (which we found led to generally better performance for initial mixtures). We validated the perfomance of our setup by finetuning Llama 3.0 on the Tülu 2 SFT mixture using a variety of learning rates, epochs, and loss types as shown in Figures 5 and 6. Ultimately, we found that using a sum loss with a learning rate of 5.00E-06 worked best. Surprisingly, we additionally found that training for longer did not yield further improvements, and so used 2 epochs for training.

Preference Data. In the standard setup, there is some preference dataset $\mathcal{D}$ consisting of prompts $x$ and two responses $y,y^{\prime}$ per prompt. Some judge(s) will choose one of $y,y^{\prime}$ as their preferred response $y_{c}$, and label the other as a rejected response $y_{r}$.

Reward Model. Given the preference dataset, a reward model (RM) $r_{\phi}$ is trained with the following objective:

where $\sigma$ is the logistic function. The RM objective maximizes the difference between the rewards, and this difference represents the log-likelihood that $y_{c}$ will be preferred over $y_{r}$ (Ouyang et al., 2022). This reward model can help train policy models to output contents preferred by the RM’s judgments.

There are a plethora of options for optimizing language models with access to preference data. Today, the two categories can be abstracted as reinforcement learning algorithms, which learn from an internal representation of value or reward, and direct alignment algorithms, which learn directly from the data.

Prior work (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022) optimizes the policy $\pi_{\theta}$ with the following objective:

where $\pi_{\text{ref}}$ is the initial reference policy and the $\beta$ coefficient helps control the Kullback-Lieber divergence (KL) divergence between the reference policy and the training policy. Here, we explain PPO and DPO as representative examples.

Proximal Policy Optimization (PPO). An approach to address the above objective is to use online reinforcement learning (RL) like PPO (Schulman et al., 2017). In each training iteration of PPO, the policy needs to generate some samples, generate rewards using the RM on those samples, and maximize $R(x,y)$ using the PPO algorithm. As PPO training loops are complex, we refer the reader to Ouyang et al. (2022); Ivison et al. (2024); Huang et al. (2024a) for more thorough descriptions of the setup and typical setups. We provide more implementation details in Sec 6.2.

Direct Preference Tuning (DPO) and Variants. Another approach is offline preference tuning. DPO (Rafailov et al., 2024) can directly optimizes for the RLHF objective with the following equivalent objective:

DPO trains an implicit reward model and a policy model simultaneously, without needing to use a trained reward model, do policy generations, and get rewards from the RM. Crucially, this allows offline preference finetuning, directly training a language model on preference pairs gathered from a variety of sources. Recently, much work has examined how to further improve the DPO objective, with a multitude of variants proposed (Meng et al., 2024; Xu et al., 2024a; Hong et al., 2024, inter alia). In this work, we explored two promising variants: SimPO (Meng et al., 2024) and length-normalized DPO$^{12}$. We find (in Section 5.4) that length-normalized DPO works best, which uses the following objective:

As seen, this is simply the DPO objective (Eq 5), but with log-probabilities normalized for length, which intuitively aids with mitigating the length bias common in human and model preferences (Singhal et al., 2024).

When developing Tülu 3, we opted to use length-normalized DPO for tuning our preference data mixtures and generation methods due to its relative simplicity and speed compared to approaches such as PPO.

We create on-policy preference data $(x,y,y^{\prime},label)$ given our prompts from §3 by adapting and advancing the UltraFeedback pipeline (Cui et al., 2023). Our early experiments show the benefit of this pipeline in creating preference data, which leads to a high-quality, synthetic preference dataset (as observed by Ivison et al. (2024)). Our data creation pipeline (shown in Figure 7) consists of three stages: prompt selection, response generation from a pool of models, and preference annotation with LLM-as-a-judge to create (preferred, rejected) pairs.

We choose the final preference mix for the 8B and the 70B model, which maximizes average performance on the development evaluations, while also exceling at targeted skills. Most of the preference data mix ablations are run for the 8B model, We start with prompts used for SFT and generate on-policy and off-policy preference data, resulting in 96,911 (off-policy) and 19,444 (on-policy) preference instances. Given this preference base we ablate adding additional prompt sources to the mix and how these additions affect downstream evaluation performance, specifically targeting skills like precise instruction following, math and general chat performance on AlpacaEval. Table 16 shows how the inclusion or exclusion of preference datasets influences the average performance. Our final mixes for Tülu 3 8B DPO and Tülu 3 70B DPO are displayed in Table 15. In summary, our preference mixes come from different prompt sources, such as SFT data, WildChat and Persona IF. It includes prompts seen during SFT training but also new, unseen prompts.

First, we investigate whether increasing the number of prompts will yield improvements in downstream DPO performance. To do so, we measure the downstream DPO model performance at different sizes of a fixed amount of preferences with unique prompts. Figure 8 shows that there are noticeable performance gains across several metrics as the size of the preference dataset increases. This suggests that dataset scaling is important to achieve improvements in downstream model performance: our final preference mixes (Table 15) contain more than 270k data points for the 8B model and more than 330k instances for the 70B model, which is considerably bigger than many available preference datasets.

We also explore whether duplicating prompts, i.e. same prompts with different responses, is a viable approach to scaling the size of a preference dataset and whether it will lead to gains in downstream DPO performance. To do so, we expanded the Ultrafeedback dataset, which originally had four responses for each prompt, by creating additional pair combinations of responses. This expansion will naturally cause duplicated prompts, but with different chosen and rejected pairs sampled from the four responses in UltraFeedback, leading to preference datasets with 64k-, 180k-, and 383k instances. Figure 9 shows that, on average, the 383k-size preference dataset performs similarly to the 64k preference dataset. We also observe a slight performance degradation on DROP, GSM8k, and AlpacaEval as the number of duplicated prompts increase. This suggests that scaling via prompt duplication does not necessarily yield into significant gains in downstream DPO performance, and investing in the collection of unique prompts and proper mixing is more important for downstream evaluations.

We then compare including new prompts and re-using prompts from the SFT stage on their effect on downstream DPO performance. To do so, we sampled 100k prompts from the SFT dataset mix that were used during training (as shown in Table 7) and compare it against prompts from the same open datasets (e.g., OpenAssistant, SciRIFF, Aya, Persona, WildChat, etc.) we subsampled from but left unused during SFT. Figure 10 shows that the unused dataset has a slightly higher performance as opposed to reusing prompts. This suggests that the presence of new prompts can help improve downstream DPO performance. Though, as seen in our best mix, combining unused and reused prompts seems to lead to the best result.

We investigate whether the inclusion of on-policy data, i.e., text generations from the SFT model that will be used as the base model for preference finetuning, improves downstream model performance. Given the same set of prompts sourced from the SFT mix in §4, we generate preferences from off-policy models and compared it to a mix that is strictly on-policy (i.e., one of the response is always from the Initial 8B SFT model, and the other response is from the off-policy models). We also compare it on a combination of both on-policy and off-policy data: we sample instances from the strict on-policy dataset and add it to the off-policy dataset so that the responses from each model is distributed equally. Figure 11 shows that including on-policy data improves aggregated downstream DPO performance compared to a completely off-policy dataset where prompt completions were sampled from other models.

In order to determine which judge to use for obtaining preference annotations, we test several commercial and open-source LLM judges such as GPT-4 (GPT-4-turbo-2024-04-09, GPT-4o-2024-08-06, gpt-4o-mini-2024-07-18) and Llama 3.1 (70B and 405B) on the same set of 10k randomly-sampled Ultrafeedback prompts and responses. In general, GPT-4o, Llama 3.1 405B, and GPT-4 Turbo perform similarly across all benchmarks, with GPT-4o leading slightly ahead on the aggregated average performance as shown in Table 17. In the synthetic preference pipeline for Tülu 3, we opted for GPT-4o-2024-08-06 due to its ease-of-use, cheaper cost per request, and batch inference speed via OpenAI’s Batch API.$^{14}$

Previous work on preference learning using openly available datasets has shown that the UltraFeedback (Cui et al., 2023) preference dataset generally outperforms other preference datasets (Ivison et al., 2023). In Figure 12 we show that we were able to significantly surpass DPO training on UltraFeedback by training on our best mix. The improvement is greater for the 70B model (+3.3 vs. +1.8), we hypothesize that this is because UltraFeedback’s completions are mainly sourced from models that are less capable than the 70B model we are starting with. Helpsteer2 Wang et al. (2024d), another high-quality preference dataset, also performs lower than our best mix on the 8B model.

From the three persona preference datasets targeting instruction following, coding and math skills, only Tülu 3 Persona IF improves the average eval score and the targeted IFEval score (see Figure 13). Neither Tülu 3 Persona Math nor Tülu 3 Persona Code improve their respective targeted evaluations and slightly harm the average score. We therefore only include the Tülu 3 Persona IF preferences in our final mix.

We created preference data targeted to improve a model’s precise instruction following skills.

For IF-augmented, we run two analyses. We generate an additional set of more than 66k instances and we then run the chosen completions through constraint verifier functions, and only add those instances to the final set which actually fulfilled the constraint(s). This leaves us with a cleaned set of about 26k preferences, which we call IF-augmented-verified. In Figure 15 we show that the IF-persona preferences significantly improve IFEval scores beyond the baseline mix, while minimally harming average performance. The IF-augmented-verified dataset improves IFEval performance only by 1 point, while also slightly harming the average performance. Combining IF-persona with IF-augmented-verified leads to the best IFEval performance, but to a slightly lower average. We therefore choose to include IF-augmented (not verified) and Persona IF in the final 8B DPO mix, which leads to both a satisfiying average and IFEval score.

Our ablations show that adding preference data consisting of WildChat prompts and chosen/rejected pairs obtained using our synthetic preference data pipeline, generally improves DPO performance. Ablations in Figure 5.2.2 reveal that adding WildChat prompts seen during SFT training to the DPO mix leads to better average performance than combining the unused with the reused WildChat prompts.

We also investigate whether the preference dataset generated by the synthetic pipeline in §5.2.1 can yield to gains in downstream DPO performance on existing datasets. To do so, we take the prompts from open-source datasets such as Helpsteer2, Ultrafeedback, and MultiPref (Miranda et al., 2024), then regenerate their completions and preference annotations using the synthetic data pipeline. Figure 15 shows that the downstream DPO performance of the regenerated dataset is better than the original dataset, suggesting that the synthetic pipeline itself can yield to performance gains.

In light of the significant amount of work on improving DPO and related algorithms since the release of Tülu 2, we revisited our hyperparameter and algorithm choices alongside our preference datasets. We ablated both algorithm and hyperparameter choices using an early SFT checkpoint and the UltraFeedback dataset. We explored using DPO, SimPO (Meng et al., 2024), and length-normalized DPO. Our results are shown in Table 18. We found that only length-normalized DPO outperformed our base checkpoint overall, and so further tuned it, resulting in the final hyperparameters shown in Table 20.

We lowered the learning rate and increased the batch size for the 70B training based on the fact that it is common to lower the learning rate and increase batch size when doing SFT with larger models (Touvron et al., 2023).

The 8B DPO model is trained for 10 hours on 8 Nvidia H100 GPUs and the 70B DPO model is trained for 19 hours on 64 interconnected H100s.

The DPO training uses a maximum sequence length of 2048.

To run the 70B DPO training, we found it useful to implement two key optimizations for reducing the GPU footprint of DPO training:

We empirically validated these two techniques on the Llama 3.1 model and found they resulted in near identical training losses. As expected, the model uses less GPU memory when using the two techniques on an 8xH100, as shown in Figure 17.

We use the GSM8k training set. We augment each sample with the standard 8-shot prompt using during evaluation to encourage the model to use chain-of-thought, and then extract the final number produced and compare to the ground-truth label to determine correctness.

We use the MATH training set. Similar to GSM8k, we augment each sample with the standard 3-shot CoT prompt used to encourage the model to generate chains of thought during evaluation, and then extract the answer and determine correctness following the ‘flex’ MATH evaluation logic.

We randomly sample instructions from the Tülu 2 SFT mix and combine them with constraints from the taxonomy in Zhou et al. (2023). We have a verification function for each of the constraint templates that is able to verify whether a completion satisfies a constraint.

Given these prompts and verification functions, we then train the models via Proximal Policy Optimization (PPO) (Schulman et al., 2017) on these verifiable rewards. We combine all prompts together, results in a mixture of roughly 30,000 prompts with ground truth labels. We summarize our verifiable prompt mixture in Table 22.

RLVR Can Improve Performance in Targeted Domains. Figure 19 shows training with RLVR results in improved test performance for all three settings. In all cases, we achieve models that outperform the initial model in that particular evaluation. We also see that the verifiable rewards (i.e., correctness on the train set) improves consistently for all three settings. Interestingly, in GSM8K and MATH, we found that incurring more KL budget does not necessarily lead to improvements in verifiable rewards.

Initializing RLVR’s Value Function from a General RM Works Best. Figure 22 shows that initializing the value from a general RM obtains the highest GSM8K test score and also higher average scores. This suggests the value function plays an important role in RLVR’s training.

Do Not Use the Scores from RM. Figure 22 shows that using only the verifiable rewards outperforms using scores from the reward model. Training with verifiable rewards with the scores from RM seems to introduce more noise, especially in the average scores.

Starting from a Weaker Model Can Converge to the Same Verifiable Rewards. Figure 20 shows that starting from both SFT and DPO can lead to the same level of verifiable rewards, but starting from the SFT model would incur a larger KL compared to starting from the DPO model. This makes sense because the SFT model is further away from good at GSM8K than the DPO model. However, we find that starting from a stronger model usually results in better test set performance.

Overoptimization Happens. As we lower the KL penalty $\beta$, the trained model incurs more KL from the initial model. We observed that more KL divergence typically results in lower average scores, as shown in Figure 22. The exception is Figure 22, where the largest KL corresponds to the highest average score. Furthermore, we showcase overoptimization of prompts with constraints cases in Appendix B.4.

is heterogeneous with regard to the type of reasoning skills required to answer the questions, containing instances that require basic factual recall as well as those that demand logical reasoning and problem-solving skills. We design a zero-shot CoT setting that asks the models to “summarize” its reasoning before answering the questions (see Table 44 for the prompt used). We find that among the various CoT settings, including prompting the model to think “step-by-step” and using no CoT, the prompt yields a systematic performance improvement over standard 5-shot multiple choice setting across tested models (See Table 46) and maximizes on the number of subject (knowledge categories) that the CoT benefits. This indicates that our “summarize” prompt is an effective strategy for dealing with the heterogeneity of the benchmark. See appendix F.2 for details. We compute a macro average over all the subjects in MMLU as the final task metric.

is an entity-centric question-answering benchmark that evaluates language models’ tendency to forget information about long-tail entities. We prompt the models in a 15-shot setting (as recommended in the dataset paper) without any additional instructions, with each QA demonstration presented in a different conversation turn, a setting that we refer to as Multiturn ICL henceforth, and use greedy sampling to obtain model predictions.

contains questions that humans tend to answer incorrectly due to common misconceptions. We use the multiple-choice version of this benchmark where the models being evaluated are presented with questions and options containing multiple correct answers (the MC2 setting).

evaluate models’ ability to complete Python code given docstrings. HumanEval+ uses a more rigorous evaluation procedure than the original HumanEval benchmark with additional tests. We use samples from the models at a temperature of 0.8 and use pass@10 as the evaluation metric.

contains grade school math word problems. We use the 8-shot chain-of-thought prompt from Wei et al. (2022b), formatted for a multiturn ICL evaluation. We obtain model responses using greedy sampling and extract the last number in the model response as the predicted answer.

contains problems from mathematics competitions spanning various categories such as algebra and calculus. We use a 4-shot multi-turn setup with CoT from Lewkowycz et al. (2022), formatted for multiturn ICL, and greedy sampling for model completions. For determining the correctness of predictions, we use a ‘flex’ scheme that attempts to extract the answer in three different ways: (1) following the minerva format (Lewkowycz et al., 2022); (2) finding the last instance of ‘<a​n​s>’; (3) taking the text between the last two ‘$’ tags. This is due to issues we found during development wherein models would often not follow the correct output format despite the few-shot examples, necessitating a varied answer extraction strategy. We find that moving from the minerva format alone to our ‘flex’ strategy can sometimes improve reported scores by up to 10 points, highlighting the need for this flexible strategy. We compute macro average across the subsections to obtain the final task metric.

contains challenging reasoning problems for which models benefit from step-by-step reasoning. We follow the setup described in the original paper and use 3-shot CoT prompts, formatted for multiturn ICL. We use greedy sampling for obtaining model predictions.

is a reading comprehension task that requires discrete reasoning. We draw 3 random few-shot examples from the train split following the setup used for Llama 3 (Dubey et al., 2024), and use greedy sampling to get model predictions.

evaluates instruction following ability of models in a setting where each instruction corresponds to constraints such that the it can be programmatically verified whether the outputs satisfy those constraints. We use greedy decoding to generate model outputs give the instructions in the dataset, and measure the prompt-level accuracy of satisfying constraints in the loose evaluation setting.

contains a set of prompts sourced that reflect real human usages of LMs, and compares model outputs to GPT-4 turbo responses with an additional length control to avoid longer answers being unfairly favoured. We generate responses using greedy decoding up to 8,192 tokens in length, following Ivison et al. (2023).

We follow Han et al. (2024) and Jiang et al. (2024) to define our safety evaluation suite using the following benchmarks.$^{20}$. Each of these benchmarks evaluates whether models refuse to respond to unsafe requests, and in the case of XSTest and WildJailbreak, additionally evaluate whether they comply with benign requests. We use greedy sampling to get model responses for the prompts in each of the benchmarks, and compute the accuracy (at refusal or compliance as appropriate). We report the macro average of the scores over all the benchmarks as our final safety evaluation metric.

includes the English language subset of the AGIEval benchmark, specifically these multiple-choice tasks: aqua-rat, logiqa-en, lsat-ar, lsat-lr, lsat-rc, sat-en, sat-math, and gaokao-english. We do not include the sat-en-without-passage task as we find these questions are typically severely underspecified without access to the passage. We formulate the task using the a simple “zero-shot CoT” prompt which encourages concise reasoning ending with a clearly stated answer choice. In Appendix F.2, Figure 44, we provide the 0-shot reasoning prompt, used for the multiple-choice tasks in the unseen evaluation suite, including AGIEval English. The model’s answer choice is extracted by first matching to the requested format, with fallback patterns if the format was not followed precisely. Specifically, we first look for the exact phrase indicated in the prompt (“Therefore, the answer is [ANSWER]”) and grab the last such match. If that fails, we look for a sequence of softer variants, like “answer is [ANSWER]” or “answer: [ANSWER]” before falling back to the last letter in parenthesis found, and if that fails, the last stand-alone capital letter.

is a 10-way multiple-choice extended version of the MMLU dataset. We use essentially the same prompt and answer extraction as used for our AGIEval setup, just adjusting for the number of answer choices. We generally find our formulation to be as effective as the traditional 5-shot CoT prompt (e.g., used in Llama 3.1 evaluations (Dubey et al., 2024)) despite being much shorter, more realistic, and easier for users to create (see Table 27 for a comparison across exploratory models).

is a set of very challenging multiple-choice questions written by domain experts in biology, physics, and chemistry. We use the same zero-shot prompt and answer extraction as for AGIEval. This is similar to the approach used in Llama 3.1 evaluations, but with less prescription on how the reasoning should be structured. Figure 28 shows that across exploratory models, that our approach leads to comparable scores.

is a dataset of 56 categories of math questions, testing mathematical and algebraic reasoning skills. We devised a “zero-shot CoT” prompt that sets the context of the task, explains the format expected for the answers (e.g., “x**2” for powers), and for each category included three example answers to specify the answer format. The prompt used and answer extraction heuristics were refined by examining example outputs from our exploratory models. As seen from Table 29, applying evaluation principles aligned with real usage (“chat” versions), often performs better than using a setup involving presenting few-shot examples adapted from base model evaluations (“base-adapted”). We provide the zero-shot reasoning prompt for the Deepmind Mathematics task in Appendix F.2, Figure 45. Aligned with the instructions for formatting in the prompt, when extracting the answer, we first look for an answer in the format “Therefore, the final answer is [answer]. I hope it is correct.” We look for this in a case-insensitive way, process the [answer] by stripping away any trailing period, and known math delimiters surrounding the answer (e.g., “$” ) based on outputs from our exploratory models. For answer comparison, we first compare the raw strings of the processed generated answer and gold answer to check if they are equal. In addition, we parse both using the SymPy (Meurer et al., 2017) package and compare the equivalence of the parsed outputs to check their mathematical equivalent. We also consider the generated answer to be correct if it is equal to the parsed gold answer after this parsing.

is a set of coding challenges. We focus on the “hard subset” of 148 (out of the total 1140) instances, using the “instruct” formulation of each task and the “calibrated” score. We follow the setup used for the original leaderboard for our implementation.

In order to test precise instruction following abilities of LLMs and whether they are able to follow constraints that go beyond the 25 constraints included in IFEval (Zhou et al., 2023), we developed IFEval-OOD (IFEval Out-of-Distribution). IFEval-OOD consists of 52 constraints across six broad categories such as the examples in Table 30. A complete list of the constraints can be found in Appendix F.3. A portion of the constraints were sourced by asking a group of researchers for constraint ideas and others were written by authors of this paper. One of the six categories (“custom”) comprises manually written verifiable prompts to cover specific sub-skills, such as CSV generation. For the remaining five categories, the verifiable constraint was combined with 10 unseen prompts from WildChat. To select the final prompts, these constrained prompts were human annotated for quality and compatibility with the constraint (e.g. a prompt to paraphrase a one-sentence reference text that doesn’t contain any names to begin with would not be compatible with a constraint to mention at least 23 different person names in the response). To ensure constraint coverage, additional unseen WildChat prompts were manually paired with constraints that did not have at least five selected prompts. Our evaluation dataset emphasizes broad coverage of constraint types to differentiate constraint-following abilities from overfitting on the existing IFEval constraints.

We constructed an automatic evaluation of instructability of language models, called Human Reference-guided Evaluation of instruction Following (HREF). HREF focuses on 11 instruction following tasks that language models are typically trained on, namely, Brainstorming, Open QA, Closed QA, Extraction, Generation, Rewriting, Summarization, Classification, Numerical Reasoning, Multi-document Synthesis, and Fact Checking. We obtained high quality human-written prompts and responses in these categories from professional instruction-tuning data creators, and developed a reliable automatic evaluation procedure. Following AlpacaFarm (Dubois et al., 2023), we use win-rate against a fixed baseline model as the evaluation metric. However, since we have a larger number of tasks in our benchmark than AlpacaFarm, and also because we have access to human-written references, we hypothesized that the setup used for win-rate computation may not be directly applicable to our new evaluation. So we experimented with various win-rate computation methods, and their details along multiple dimensions:

To make decisions about the evaluation setup, we collected human judgments comparing responses from a diverse set of 16 models, when prompted using the instructions taken from a subset of our evaluation dataset. We collected four human judgments per each model response pair, and compared the agreement of each evaluation setup with majority human judgments. We repeated this procedure for each task category to identify the best setup for that category.

Table 33 shows a comparison between Tülu 3 models, Llama 3.1 Instruct models, and Hermes 3 Llama 3.1 models at 8B and 70B sizes, and Table 48 shows a subtask-level breakdown of the performance of these models on HREF. It is important to note that while all these evaluations are unseen for the Tülu 3 models, we do not know if GPQA, MMLU-Pro, AGIEval, DeepMind Math, and BigCodeBench were used for developing the two other models. We summarize below our key takeaways from this comparison and from our qualitative analysis of the outputs of Tülu 3 models on these datasets:

Standard DPO methods use preference datasets that are usually collected ahead of time, often from a distinct language model, and are thus considered as offline. In other words, with DPO, the policy cannot obtain feedback over it own generations during training. This is in contrast to online methods like PPO where the RM provides online feedback to generations from the policy $\pi_{\theta}$ being trained. To mitigate the distributional shift issue, recent works proposed Online DPO (Guo et al., 2024) following a three-step process: (1) sample 2 responses to a prompt from the current policy; (2) obtain online feedback over the response pair to create pairwise data, and (3) use this pairwise data update the policy $\pi_{\theta}$ via standard DPO loss. While the original paper proposed using online AI feedback for step 2, to better scale our experiments, we obtain feedback from a trained reward model.

We tried online DPO both to enhance general, and target capabilities, i.e., mathematical reasoning. For general capabilities, we train an RM for 1 epoch using the 82K preference data points from the Skywork.$^{24}$ For targeting mathematical reasoning, we continue training the same RM on our synthetic on-policy math-specific preference data (described in §5.2.1). Training online DPO on top of one of our Tülu 3 DPO checkpoint for total of 200K episodes on math problems (prompts are taken from the same RM training data), resulted in no or little improvement on GSM8K and degradation on MATH performance (we experimented with various sampling temperatures and KL penalty coefficients). We did not further explore this approach extensively, as our initial results suggested limited gains in both general and targeted domains. Future work could investigate alternative training strategies, such as different sampling methods, or finetuning RM architectures, to better align the optimization process with the desired capabilities.

Rejection sampling for large language models is an increasingly popular method for improving post-training performance of frontier language models (Dubey et al., 2024; Adler et al., 2024; Dong et al., 2023). Using an initial SFT and preference data mix to train an initial model, that model is used to then generate n responses to each SFT prompt. These n responses, in addition to the original response, are then ranked using a reward model or an LLM as a judge, and the best response is kept. The other responses can then be used to create chosen/rejected pairs for preference optimization. The full post-training pipeline is then run on these datasets, and the process repeats until performance converges.

We tried rejection sampling, but found that for our setup the performance gains were minimal for the amount of compute required, and thus we leave a deeper exploration for future work. Qualitatively, we found that strong judges are vital, and publicly available models can struggle to choose the best response out of the candidates. We also found that including the original response as a choice for the judge (in other words, choosing the best response out of the n generations in addition to the original response) performed much better than only choosing from the newly generated responses.

Currently, the data collected for Tülu 3 is relatively short and does not contain long multi-turn data (the average number of turns in our mixture is 2.4 turns and majority of samples are under 2,048 tokens in length). However, long-context has been popular area of focus in recent work (Pawar et al., 2024), as improving the context window of LMs enables new use-cases (Gemini Team, 2024) and more in-context examples, potentially improving performance (Agarwal et al., 2024). Relatedly, improving multi-turn capabilities can better improve end-user experience, with a non-trivial number of real-world user conversations with LMs going over 2 turns (Zhao et al., 2024). We hope to address both skills in the future with dedicated training and evaluations.

We specifically focus on English data and evaluations for Tülu 3 (although we do include the multilingual Aya (Üstün et al., 2024) dataset due to its high quality). However, this neglects the myriad languages apart from English spoken around the world, speakers of which may benefit from or desire LMs that can process their languages. Future work may examine the current multilingual abilities of Tülu 3 and how to further improve them. We also note that multilingual post-training can make use of different techniques to monolingual post-training – for example, cross-lingual alignment (Wu et al., 2024) or careful data balancing strategies (Li et al., 2024b). This makes multilingual post-training an interesting and impactful area for future work.

While we evaluate Tülu 3 on its own, LMs are being increasingly deployed as parts of larger systems, in which they have access to tools (Qu et al., 2024) or are themselves part of a larger ‘agent’ framework.$^{25}$ Furthermore, training models to use tools is a natural way to dramatically improve their reasoning and mathematical skills (Gou et al., 2024), rather than trying to accomplish everything ‘in the weights.’ Future work involves training or evaluating Tülu 3 for tool-use either on its own or as part of a larger framework.