SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
arXiv:2502.18449v2 · This workspace stays on this version. Check arXiv for newer versions
Some source details need review 2 to check
Original formulas, table values and figures are preserved. Comparison can miss differences; use the original PDF to check details.
Reading view checked: 16 math expressions, 2 rendering failures; 2 of 2 images available.
Compared 17 of 17 source passages against 26 of 26 PDF pages.
The HTML source contains failed formatting commands; these were omitted from the reading view. The original sources are retained. (4)
- Some formulas could not be rendered; their original TeX is shown. Check the original PDF. (2)
PDF page 3 reads “More details are explained in §A.”, while HTML S2_p1_1 reads “More details are explained in .”; the PDF image visibly includes §A. Check PDF page 3
2 of 2 figure, table and graphic images captured. Open original PDF
Abstract
The recent DeepSeek-R1 release has demonstrated the immense potential of reinforcement learning (RL) in enhancing the general reasoning capabilities of large language models (LLMs). While DeepSeek-R1 and other follow-up work primarily focus on applying RL to competitive coding and math problems, this paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for real-world software engineering. Leveraging a lightweight rule-based reward (e.g., the similarity score between ground-truth and LLM-generated solutions), SWE-RL enables LLMs to autonomously recover a developer’s reasoning processes and solutions by learning from extensive open-source software evolution data—the record of entire software development cycles, including code snapshots, code changes, and events such as issues and pull requests. Trained on top of Llama 3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve rate on SWE-bench Verified—a human-verified collection of real-world GitHub issues. To our knowledge, this is the best performance reported for medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs like GPT-4o. Surprisingly, despite performing RL solely on software evolution data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For example, it shows improved results on five out-of-domain tasks, namely, function coding, library use, code reasoning, mathematics, and general language understanding, whereas a supervised-finetuning baseline even leads to performance degradation on average. Overall, SWE-RL opens up a new direction to improve the reasoning capabilities of LLMs through reinforcement learning on massive software engineering data.
The application of large language models (LLMs) to software engineering (SE) tasks has received significant attention, with researchers exploring their potential to automate various complex SE tasks, such as library-level and complex code generation bigcodebench; commit0, real-world bug/issue resolution alpharepair; swebench; swebenchm, and software testing titanfuzz; testgeneval. Among these tasks, SWE-bench swebench—a benchmark for solving real-world software issues—has emerged as a focal point of research efforts, and researchers have proposed various agentic sweagent; openhands; aider and pipeline-based agentless; moatless; autocoderover methods to push LLMs’ real-world issue solving capability. However, most current techniques rely on powerful proprietary LLMs like GPT-4o gpt4o or Claude-3.5-Sonnet claude35, where advancements are driven more by enhanced prompting strategies than by improvements in the underlying LLM.
With the release of DeepSeek-R1 deepseekr1, reinforcement learning (RL) using rule-based rewards has become a crucial technique for enhancing the reasoning capabilities of LLMs across various downstream tasks, including coding acecoder and mathematics cotstudy. However, their effectiveness in SE tasks remains limited deepseekr1, and their substantial total parameter size (671B) poses challenges for researchers attempting to train them. For mathematics, the reward is generally defined as whether the answer predicted by the LLM can exactly match the ground truth deepseekr1. While in coding, existing RL research typically utilizes execution feedback deepseekr1; rlef as the reward signal and is limited to competitive programming tasks codecontests; lcb, where code is self-contained and easily executable. This is challenging to apply to real-world SE tasks due to the execution cost and lack of executable environments swegym. Meanwhile, previous research swegpt; swegym; swefixer relied on proprietary teacher models and focused primarily on supervised fine-tuning (SFT), which, as we show in the paper, is less effective and less generalizable.
To address these limitations, we propose SWE-RL, the first RL method to improve LLMs on SE tasks by directly using rule-based rewards and software evolution data—the record of entire software lifecycle, including all code snapshots, changes, and events like PRs and issues. As shown in Figure 1, we begin by curating a comprehensive dataset of GitHub pull requests (PRs), which is then transformed into the seed dataset for RL. Each data item includes an issue, the corresponding code context, and the oracle patch merged by the PR. During RL, the policy LLM is tasked with solving a given issue through reasoning and producing the code changes. The code changes are then converted into a consistent patch format for reward calculation. If the response is incorrectly formatted, the reward will be ; otherwise, the reward is a similarity score (between 0 and 1) of the predicted and the oracle patch calculated by Python’s difflib.SequenceMatcher Gestalt_pattern_matching. Notably, we provide the complete content of each file in the input prompt, which implicitly teaches the model to reason about the precise fault locations before suggesting repair edits.
Applying SWE-RL to Llama-3.3-70B-Instruct llama31, our model Llama3-SWE-RL-70B solves 41.0% of the issues in SWE-bench Verified swebverified, a human-verified subset of SWE-bench, with Agentless Mini, our pipeline-based scaffold built upon Agentless agentless, featuring simplifications to match our RL process and enhancements for scaling. This performance is comparable to leading proprietary LLMs like GPT-4o gpt4o and state-of-the-art among medium-sized LLMs with less than 100B total parameters. Our ablation studies demonstrate that Llama3-SWE-RL-70B significantly outperforms its Llama baseline. Additionally, we developed a competitive supervised fine-tuning (SFT) model from Llama-3.3-70B-Instruct using synthetic data generated in the Magicoder magicoder style to enhance the chain-of-thought cot process, employing the same seed as SWE-RL. We show that Llama3-SWE-RL-70B, trained with SWE-RL solely for solving issues, not only surpasses the SFT model in SWE-bench but also excels in other out-of-domain (OOD) tasks, including function-level coding codex; liu2023code, practical code generation with library use bigcodebench, code reasoning cruxeval, mathematics mathbench, and general language understanding mmlu. In these OOD tasks, Llama3-SWE-RL-70B even outperforms Llama-3.3-70B-Instruct, whereas the SFT model results in decreased performance.
In summary, our contributions are as follows:
- •
We introduce SWE-RL, the first RL approach specifically designed to enhance LLMs for SE tasks using software evolution data (e.g., PRs) and rule-based rewards.
- •
We develop Llama3-SWE-RL-70B, trained with SWE-RL on Llama-3.3-70B-Instruct. It achieves 41.0% on SWE-bench Verified, the best performance among medium-sized language models (<100B) and even comparable to leading proprietary models like GPT-4o.
- •
We show for the first time that applying RL solely to real-world SE tasks, such as issue solving, can already enhance an LLM’s general reasoning abilities, enabling it to improve on out-of-domain tasks like math, code generation, and general language understanding.
To prepare the initial dataset for RL, we extract 273k high-quality PR seeds from the raw PR dataset we collected from GitHub. These seeds are selected based on specific heuristics. For example, a PR instance should include at least one linked issue, the issue should describe a bug-fixing request, and the code changes should involve programming files. For each seed, we extract the issue descriptions and code context, including all changed files and some relevant but unchanged files. These are converted to input prompts for the policy LLM. We also take the oracle patch from each seed, which will be used for reward calculation. More details are explained in .
We bootstrap the policy LLM with the prompt template shown in Figure 2. Assuming that the LLM has generated a rollout given an issue and its code context, the reward function is defined as follows:
Here, is the patch extracted from the LLM generation if it is correctly formatted, and means the oracle patch for this issue.
In the implementation, we use Python’s difflib.SequenceMatcher as the function, which returns a floating point between 0 and 1 indicating the sequence similarity, and adopt Group Relative Policy Optimization (GRPO) deepseekmath for policy optimization.
Given a seed RL dataset where each item contains (1) , the issue description, (2) , the code context required to solve this issue, including both files to repair and relevant files not intended for editing, and (3) , which is the oracle patch. The input prompt for each data item is formed by instantiating the prompt template in Figure 2 with the issue description and code context, which we denote as \question= \mathsf{form\mbox{-}prompt}(\mathsf{issue}, \mathsf{ctx}).ThepolicyLLMtriestosolvetheissuebygeneratingcodechangesthroughreasoning,andproducesmultipleoutputsforeachinputprompt\questiongiventhegroupsize.Then,thepolicyLLMaimstomaximizethefollowingGRPOobjective:\begin{equation}\small\begin{gathered}\end{gathered}$