SWE-smith: Scaling Data for Software Engineering Agents

Building execution environments for repositories with passing tests. Given a repository, we run SWE-agent (Yang et al., 2024a) on the latest commit for at most $100$ steps, instructing it to install the codebase and run the test suite. We then manually verify the installation and testing instructions, check if more than $80$% of existing tests pass, and finally create a Docker image for the repository. We target repositories for the $5,000$ most downloaded packages listed in the Python Package Index (PyPI) as of November 18, 2024, sort the PyPI packages by GitHub stars, and then remove any PyPI package with less than $1,000$ stars, as well as all $12$ SWE-bench test repositories from consideration. More in §A.2.

Creating task instance candidates. Per repository, we employ four different strategies to create candidates. As shown in Figure 2, each strategy takes in a repository as input, then produces task instance candidates represented as .diff files. Extensive details in §B.

Execution-based validation of candidates. We apply each candidate patch to the corresponding repository, run the test suite, and only keep patches that break one or more existing, passing tests (referred to as Fail-to-Pass or F2P test(s)). For efficiency purposes, we also limit testing runtime to two minutes; bug candidates that cause test runtimes in excess of this time limit are discarded. Minor additional details in §A.3.

Generating problem statements. The issue text associated with a bug can significantly alter the difficulty and feasibility of the task instance. Detailed descriptions of “expected” vs. “observed” behavior or bug-reproduction code in issue text heavily affect an agent’s capacity to localize bugs or iterate on proposed solutions. We explore several techniques covered fully in §D, and ultimately settle on a simple strategy. Per task instance, we provide an LM with the .diff patch, source code of a random F2P test, and execution output from running the repository’s test suite with the bug patch applied. We prompt the LM for GitHub issue-style text that includes reproduction code based on the F2P test.

What human labor remains? The steps requiring manual effort are (1) parsing the correct installation setup procedures from the agent trajectory ($\sim 7$ min per repository), and (2) implementing the parser for test outputs ($\sim 1$ min per repository). Step two requires very little time because parsers can be reused for repositories with the same testing infrastructure (e.g., pytest). SWE-smith removes the need for manual efforts to determine installation specifications for multiple versions of a codebase across time, the most costly step of SWE-bench collection. Creating SWE-smith took one author $\sim{}20$h of human labor.

We apply SWE-smith to $128$ Python repositories, generating a total of $50$k instances. Table 1 captures the key statistics. On average, we generate $381$ task instances per repository, with as many as $2277$ for pandas-dev/pandas. We summarize the distribution of task instances per repository in Figure 1, where repositories are grouped into one of six general categories. SWE-smith took $ $1360$ to create ($ $1000$ to generate bugs, $ $160$ for automatic repository installation with SWE-agent, $ $200$ to generate issues for $10$K bugs). Generating an issue costs $2.54$¢ on average. More dataset analyses in §C.

Bug generation strategies vary in cost and yield rate. Of methods relying on LMs, PR Mirrors are more expensive because the task entails rewriting entire files, as opposed to individual functions for LM Modify and LM Rewrite. Yield rates are limited by either lack of test coverage for the change or because the bug candidate did not actually introduce relevant issues. For example, for LM Rewrite, the LM is asked to re-implement the function; it is not explicitly asked for bugs. When requested outright (LM Modify), the yield is higher.

How difficult are SWE-smith task instances? To determine whether task instances produced by SWE-smith are realistic and challenging, we train a Qwen $2.5$ $32$B model on $1{,}699$ human-annotated (task, label) pairs from Chowdhury et al. (2024) to rate tasks as (easy, medium, hard) by training. To quantify difficulty, each difficulty label corresponds to values of $1$/$5$/$9$. The model achieves $75.3$% test accuracy. We then rate difficulty of task instances from both SWE-smith and prior SWE-bench style datasets (Chowdhury et al., 2024; Jimenez et al., 2024b; Pan et al., 2024; Yang et al., 2024b). SWE-smith task instances span a broad range of difficulties, similar to SWE-bench and SWE-gym. The average difficulty score for SWE-smith ($5.27$–$5.72$ across bug generation strategies) is comparable to SWE-bench ($5.01$) and SWE-gym ($5.62$). This suggests SWE-smith enables realistic and appropriately challenging evaluation. We discuss why bug strategies yield different levels of difficulty and visualize difficulty per dataset in §E.

Scaling execution environments. Unlike SWE-bench which creates a Docker image per task instance, SWE-smith leverages a simpler design where tasks from the same repository share the same environment, reducing storage overhead significantly, as shown in Table 2. This approach not only makes scaling task instances more affordable, but also renders SWE-smith more accessible and maintainable than existing datasets. We estimate that creating a similar quantity of task instances ($50$k) using SWE-bench would require $50$ to $150$ TBs of storage for environments, a $500$x difference. Extended discussion in §C.1.

We perform several ablations of how SWE-smith’s bug and problem statement generation strategies impact the quality of training data. We use Claude 3.7 Sonnet as the expert for fine-tuning Qwen $2.5$ $7$B Coder Instruct, and report the performance on SWE-bench Verified.

LM Rewrite and Procedural bugs are comparable to PR mirrors. We randomly sample $1000$ instances per bug generation strategy (LM Modify, LM Rewrite, Procedural Modifications, PR Mirrors). Per instance, we generate issue text with an LM and run expert trajectory generation. We then fine-tune a student model per strategy, capping training points to the minimum number of successful trajectories from any strategy ($507$) for fair comparison.

Table 5 summarizes the results. Trajectories generated from PR mirrors are empirically the most effective training data — this is expected, since they are most reflective of SWE-bench. What’s noteworthy is that trajectories from Procedural Modification and LM Rewrite instances lead to competitive models. There is a steep drop-off with LM Modify bugs.

LM generated issues are comparable to real issues. We randomly sample $600$ PR Mirror task instances. We compare LM generated issues with three alternatives — fixed issue templates, the source code + test logs of a random Fail-to-Pass test, and the original issue text associated with the PR. We again cap training points to the minimum number of successful trajectories ($259$) for fairness.

As shown in Table 5, training on task instances with LM generated issues is empirically comparable to using the original issue text. Using fixed issue templates not only leads to the fewest successful trajectories, but also results in relatively homogeneous problem solving sequences. The expert trajectories from fixed issue templates have $31$% fewer unique actions compared to LM generated text ($379$ vs. $550$). While providing a Fail-to-Pass test case leads to more successful expert trajectories, leaking the evaluation criteria causes the model to skip over writing a reproduction script, which accounts for the performance drop. Of $500$ SWE-bench Verified instances, the student model trained on LM-generated issues attempts to reproduce the bug for $379$ of the runs. The model trained on test-based issues only does so for $127$ cases, a $66$% decrease.

Task difficulty correlates with solvability but not with effectiveness as training data. First, we run our difficulty rating model on $10$k randomly selected SWE-smith task instances. From this pool, we curate subsets of $1000$ instances corresponding to the three difficulty levels, then run expert trajectory generation per subset $3$ times. For the easy/medium/hard subsets, the resolve rate by the expert model are $58.6$%, $41.0$%, and $17.0$% respectively.

Next, from all successful trajectories, we create four fine-tuning datasets of $500$ trajectories each corresponding to difficulty scores of $2$, $4$, $6$, and $8$. As mentioned in Section 2.2, the corresponding scores for easy/medium/hard are $1$/$5$/$9$. Therefore, the SFT dataset for score $2$ is made up of trajectories corresponding to $375$ easy and $125$ medium instances, and so on. Somewhat surprisingly, we do not observe strong correlation between increased difficulty and downstream performance. For the student models trained on the $2$/$4$/$6$/$8$ difficulty SFT datasets, we get pass@1 scores of $12.4$%, $10.8$%, $13.6$%, and $12.2$% on SWE-bench Verified.

Training on more repositories improves general performance. We train models in four settings by sampling $700$ expert trajectories on Procedural Modification tasks from pools of $4$, $25$, $50$, and $100$ repositories. Echoing similar findings for code generation tasks (Xie et al., 2025b), we find that increasing repositories represented in the training set improves performance, as shown in Figure 5, with an approximately logarithmic relation between model performance and number of repositories.

Repository-specialized models excel on the target repository with minor generalization loss. We experiment with training models to be specialists on one particular repository. To assess performance, we evaluate models on a subset of SWE-bench Verified tasks that are (1) from SymPy, and (2) created after January 1st, 2022, a total of $22$ instances. To create SymPy specific training data, we first select a base commit of SymPy just before the cutoff date. Next, we create $1276$ Procedural Modification task instances, then generate $700$ expert trajectories. We evaluate specialization in two settings: (1) single-repository fine-tuning, and (2) specialist stage fine-tuning, both shown in Figure 5. For single-repository tuning, we compare a model initialized with Qwen-2.5-Coder-Instruct 7B and trained on $700$ instances sampled from $100$ repositories, to the same Qwen base model but fine-tuned on the $700$ SymPy instances only. For specialist stage fine-tuning, we simply compare SWE-agent-LM-32B to the same model further fine-tuned on the $700$ SymPy instances.

Specialization significantly boosts performance for the target repository with only slight drops in general performance in both the single-repository fine-tuning ($21.2$% vs. $13.6$%) and specialist stage fine-tuning ($42.4$% vs. $33.3$%) settings.

This section analyzes the behavior, failure modes, and efficiency of SWE-agent when run with SWE-agent-LM-32B or Claude 3.7 Sonnet on SWE-bench verified.

SWE-agent-LM-32B can solve tasks efficiently. SWE-agent-LM-32B resolves tasks in fewer steps on average (24.9) than Claude 3.7 Sonnet (29.1), though the difference becomes marginal when accounting for different average difficulties of the resolved tasks: On the overlap of tasks that are resolved by both LMs, SWE-agent-LM-32B uses 24.8 steps compared to 25.6 used by Claude 3.7 Sonnet (see Fig. 7). While shorter trajectories are not always preferred (additional actions can be used for additional validation purposes, for example), this shows that SWE-agent-LM-32B solves tasks very efficiently. At the same time SWE-agent-LM-32B also demonstrates that it can remain focused throughout long trajectories, with 31 instances being resolved after 40 steps or more. We further highlight that the accuracy of naturally terminating $^{1}$ agent submissions with SWE-agent-LM-32B achieve an accuracy nearly matching that of Claude 3.7 Sonnet (60% vs 63%), showing that SWE-agent-LM-32B is adept at determining whether an instance has been resolved. As the overall cost and turn count averages scale strongly with the cost and turn limits, we reserve a more thorough analysis for §F.5.1.

Repetitive actions are a key problem. We observe a tendency for SWE-agent-LM-32B to get stuck in long sequences of repetitive actions, in particular long sequences of calls that display different portions of a file instead of using search commands. $^{2}$ More than 25% of SWE-agent-LM-32B trajectories have a repetitive sequence of at least length 10, compared to less than 4% for Claude 3.7 Sonnet (see Figure 7). The occurrence of long repetitive sequences correlates strongly with the agent’s ability to solve the corresponding task instance, largely because the LM continues issuing similar commands until either the agent cost or turn limit is reached, at which point the run is terminated. For example, repetitive sequences of length 10 correspond to an 89% failure probability. Simple interventions from the agent scaffold can mitigate repetitive actions, but do not seem to improve resolve rates (see §F.5).

Localization is the dominant failure mode. Guided by a short plan in the system prompt, SWE-agent typically starts by localizing (search and read actions), reproducing (test file creation and execution), before modifying source files and validating the fixes. If the agent gets stuck at any of these stages or keeps on iterating, the agent loop is eventually interrupted by runtime limits (cost, number of LM calls, runtime). While this rarely happens with Claude 3.7 Sonnet, 53% of SWE-agent-LM-32b’s failures are associated with such limits (Figure 8). The agent often already gets stuck during localization or initial efforts to reproduce a bug, with endlessly repeated actions being a persistent issue. More on failure modes in §F.5.

We briefly review the format of a SWE-smith task instance, highlight how it is different from a SWE-bench task instance, and discuss why SWE-smith’s relatively simple infrastructure compared to SWE-bench allows us scale task collection much more efficiently.

A SWE-smith task instance is very similar to the form factor of a SWE-bench task instance, with several minor differences. A SWE-smith task instance includes the following fields:

We summarize the key distinctions between a SWE-smith and SWE-bench task instance:

In addition to the criteria discussed in Section 2.1, we also ensure that a repository has a license that allows non-proprietary use. The majority of software licenses are permissive (BSD, MIT, Apache), while the remainder are largely protective licenses (GPL) that still allow for non-commercial use. We inspected the repositories with custom licenses and confirmed they allowed for the use cases exercised in our work. The licenses for each repository are fully listed in Table 6.

We deliberately limit the search scope for repositories to those predominantly written in Python. Following precedents, focusing on Python repositories allowed us to form assumptions about installation and testing procedures (e.g. repository is organized as a PyPI package, pytest is the testing framework) that made scaling up automatic repository setup with SWE-agent more tractable. A worthwhile direction to consider for future work is expanding the coverage of repositories to be more comprehensive of codebases written in different programming languages, as  Yang et al. (2024b) does, extending SWE-bench style evaluation to JavaScript repositories with multimodal inputs.

Automated repository installation. The goal of this step is to first, get the installation and testing instructions for a repository, and second, create a Docker image containing the repository with the development environment set up.

We provide the system prompt given to SWE-agent that asks it to install a repository in Figure A.2. Each repository installation task is initialized with a clone of the original repository. No additional steps (e.g. pypi package downloads, conda environment setup) are performed.

We run SWE-agent with claude-3-5-sonnet-20241022 with a maximum cost limit of $ $2$ and a maximum call limit of $150$. The installation run terminates whenever one of these conditions is met. For every run, we record the interactions. We then manually review the trajectory, identifying the appropriate installation and testing specifications.

Each run incurs an average cost of $ $0.72$ and an average of $17$ steps before SWE-agent issues the submit command. The runs typically finish within two minutes. The majority of Python repositories require fewer steps — typically, SWE-agent will view the CONTRIBUTING.md, run the installation command provided verbatim in the text, and then runs pytest, showing all tests passing. A minority of repositories will require several steps because additional dependencies must be installed with apt-get. The manual review process following this requires $3$ to $20$ minutes. One author carried out this effort for $128$ repositories, taking an estimated $18$ human hours to accomplish. In the process of reaching $128$ repositories, the author gave up on $17$ repositories at the manual review stage.

Figure 10:

Prompt provided to SWE-agent + an LM asking it to install a repository.

We adapt SWE-bench’s validation script to convert each bug patch into a SWE-bench style task instance. This step ensures SWE-smith can be run by existing SWE-bench solutions. The conversion involves two steps. First, the bug patch is applied and pushed as a branch to a mirror clone of the repository. Second, we create a SWE-bench style task instance from the bug patch, populating important fields such as Fail-to-Pass and Pass-to-Pass tests with information from the validation logs.

We describe our workflows for generating bugs with an LM. For each function or class in a codebase, we prompt an LM to generate either a rewrite that introduces bugs or a complete re-implementation from scratch. This strategy is illustrated in Figure 12.

Modify existing functions. Given a Python codebase, we use the ast library to identify all unique functions, excluding any functions found under a testing related directory (e.g. tests, testing). Next, given a function, the LM is asked to write a new version that introduces logical, runtime bugs. Within the prompt, shown in Figure B, several suggestions of types of bugs along with a demonstration of a rewrite are provided.

Figure 13:

System prompt provided to an LM to generate bugs by re-implementing an existing target function.

file_src_code

refers to the original source file minus the target function’s original implementation.

func_to_write

refers to the signature and docstring of the target function.

In our experiments, we use OpenAI’s o3 mini model (OpenAI, 2024b) (o3-mini-2025-01-31) as the main base model for bug generation. Based on our empirical observations of an LM’s tendencies, we include several explicit guidelines in the prompt about what the rewrite should not do. Notably, it is important to ask the LM to not generate any inline comments denoting the location of a bug; we observe that without explicitly specifying this, model generation outputs tend to have inline comments pointing out the bug. We also want to avoid the complexities of identifying and removing such comments from a file diff representation. Second, we state that rewrites causing compilation or syntax errors (e.g. undeclared variables, function definition modifications) should be avoided because such bugs are relatively trivial to solve. We do not experiment extensively with different prompts or generating multiple buggy rewrites per function.

Modify existing classes. This method involves a simple amendment to the function rewriting approach. Instead of identifying unique functions (ast.FunctionDef), the codebase traversal logic instead looks for classes (ast.ClassDef). Otherwise, all other aspects of the implementation are near identical to function rewriting, with minor changes to the prompt to make bug suggestions and the demonstration more class oriented.

Rewrite existing functions. Instead of providing an LM with the original function, we explore an alternative strategy of asking an LM to re-implement a function from scratch. Similar to above, we again use the ast library to identify all unique functions. However, instead of directly asking for a bug, we remove the function’s implementation, then prompt the LM with the entire file containing the function (minus the original implementation). In the task description, we then explicitly ask for the LM to implement the function without changing the function signature.

We explore a zero-cost approach to create bugs by performing random modifications to the ast representation of a function or class. A “procedural modification” refers to a function that takes in an ast and applies a fixed transformation to it, such as removing a loop or swapping the blocks of an if/else clause. This strategy is illustrated in Figure 14.

Similar to the workflow for generating bugs with an LM, we first identify all functions or classes in a repository. Per procedural modification, we first impose a set of criteria that filters out any candidates for which the modification would be impossible. For instance, if the procedural modification removes a random conditional from a function, the modification’s criteria will filter out any candidates that are not functions or do not have a conditional. For the remaining candidates, the procedural modification is applied with controlled likelihood, where likelihood is a fraction indicating how often the procedural modification is applied within a candidate. For example, if the procedural modification removes a random function with a likelihood of $0.5$, then for every conditional declared within the function, there is a $50$% chance it gets removed. We introduce likelihood so procedural modifications do not lead to changes that are too difficult. Finally, the modified ast is converted back into source code.

Table 7 is a complete list of filtering criteria that is used for any procedural modification. For the filter_min_complexity and filter_max_complexity criteria, we define a simple definition of “complexity” as a sum of the number of conditional blocks, loops, boolean operators, exception handling blocks, and comparison operators in a function. The purpose of filter_min_complexity is to remove both simple, uninteresting functions (e.g. getter, setter methods) from consideration. filter_max_complexity is occasionally used to avoid changing long, monolithic functions.

Table 8 is an exhaustive list of all procedural modifications used to create bugs in a codebase.

We discuss the two strategies we use to combine bug patches from the same file or the same module. In practice, we combine LM and procedurally generated bugs that have been validated successfully as usable task instances.

From the same file. If two or more functions are defined within a single file, this strategy merges the function-level bug patches together. Given $n$ function-level bugs and $k$ as the number of bugs to combine, there are ${n\choose k}$ unique file-level candidate bug patches, which can be a large search space to cover. To make the search space tractable, ensure no single function-level bug is repeatedly used, and generate instances that reliably have $1$+ Fail to Pass tests, we implement the following approach described in Algorithm 1.

For each file in a codebase, we first identify the function-level bugs (or bug patches) that edit that file. The pool of bugs we draw from have been validated, meaning we have already ensured there is $1$+ Fail to Pass test(s) associated with the bug. From these pool of file_bugs, the get_combos function then generates up to max_combos sets of bugs, where the size of each set is num_bugs. For each combo, or set of bugs, the bugs are applied to the codebase one by one. If all patches are successfully combined, this means they were successfully merged, and the merged patch, which consists of multiple function-level bugs, is saved and re-validated as a single bug. Merging patches occasionally fails if there is an overlapping conflict between two files, akin to a merge conflict with git; this usually happens when a function is declared within another. To ensure a function-level bug is only used once, any remaining bug sets in combinations using any patch in combo are removed.

The limit_per_file and max_combos parameters prevent any one file from being over-represented and constrains an otherwise combinatorial large search space. We run this algorithm across all codebase files, typically setting num_bugs$=[2,4]$, limit_per_file$=3$, max_combos$=40$. Decreasing num_bugs or increasing the other three parameters improves the yield.

From the same module. There are several ways one could imagine composing function-level bugs from multiple bugs, such as combining those that break the same test or have a programmatic relationship (e.g. function a calls function b). We found a relatively straightforward and effective approach to be combining files that edit the same “module”. By “module” we are referring to a subdirectory within the source code (e.g. sklearn/feature_extraction, astropy/convolution). Out of all SWE-bench instances that edit $2$+ files, $75$% modify files within the same submodule, suggesting a high degree of intra-module code changes. The implementation for our approach is described in Algorithm 2

The implementation for this approach is similar to Algorithm 1 with two key changes. First, we do not do file-by-file or folder-by-folder traversal. Instead, using the diff patches, we create a dictionary map_path_to_bugs that mimics the file structure of a codebase. For example, if bug modifies path a/b/c/d.py, it is represented as map_path_to_bugs[a][b][c][d.py] $=$ [bug]. Additional bugs that modify the same path are appended to the list. Since every bug is a function-level bug, there will never be a bug registered in multiple lists. We then “collapse” up to depth indices. So for instance, at depth $=3$, the above data structure is collapsed into map_path_to_bugs[a/b/c][d.py] $=$ [bug]. Finally, any nested dictionaries are collapsed into a single list of patches (e.g. map_path_to_bugs[a/b/c] $=$ [bug]). Mirroring the procedure in Algorithm 1, we then iterate across this dictionary’s values (lists of bugs). Second, we only save patches that modify $2+$ files; aggregate bugs (represented by combo) modifying a single file are not considered.

Again, we run this strategy across all $100$ repositories, with parameters num_bugs$=[2,5]$, limit_per_module$=10$, max_combos$=100$, and depth$=2$. Reducing num_bugs, depth and increasing the other parameters yields more bugs. We choose a depth of $2$ because empirically, we find that meaningful modules are usually declared as immediate sub-folders of the main source code folder (e.g. in sklearn/feature_extraction, sklearn is the source code folder while feature_extraction is the module). A shallower depth leads to less meaningful groupings, while yield decreases significantly for every increased level of depth, particularly for smaller repositories.

We finally discuss the fourth and last strategy for generating bugs - mirroring real world pull requests (PR). We visualize this process in Figure 16.

Why use an LM? When we initially implemented this approach, we attempted to directly perform a git apply --reverse [patch] on the codebase. However, for the large majority of patches, this fails. We performed troubleshooting by inspecting $100$ PR patches on the sqlfluff/sqlfluff repository, leading us to two observations.

Therefore, we employ LMs to perform patch reversal, and find that reasoning models (e.g. o3-mini (OpenAI, 2024b)) are particularly effective.

Description of method. We follow SWE-bench’s methodology for crawling PRs created January 1st, 2023 and onwards, with minor and arbitrary exceptions for some repositories where we crawl older PRs as well. Per PR, we iterate across the file(s) changed by the patch. Per file, we prompt an LM with the file-specific changes from the patch along with the file’s source code in the current state of the repository (not the repository’s state corresponding to when the PR was applied, referred to as the base_commit in SWE-bench). The LM is asked to generate a rewrite of the file that reverts the changes reflected in the PR. We aggregate the changes across all file(s) into a single patch.

Because we are interested in problems that our expert trajectory generation method (SWE-agent + Claude 3.7 Sonnet) has a chance of solving, we do not attempt to reproduce PRs that change more than $8$ files. This constraint is imposed because no SWE-bench instance that edits more than $6$ files has ever been solved (Jimenez et al., 2024a).

How well does PR mirroring work? We scrape the PRs corresponding to $100$ randomly selected SWE-bench task instances from the django/django GitHub repository and attempt to recreate these task instances with SWE-smith’s collection process. We successfully recovered $92$ of $100$ task instances. Of these, $84$ break identical F2P test(s), with the remaining $8$ breaking a subset because some tests were removed over time. This sanity check gives us confidence that the PR mirroring strategy lives up to its name.

Comparison to SWE-bench. This approach has several benefits and drawbacks compared to SWE-bench’s collection pipeline. First, it removes the need to create instance-specific Docker images — all PRs are mirrored against the same version of a repository. This also implies that there is no need to write installation specifications for past versions of a repository, which is typically the most laborious step in task construction with SWE-bench. Finally, this strategy also allows us to loosen the requirements on what PRs we attempt to convert into a task instance. In SWE-bench, the core requirements for what PRs to attempt to convert into a task instance include:

With this collection strategy and SWE-smith’s focus on training data, the second and third requirements are no longer necessary. If there is no associated issue, issue text can simply be generated. If the patch does not contain any testing related changes, this is tolerable, as the validation stage will determine whether the PR breaks any tests. With these considerations, we purport that SWE-smith’s PR mirroring strategy can re-purpose a higher percentage of real world code changes for training purposes.

The main downside is that the rest of the repository is out of sync with the state of the codebase when the PR was applied. As a result, it’s possible that changes in the behavior of the rest of the codebase may affect the issue’s reproducibility or the accuracy of the issue description (e.g. line numbers referenced in the issue text are likely somewhat off with respect to the codebase). However, a simple mitigation for this is to create a Docker image for a repository at an earlier commit that’s closer to the original creation date of the issue. While we do not carry out a targeted experiment, we hypothesize that using SWE-smith, we would be able to reproduce SWE-bench entirely with $10$x less human hours with an estimated $2294$ x $ $0.055$ = $ $126.17$ in costs.

We provide extensive details about different aspects of each of the bug generation strategies, including the yield rates, labor/monetary costs, and dataset characterizations.

Yield rates. In Table 10, we provide the yield rates for each bug generation method across all repositories in SWE-smith. In general, we find that the PR Mirroring has the lowest yield rate at $13.18$% (although this rate is somewhat higher than SWE-bench’s yield rate of $2294/93139=2.46$%). For using LMs to generate bugs, modifying functions to introduce bugs intentionally has a higher yield than asking LMs to perform a best-effort rewrite. The efficacy of Procedural Modifications varies by strategy. For instance, shuffling the functions declared in a class only breaks existing test(s) $1.93$% of the time, but inverting a conditional will lead to a task instance for $47.04$% of modifications. Finally, combining bug patches has an extremely high yield rate - this is to be expected because we only attempt to combine bug patches that have been validated as usable task instances breaking $1+$ tests.

The number of repositories captured by each bug generation technique varies due to each strategy’s specific preconditions, which at times may not be effective for some repositories. For instance, the Procedural (Class *) set of methods only mutates Python classes. This strategy is fruitless for the minority of SWE-smith repositories that do not define any classes. The Procedural (Op Break Chains) method randomly removes operations and operands from expressions with two or more operations (e.g. $a+b+c\rightarrow a+b$) — such expressions are not always present in SWE-smith repositories.

The collective yield rate across SWE-smith’s bug generation strategies is significantly higher than SWE-bench’s collection strategy.

The yield rate also varies with respect to the repository it is being applied to. We provide a summary of yield rates by repository in Table 11. We generally observe that lower test coverage correlates with a lower yield rate.

Dataset characterizations. In Table 12, we provide statistics about the validated task instances produced by different bug generation strategies. Our work’s LM-based strategies rewrite one function in one file. Procedural modifications will also only change one file, but depending on the strategy, $1+$ functions or classes may be changed. Combining multiple patches from the same file always produces a patch with $2+$ functions edited. Combining across modules produces a patch with $2+$ files edited. The targeted nature of each of the bug creation strategies is reflected in the typical number of functions and files that the bugs produced by each strategy edits.

In Figure 17, we show the distributions for different attributes of SWE-smith compared to other SWE-bench style datasets. Compared to prior works, there is a much higher proportion of task instances with more than one Fail-to-Pass test. For any one repository, we find that SWE-smith task instances collectively cause failures for a much higher percentage of the testing suit than other datasets; a potential benefit of this is that training on SWE-smith based trajectories may expose models to a much broader set of functionalities in a codebase. The number of lines and files edited by SWE-smith task instances is highly similar to the trend lines for SWE-bench Verified.

We note that unlike other datasets, the trend line of SWE-smith task instances is “adjustable”. In other words, the Figure 17 distributions are a capture of the task instances provided in this release of SWE-smith. However, because of SWE-smith’s flexible bug creation techniques, the distribution can be “shaped” if needed. For instance, generating more task instances using the bug patch combination method would shift all three curves in Figure 17. We make this point to highlight the fact that the attributes of SWE-bench task instances are, in a sense, constrained by real world software development behavior. On the other hand, SWE-smith can be used to break tests and code that may not be reflected at all in any existing pull request. In this sense, we argue that LMs trained on SWE-smith have better “exposure” to a codebase compared to exclusively training on pull requests.

Continuation of scaling execution environments. The validation and evaluation procedures for SWE-smith deviate slightly from SWE-bench’s harnesses. The main reasons for these differences can largely be attributed to the granularity of installation specifications. In SWE-bench, each task instance corresponds to a unique base commit, with additional version and environment_setup_commit keys needed as indirection for mapping an instance to the correct set of installation and testing instructions. Across time, the continuous evolution of a repository and its dependencies make for an incredibly high degree of variability in how a repository should be installed correctly. To solve this variability, the community has resorted to creating an image per task instance, as done in Chowdhury et al. (2024). Therefore, for $2294$ SWE-bench task instances, there are $2294$ unique Docker images, each at a size of at least several gigabytes ($\sim 5$-$6$ GBs).

On the other hand, the simplicity and scalability of SWE-smith’s design allows one to support many task instances with comparatively much fewer Docker images. As mentioned above, installation and testing procedures are (repository, commit) specific. Therefore, when bugs are generated from each (repository, commit), all bugs can be reproduced and tested successfully from the same Docker image. In other words, if I generate $100$ bugs for a repository at some commit, instead of $100$ Docker images, only a single Docker image is required to run inference on any of the $100$ task instances.

This design is what enables SWE-smith to be significantly more space-efficient than SWE-bench. Based on the publicly released images, for SWE-bench’s $2294$ task instances, $1.2$ TBs of storage are required to download all Docker images locally. for SWE-bench Multimodal’s $517$ task instances, $1.2$ TBs are required. The higher per-instance Docker image size for SWE-bench Multimodal is due to how JavaScript dependency management tools (e.g. npm) require more storage compared to equivalent Python infrastructure (e.g. pypi). Pan et al. (2024) states that each image for the $2438$ instances an average of $2.6$GB, totaling 6 TB of storage total. Such a storage requirement can be a significant barrier for academic practitioners.

On the other hand, with more than $20$x the number of bugs, SWE-smith requires only $125$ Docker images total, corresponding to the number of unique (repository, commit) pairs (in this work, for each repository, we only determine installation and test specifications for one commit). The $125$ images require a total of $290.54$ GBs. In summary, compared to SWE-bench’s task collection strategy, SWE-smith’s design makes it easier to not only create task instances, but also train on them as well.

To better understand the differences between the SWE-bench and SWE-smith collection strategies, we perform SWE-smith collection on the pallets/flask GitHub repository, one of the $12$ test split repositories from the original SWE-bench benchmark. We review the steps covered in Section 2.1 applied to pallets/flask in detail. First, we defined the installation and testing specifications for the pallets/flask repository at commit bc09840. Next, we apply the LM modification bug generation strategy to this version of the repository, generating $267$ unique bugs.

We observe several differences. First, the SWE-smith collection strategy yields a much higher number of bugs outright. From SWE-bench, $11$ task instances are from the pallets/flask repository. The task instances were originally filtered from $2434$ pull requests (PRs), with $107$ satisfying SWE-bench’s filtering criteria of (1) being linked to one or more issues and (2) featuring 1+ new tests. Out of these $107$, the $11$ ($0.45$% of $2434$) task instances represent the proportion of PRs that execution environments could be successfully constructed for. On the other hand, running the function-level rewriting strategy for bug generation originally yielded $402$ candidates, of which $267$ were determined to be valid task instances.

Second, SWE-smith requires significantly less human effort while only incurring minor costs. Collecting the $11$ pallets/flask task instances (steps include scraping PRs, determining repository versions across time, defining version-specific installation/test specifications, running execution-based validation multiple times) took an estimated $38$ hours worth of human labor. On the contrary, defining installation and testing specifications for the latest commit of pallets/flasks took $10$ minutes. The subsequent function-level rewriting strategy for bugs took $23$ minutes to run, incurring a total cost of just $ $2.47$ ($\sim$$ $0.00613$ per instance). The final execution-based validation step that filters out $402-267=135$ unqualified bug candidates ran in $14$ minutes. Since both the bug and problem statement generation strategies are repository agnostic, no additional human intervention is necessary for these steps. Head to head, per instance for the pallets/flask repository, SWE-bench style collection requires $38\times 60/11=207.27$ minutes compared to $0.176$ minutes ($\sim 10.6$ seconds) and $ $0.00613$ in API costs using SWE-smith.

Third, collectively, SWE-smith task instances break a significantly larger proportion of existing tests in a codebase. We define “bug coverage” as the proportion of tests broken by $1$+ instance across all task instances. For the SWE-bench split of pallets/flask, there are $207$ unique tests across all $11$ instances. Of these $207$ tests, $15$ are broken by $1$+ instance, corresponding to a bug coverage rate of $7.25$%. For the SWE-smith split of pallets/flask, there are $474$ unique tests across $267$ instances. The larger amount of tests is due to increased test coverage in the pallets/flask repository as of Nov. 28, 2024 (when SWE-smith was collected) compared to June 2023 (when SWE-bench was collected). Of these $474$ tests, $422$ are broken by $1$+ instance, a bug coverage rate of $89.03$%. We attribute the significant difference to a consistent tendency in real world open source software development workflows, that is, the minority of tests are introduced to capture existing, errant behavior in the repository. The significant majority of tests are committed alongside working code, ensuring that already correct behavior is upheld. Well-maintained repositories will typically not merge commits that cause such tests to fail. This results in a large number of tests where few to no commits correspond to those tests’ failures.

Finally, SWE-smith does not yield instances appropriate for evaluation. The SWE-smith pipeline as presented does not produce hidden tests, a crucial difference that makes SWE-bench more suitable for evaluation. Consequently, when expert trajectories are generated, the Fail-to-Pass tests are present in the repository at inference time. Furthermore, our issue generation strategy does not include checks for known problems such as underspecified text descriptions or solution leakage (Chowdhury et al., 2024). Simple amendments could make SWE-smith task instances suitable for evaluation, such as deleting Fail-to-Pass test functions or files along with a validation procedure around the ambiguity and leakage of the issue text. Finally, thorough analyses of how faithful SWE-smith task instances are to real world issues and PRs would be necessary to justify synthetic bugs for evaluation.

Rejection sampling fine-tuning. Our fine-tuning setup heavily inherits from  Pan et al. (2024)’s work. We perform full parameter fine tuning using the torchtune (PyTorch, 2024) library, with learning rate 5e-5, maximum $3$ epochs, and max context length of $32768$. Training was carried on Modal (Modal, 2025) on $2$-$8$ NVIDIA H100 80G GPUs. As discussed in Section 3, the procedure for rejection sampling fine-tuning (RFT) is as follows. We first generate expert demonstrations/trajectories using SWE-agent and a “strong” model (e.g. Claude 3.7 Sonnet, GPT 4o) on SWE-smith task instances. Of these, we then only train a student model on the trajectories corresponding to resolved instances.

SWE-agent configuration. We use two different configurations, one for generating trajectories with an expert model, and a separate one for running inference on the fine-tuned Qwen, student models. The configurations are generally quite similar, with minor differences around how LMs’ responses are elicited, the parsing mechanism for an LM response, constraints around message sizes, and the system prompt.

We will first review the information common to both configurations. The prompt template informing an agent of the task’s nature and problem statement is included in Figure F.1. This prompt is very similar to the original SWE-agent prompt used in Yang et al. (2024a). The prompt templates for showing environment feedback are identical as well. If there is execution output, the text is simply preceded by OBSERVATION: [output]. If there is no output (e.g rm -r succeeds silently), then the agent is informed “Your command ran successfully and did not produce any output”. The agent computer interface (ACI) provided is also identical; SWE-agent provides LM with access to three general tools:

Figure 20:

A copy of the prompt provided to an LM via SWE-agent informing the LM of the nature of the task, the task description itself, and several tips on how to proceed.

We briefly review the distinctions. First, tool invocation works differently for expert versus student models. For the Claude and GPT series models that are used as experts, we use function calling for models to invoke the aforementioned tools. On the other hand, the student model is asked to generate a response with XML tags to delineate the thought and action. Therefore, when fine-tuning on expert trajectories, a key processing step is to convert the expert trajectories’ function calling format into the XML style response — fine-tuning directly on the expert trajectories does not work.

We note that we use these particular settings because as of the publication of this paper, this tool setting reflects the absolute state-of-the-art performance achieved with an open source agent system (SWE-agent) and any existing LM (Claude 3.7 Sonnet). It is certainly possible to explore more tool designs and experiment with different formatting calls, as many existing prior works, notably Yang et al. (2024a), have performed. However, given the focus of our work, we do not bother with repeating such a ”hyperparameter sweep” across configurations for the agent system, as this effort is expensive and has already been performed to suggest that the configuration we are using is ideal for expert level performance.

For generating trajectories with expert models, we run with a maximum of $75$ steps and a cost limit of $ $2.00$. A run terminates automatically when either of these limits are reached or the context window of the expert model is exceeded. The overwhelming majority of automatic terminations are due to the $75$ maximum steps limit.

For running inference with student models, we run with a maximum of $75$ steps or a cost limit$^{3}$ of $ $2.00$, where the run similarly terminates when either the steps, cost or context window limit is reached. For the student model, per LM inference call, we truncate the message history to only keep the $5$ most recent tool outputs. While we occasionally sample trajectories with the expert model set at various temperatures, for the student model, the temperature is fixed at $0.0$.

SWE-bench. SWE-bench is a widely used benchmark that evaluates AI systems on their ability to resolve GitHub issues (Jimenez et al., 2024b). Given a codebase along with a description of a bug or feature, the AI system is asked to modify the codebase in such a way that the issue presented in the description is resolved. SWE-bench consists of $2294$ such task instances, collected from real world pull requests (PRs) and issues in $12$ GitHub repositories that are predominantly Python. As discussed in Section 3, the Lite and Verified subsets are curated from the main SWE-bench repository with the goal of making evaluation either more efficent or more reliable. Since evaluation on the entirety of SWE-bench is fairly costly and does not have as many comparable references, we do not evaluate SWE-agent-LM-32B on the entire SWE-bench test set.

SWE-bench Multimodal. SWE-bench Multimodal applies SWE-bench collection strategy to $12$ additional predominantly JavaScript and TypeScript GitHub repositories, where task instances are associated with issues that have visual asset(s) in them (Yang et al., 2024b). The evaluation dataset consists of $510$ task instances. While the original work evaluates vision language models (VLMs) specifically, we do not evaluate SWE-agent-LM-32B which, as it is based on Qwen $2.5$ Coder Instruct, does not have the ability to process images as inputs.

SWE-bench Multilingual. SWE-bench Multilingual is an evaluation dataset consisting of 300 task instances that we introduce with this work. A single author carried out SWE-bench’s collection strategy for $42$ additional GitHub repositories, covering the following $9$ programming languages: JavaScript, TypeScript, C, C++, Go, Java, PHP, Ruby, and Rust. These repositories span a wide range of application domains, including web frameworks, data storage and processing tools, core utilities, and widely used libraries. A brief summary of the dataset is presented in Table 15.