--- title: Code Editing Tool Benchmarking description: Benchmarking new and existing code editing tools, we find nearly identical performance in success rates with various tool calling techniques. image: https://blog.enginelabs.ai/hubfs/AI-Generated%20Media/Images/a%20belt%20overloaded%20with%20tools%20in%20a%20realistic%20style.jpeg --- [Skip to content](https://blog.enginelabs.ai/code-editing-tool-benchmarking#main-content) [![66b0ede4e964df8d4548c1f9_web](https://blog.enginelabs.ai/hs-fs/hubfs/66b0ede4e964df8d4548c1f9_web.png?width=100&height=100&name=66b0ede4e964df8d4548c1f9_web.png)](https://blog.enginelabs.ai/) - [Home](https://enginelabs.ai/) - [Docs](https://docs.enginelabs.ai/) - [Github](https://github.com/engine-labs) Open main navigation Close main navigation - [Home](https://enginelabs.ai/) - [Docs](https://docs.enginelabs.ai/) - [Github](https://github.com/engine-labs) - [Hire Engine](https://calendly.com/engine_simon/engine-meeting) [Hire Engine](https://calendly.com/engine_simon/engine-meeting) Dec 2, 2024 # Code Editing Tool Benchmarking [Sudhir Balaji](https://blog.enginelabs.ai/author/sudhir-balaji) Share: [facebook-f icon](http://www.facebook.com/share.php?u=https://blog.enginelabs.ai/code-editing-tool-benchmarking) [linkedin-in icon](http://www.linkedin.com/shareArticle?mini=true&url=https://blog.enginelabs.ai/code-editing-tool-benchmarking) [Twitter icon](https://twitter.com/intent/tweet?url=https://blog.enginelabs.ai/code-editing-tool-benchmarking) [pinterest-p icon](http://pinterest.com/pin/create/link/?url=https://blog.enginelabs.ai/code-editing-tool-benchmarking) [envelope icon](mailto:?body=https://blog.enginelabs.ai/code-editing-tool-benchmarking) ![a belt overloaded with tools in a realistic style](https://blog.enginelabs.ai/hs-fs/hubfs/AI-Generated%20Media/Images/a%20belt%20overloaded%20with%20tools%20in%20a%20realistic%20style.jpeg?width=1344&height=768&name=a%20belt%20overloaded%20with%20tools%20in%20a%20realistic%20style.jpeg) We frequently monitor the leaderboard for [SWE-bench](https://www.swebench.com/), which is a benchmark for measuring the performance of LLMs at editing real-world Python codebases to resolve real-world GitHub issues. Although this is slightly different to what we're doing at Engine Labs, SWE-bench still provides a reasonable way to track performance and keep an eye on the state of the art. A new version of Anthropic's Claude 3.5 Sonnet was released in October 2024 - with this came a jump in success rate of submissions to the SWE-bench leaderboard. At the same time, Anthropic also released tools to [allow Claude to use a computer directly](https://www.anthropic.com/news/3-5-models-and-computer-use). Examining submission logs, we saw that two new high-scoring leaderboard entrants were both using tools that looked like they were based on code from Anthropic's [computer use demo quickstart](https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo/computer_use_demo/tools). The `EditTool` from the quickstart in particular looked like a novel way of providing an LLM with the ability to view, create, and edit files. There have been a few different approaches to providing LLMs with these abilities over time - we had been successfully using a set of tools based on the [SWE-agent paper](https://arxiv.org/abs/2405.15793) to achieve the same things. We wanted to see if `EditTool` was responsible at all for the increased benchmark success rates. We used the [Aider](https://aider.chat/) project's [code editing benchmark](https://aider.chat/docs/benchmarks.html) to run two evaluations - one for the new tool and one for our existing toolset, using a basic agent loop with a maximum of 5 iterations and the newest Claude model `claude-3-5-sonnet-20241022`. ## Existing toolset results ``` { "successRate": "55.6%", "successes": 74, "failures": 59, "errors": 0} ```   ## `EditTool` results ``` { "successRate": "54.9%", "successes": 73, "failures": 60, "errors": 0} ```   Overall performance is nearly identical between the two toolsets, suggesting that the new toolset is unlikely to have made a difference to benchmark success rates. An interesting observation from the tests is that each toolset succeeded in some cases where the other failed. We did not repeat any of these tests but we would expect different outcomes for the same inputs due to inherent LLM non-determinism depending on the temperature settings (we used the default Anthropic API temperature here). If you would like to see more detailed results, please reach out to us at contact\[at\]enginelabs.ai! (We also recently submitted results for our agent to the Verified leaderboard of SWE-bench, with a 51.8% success rate - keep an eye out!) ## Related posts [![](https://blog.enginelabs.ai/hs-fs/hubfs/AI-Generated%20Media/Images/an%20image%20suitable%20for%20a%20comparison%20article%20between%20an%20AI%20software%20engineering%20agent%20vs%20an%20AI%20copilot%20in%20the%20IDE.jpeg?height=200&name=an%20image%20suitable%20for%20a%20comparison%20article%20between%20an%20AI%20software%20engineering%20agent%20vs%20an%20AI%20copilot%20in%20the%20IDE.jpeg)](https://blog.enginelabs.ai/engine-vs-github-copilot-autonomous-ai-software-developer-vs-ide-assistant) ## [Engine vs GitHub Copilot - Autonomous AI Software Developer vs IDE Assistant](https://blog.enginelabs.ai/engine-vs-github-copilot-autonomous-ai-software-developer-vs-ide-assistant) [Simon Spurrier](https://blog.enginelabs.ai/author/simon-spurrier) Jan 12, 2025 [Read more](https://blog.enginelabs.ai/engine-vs-github-copilot-autonomous-ai-software-developer-vs-ide-assistant) [![](https://blog.enginelabs.ai/hs-fs/hubfs/AI-Generated%20Media/Images/an%20image%20with%20a%20coding%20ide%20compared%20with%20an%20AI%20software%20engineer%20humanoid.jpeg?height=200&name=an%20image%20with%20a%20coding%20ide%20compared%20with%20an%20AI%20software%20engineer%20humanoid.jpeg)](https://blog.enginelabs.ai/cursor-ai-vs-engine-autonomous-ai-software-developer-vs-ide-assistants) ## [Cursor AI vs Engine: Autonomous AI Software Developer vs IDE Assistant](https://blog.enginelabs.ai/cursor-ai-vs-engine-autonomous-ai-software-developer-vs-ide-assistants) [Simon Spurrier](https://blog.enginelabs.ai/author/simon-spurrier) Jan 12, 2025 I you're searching for a Cursor AI alternative Engine might be a good choice. It can also be a good... [Read more](https://blog.enginelabs.ai/cursor-ai-vs-engine-autonomous-ai-software-developer-vs-ide-assistants) [![](https://blog.enginelabs.ai/hs-fs/hubfs/AI-Generated%20Media/Images/padlock.jpeg?height=200&name=padlock.jpeg)](https://blog.enginelabs.ai/security-for-ai-software-engineering) ## [AI Software Engineering - Security and QA](https://blog.enginelabs.ai/security-for-ai-software-engineering) [Simon Spurrier](https://blog.enginelabs.ai/author/simon-spurrier) Nov 19, 2024 Security [Read more](https://blog.enginelabs.ai/security-for-ai-software-engineering) [linkedin-in icon](https://www.linkedin.com/company/enginelabs) [X Twitter icon](https://x.com/enginelabsai) [![Group 11 (1)-1](https://blog.enginelabs.ai/hs-fs/hubfs/Group%2011%20(1)-1.png?width=100&height=17&name=Group%2011%20(1)-1.png "Group 11 (1)-1")](https://enginelabs.ai) Copyright © 2025 Engine Labs ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Sudhir Balaji", "url" : "https://blog.enginelabs.ai/author/sudhir-balaji" }, "dateModified" : "2024-12-02T22:47:33.045Z", "datePublished" : "2024-12-02T22:47:33.000Z", "headline" : "Code Editing Tool Benchmarking", "image" : [ "https://blog.enginelabs.ai/hubfs/AI-Generated%20Media/Images/a%20belt%20overloaded%20with%20tools%20in%20a%20realistic%20style.jpeg" ], "mainEntityOfPage" : { "@id" : "https://blog.enginelabs.ai/code-editing-tool-benchmarking", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://blog.enginelabs.ai/hubfs/Group%2011%20(1)%20(1).png" }, "name" : "Era Technologies Limited" } } ```