Beyond Devin! Led by Yao Ban, they set a new world record for large model programming

Beyond Devin! SWEBench has welcomed a new player on the ranking list -

StarShip CodeGen Agent, produced by the start-up company OpenCSG led by Yao Ban, and ranked second in the world with a score of 23.67% score.

At the same time, it created the highest record of non-GPT-4o basic model (SOTA) .

We all know that SWebench evaluation is highly close to real programming scenarios and extremely difficult. It not only requires the model to understand the requirements, coordinate changes to multiple functions/classes and even files, but also Models are required to interact with the execution environment, handle extremely long contexts, and perform complex logical reasoning for traditional code generation tasks.

In this difficult real test, the most advanced GPT4 and Devin in the industry can only solve 1.74% and 13.86% of the problems.

This achievement is a leading move based on OpenCSG to promote the development of language models in a more practical, intelligent and autonomous direction. This move marks an important step taken by domestic companies in promoting the development of language model applications in a more practical, intelligent and autonomous direction.

How difficult is large model programming?

In March 2024, the emergence of Devin, the first AI software engineer, detonated the entire technology world. Although accompanied by a series of controversies, Devin's strong innovation capabilities and huge potential have brought new expectations to many AI enthusiasts and practitioners. Devin has profound technical skills and extensive knowledge reserves. He is known for his excellent algorithms and powerful programming abilities. His research results and developed software have been constantly breaking through and innovating, bringing

to many AI enthusiasts and practitioners. Devin is not only able to easily solve coding tasks, but can also complete the entire software development cycle independently - —From project planning to deployment, covering but not limited to building websites, independently searching and fixing bugs, training and fine-tuning AI models, etc.

Why does Devin dare to challenge the programming capabilities of basic models such as GPT4?

The core is that software engineers not only write code, but also involve requirements understanding, code interpretation, programming planning, code generation, debugging and exception repair, etc. Each link here will affect large model programming. usability and effectiveness.

For such real-life scenarios, Princeton University proposed SWEBench, a tool to quantitatively evaluate end-to-end code generation capabilities.

GPT-4’s score on SWEBench is only 1.74%. Even with RAG technology, the score is less than 3%. This shows that relying solely on basic models can directly solve real-world problems. The programming problem is impossible to do.

And Devin’s technological innovation is based on Agent-based workflow construction, which raises SWEBench’s solution rate to a new level.

In March, Devin topped the list with an independent problem solving rate of 13.86%, which directly improved "large model programming" from a state of being almost unusable to "seeing the light of day." Major Silicon Valley companies and large model startups have entered the field of LLM for SE, and this record has been continuously rewritten.

As of the end of April 2024, the best record was 20.33% created by the Amazon Q Developer Agent launched by the Amazon AI team.

Regrettably, compared to the "letting a hundred flowers bloom" of Chinese companies on the basic model list, Chinese companies rarely participated in this difficult challenge until this time OpenCSG rewrote this record.

From Chinese startup company

SWEBench’s latest evaluation results have been updated. OpenCSG has jumped to second place on the list. The OpenCSG StarShip CodeGen Agent launched by the company achieved a pass rate of 23.67% in the Lite evaluation. , this result not only exceeds the results of Devin and Amazon.

OpenCSG（Open Expression） was established only one year ago. It is a company dedicated to the construction of large model ecological community and brings together the upstream and downstream enterprise chains in the artificial intelligence industry to jointly build large models. A company that provides solutions and tool platforms for applications in vertical industries.

The team has deep experience in open source and large model compounding——

CEO Chen Ran is a well-known entrepreneur in the field of open source software and has successfully built many companies in the open source field business company.

CTO Wang Wei comes from Yao Class 05 of Tsinghua University and has many years of research and development experience in the field of artificial intelligence.

The company’s core R&D team also brings together elite students from Tsinghua University, Peking University, Wharton, Hong Kong University of Science and Technology and other universities.

So how does such a team create a new record?

Currently, many companies are actively exploring and practicing technologies such as basic models, vertical domain models, and RAG, while OpenCSG has chosen a focused direction: Dedicated to the innovative development of programming agents and the depth of large-scale model algorithms optimization.

Agent level: Different from LLM+RAG or general Agent framework, OpenCSG StarShip CodeGen Agent is designed for highly customized and optimized agents in the field of software research and development: integrating various stages of research and development (requirements understanding, code retrieval , programming plan, writing code, cycle verification, etc.) is implemented through LLM Agent, and combined with software engineering methods, such as AST syntax analysis, dependency retrieval, etc. for in-depth optimization, we strive for excellence in every link, and finally integrate to achieve a higher Accurate code generation.

Algorithm level: In response to typical problems such as API conflicts caused by code version changes, OpenCSG proposes an adaptive teacher model. The teacher model analyzes code version change records and generates high-quality programming data for use. To improve the generation effect of the basic model. According to the evaluation, the improvements brought by these innovations are significantly better than the current RAG model, especially in popular project scenarios where the API structure is updated frequently. The relevant results of this part have been formed into papers and submitted to international conferences.

It is this algorithm + engineering two-pronged approach and continuous improvement model that allows OpenCSG CodeGen Agent to stand out among other models.

"StarShip is all kinds of home appliances"

If the real evaluation of CodeGen Agent is a small test, then StarShip carries the grand blueprint of OpenCSG.

Regarding StarShip’s product positioning, OpenCSG CEO Chen Ran said:

StarShip bears our vision of reshaping software development for large models. Users form their own digital employee team through StarShip's built-in agent. CodeGen Agent is a digital programmer built into the platform. Currently, CodeReview Agent code reviewer and CodeSearch code question and answer engineer have been released. Unlike coding assistance tools, we expect these digital workers to work directly and independently without the need for human assistance intervention. In the future, we will release more types of digital employees to fully cover all aspects of requirements, design, coding, testing, and operation and maintenance.

CTO Wang Wei said that this path is full of challenges but very interesting, "From first principles, large models are no longer a question of 'yes' or 'no' for improving productivity. It’s a question of when, where and in what form. StarShip is the answer we are trying to give.” High yield:

CSGHub open source model platform, wukong pre-training model, CSGCoder fine-tuned code model

, etc. These products are accurately positioned and well received in the industry.

The rapid launch and iteration of these products not only meet market demand, but also serve a common goal: to empower everyone in every enterprise with large models.

To allow big models to empower every enterprise and every person, we need to make big models the same as water and electricity. If the big model is electric energy, then CSGHub is the electric power network, and StarShip is a variety of home appliances that will eventually empower thousands of households.

The concept of OpenCSG is open source. As a company that insists on open source as its core, it not only realizes open source models and code, but also makes the platform open source.

CTO Wang Wei summed it up this way: We are a young company that benefits from open source so that we can make some results in a short time. At the same time, we will also give back to the open source community in an all-round way. This is the basic principle of the open source community. In addition, I very much agree with Sam Altman's statement that open source is just a model, and product value is more important than the model.

"Benchmark itself is just a number. With the launch of GPT4-o, SWEBench's test scores are expected to exceed 30% soon, and optimistic estimates can exceed 50% next year. And we are more concerned about the factors behind these numbers. Product value: With the improvement of model capabilities and engineering technology, digital employees will change from quantitative changes to qualitative changes, from usable to easy to use, ushering in a comprehensive explosion in various industries." Wang Wei explained, "This may be the era of big models. A major change in the context, from companies to individuals, we must prepare for this.”

The above is the detailed content of Beyond Devin! Led by Yao Ban, they set a new world record for large model programming. For more information, please follow other related articles on the PHP Chinese website!