Surpassing Devin, Yao Ban leads OpenCSG to set a new world record for large model programming

StarShip CodeGen Agent launched by OpenCSG, a large-scale model startup in China, has achieved second place in the world in the Prince WEBench (Real Independent Programming Evaluation of Large Models) rankings through innovation. At the same time, it created a highest record (SOTA) based on the GPT-4o model. (Word count: 37)

SWEBench evaluation is highly close to real programming scenarios and extremely difficult. It not only requires the model to understand the requirements and coordinate changes between multiple functions/classes, but also requires the model to interact with the execution environment and process Very long contexts and perform complex logical reasoning that far exceeds traditional code generation tasks. In this difficult real test, the most advanced GPT4 and Devin in the industry can only solve 1.74% and 13.86% of the problems. This achievement of OpenCSG marks a leading step taken by domestic companies in promoting the development of language models in a more practical, intelligent and autonomous direction.

Devin - Innovation in large model programming

In March 2024, the first AI software engineer Devin was born, detonating The entire technology community. Although it is accompanied by a series of controversies, Devin's strong innovation capabilities and huge potential have brought new expectations to many AI enthusiasts and practitioners. Devin can not only easily solve coding tasks, but also complete the entire software development cycle independently - from project planning to deployment, including but not limited to building websites, independently finding and fixing bugs, training and fine-tuning AI models, etc.

The real challenge of large model programming

Why does Devin dare to challenge the programming capabilities of basic models such as GPT4? The core is that software engineers not only write code, but also involve requirements understanding, code interpretation, programming planning, code generation, debugging and exception repair, etc. Each link here will affect the usability and effect of large model programming.

For such real-life scenarios, Princeton University proposed SWEBench (https://arxiv.org/abs/2310.06770), which is a tool to quantitatively evaluate end-to-end code generation capabilities. GPT-4's score on SWEBench is only 1.74%. Even with RAG technology, the score does not exceed 3%, which shows that it is not feasible to directly solve real-world programming problems by relying solely on basic models.

Devin’s technological innovation is based on Agent-based workflow construction, which raises SWEBench’s solution rate to a new level. In March, Devin topped the list with a problem solving rate of 13.86% independently solved, which directly improved "large model programming" from a state of being almost unusable to "seeing the light of day." Major Silicon Valley companies and large model startups are recruiting LLM for SE, and this record has been continuously revised. As of the end of April 2024, the best record was set by the Amazon Q Developer Agent launched by the Amazon AI team at 20.33%. Regrettably, compared to the "letting a hundred flowers bloom" of Chinese companies on the basic model list, Chinese companies rarely participated in this difficult challenge until this time OpenCSG revised this record.

Breakthroughs of Chinese startups

Recently, the latest evaluation results of SWEBench were updated. OpenCSG jumped to second place on the list. The OpenCSG StarShip CodeGen Agent launched by the company was evaluated in Lite Achieved a pass rate of 23.67%. This result not only exceeded the results of Devin and Amazon, but also created a historical record for the revolutionary GPT4-o base model (SOTA).

OpenCSG has only been established for one year. It is a team with profound experience in open source and large model composites: CEO Chen Ran is a well-known entrepreneur in the field of open source software and has successfully built There are many commercial companies in the open source field; CTO Wang Wei comes from Tsinghua University's Yao Class 05 and has many years of R&D experience in the field of artificial intelligence; the company's core R&D team also brings together elite students from Tsinghua University, Peking University, Wharton, Hong Kong University of Science and Technology and other universities . How does such a team create a new record?

New record, model+AgentQifali

Currently many companies are actively exploring and practicing basic models and vertical fields Model and RAG and other technologies, while OpenCSG has chosen a focused direction: committed to the innovative development of programming agents and the in-depth optimization of large-scale model algorithms.

AgentLevel: Different from LLM+RAG or general Agent framework, OpenCSG StarShip CodeGen Agent is designed for highly customized and optimized agents in the field of software research and development: integrating all stages of research and development (Requirements understanding, code retrieval, programming planning, code writing, cycle verification, etc.) are implemented through LLM Agent, and combined with software engineering methods, such as AST syntax analysis, dependency retrieval, etc. for in-depth optimization, strive for excellence in each link, and finally integrate and implement for higher-precision code generation.

Algorithm level: In response to typical problems such as API conflicts caused by code version changes, OpenCSG proposes an adaptive teacher model. The teacher model analyzes code version change records and generates high-quality programming data for use. To improve the generation effect of the basic model. According to the evaluation, the improvements brought by these innovations are significantly better than the current RAG model, especially in popular project scenarios where the API structure is updated frequently. The relevant results of this part have been formed into papers and submitted to international conferences.

It is this two-pronged approach of algorithm + engineering and continuous improvement model that allows OpenCSG CodeGen Agent to stand out among other models.

StarShip的星海

If the real evaluation of CodeGen Agent is a small test, then StarShip carries the grand blueprint of OpenCSG. Regarding StarShip's product positioning, OpenCSG CEO Chen Ran said: "StarShip bears our vision of reshaping software development with large models. Users form their own digital employee teams through StarShip's built-in agents. CodeGen Agent is built into the platform Digital programmers, currently including CodeReview Agent code reviewers and CodeSearch code question and answer engineers, are different from code assistance tools. We hope that these digital employees can work directly without manual assistance intervention. We will release more types in the future. digital employees, comprehensively covering all aspects of requirements, design, coding, testing and operation and maintenance."

CTO Wang Wei said that this path is full of challenges but very interesting, "From first principles, large models. The improvement of productivity is no longer a question of 'yes' or 'no', but a question of when, where and in what form. StarShip is the answer we are trying to give."

Highly productive team, gathering sand into a tower

In addition to StarShip, the OpenCSG team is also quite productive: CSGHub open source model platform, wukong pre-training model, CSGCoder fine-tuning code model, etc. These products are well positioned and well received in the industry.

The rapid launch and iteration of these products not only meet market demand, but also serve a common goal: to empower everyone in every enterprise with large models. "To enable large models to empower every enterprise and every person, we need to make large models the same as water and electricity. If large models are electric energy, then CSGHub is the power network, and StarShip is various home appliances. Ultimately, Empowering thousands of households. ”

Embrace open source and transcend open source

OpenCSG’s philosophy is open source. As a company that insists on open source as its core, it not only achieves Open source the model, open source the code, and even open source the platform.

"We are a young company. We benefit from open source so that we can produce some results in a shorter time. At the same time, we will fully give back to the open source community. This is the basic principle of the open source community. In addition, I strongly agree with Sam Altman’s statement that open source is just a model, and product value is more important than the model,” CTO Wang Wei concluded.

"Benchmark itself is just a number. With the launch of GPT4-o, SWEBench's test scores are expected to exceed 30% soon, and optimistic estimates can exceed 50% next year. And we are more concerned about the factors behind these numbers. Product value: With the improvement of model capabilities and engineering technology, digital employees will change from quantitative changes to qualitative changes, from usable to easy to use, ushering in a comprehensive explosion in various industries." Wang Wei explained, "This may be the era of big models. A major change in the context, from companies to individuals, we must prepare for this.”

The above is the detailed content of Surpassing Devin, Yao Ban leads OpenCSG to set a new world record for large model programming. For more information, please follow other related articles on the PHP Chinese website!