Which AI programmer is the best? Explore the potential of Devin, Tongyi Lingma and SWE-agent

On March 3, 2022, less than a month after the birth of the world’s first AI programmer Devin, the NLP team of Princeton University developed an open source AI programmer SWE-agent. It leverages the GPT-4 model to automatically resolve issues in GitHub repositories. SWE-agent's performance on the SWE-bench test set is similar to Devin, taking an average of 93 seconds and solving 12.29% of the problems. By interacting with a dedicated terminal, SWE-agent can open and search file contents, use automatic syntax checking, edit specific lines, and write and execute tests. (Note: The above content is a slight adjustment of the original content, but the key information in the original text is retained and does not exceed the specified word limit.)

SWE-Agent’s Agent-Computer Interface (ACI) has been simplified and the large model The command and feedback formats have been redesigned to make it easier to browse the repository, view, edit, and execute code files. The research team also provides some useful features such as code inspector, file viewer and global catalog string search command to improve efficiency. These adjustments help improve productivity.

On the same day, Alibaba Cloud announced a special new employee-AI programmer "Tongyi Lingma", job number AI001. This AI programmer had been interning for several months inside and outside Alibaba Cloud before joining the ranks of regular employees, and received a favorable rating of more than 80%. The addition of Tongyi Ling Code shows that the addition of universal code has improved the work efficiency of engineers in writing, reading, finding BUGs and optimizing code. Its working ability is described as being on call 7x24 hours, greatly omitting the need for human beings. The amount of work programmers put into testing code.

Universal Lingma was jointly developed by Alibaba Cloud and Universal Labs. It has been downloaded more than 2 million times, and millions of lines of code are adopted every day. It is proficient in more than 200 programming languages ​​and proficient in 16 mainstream languages. Universal Lingma not only has skills such as code continuation, unit test generation, code optimization, etc., it can also perform intelligent question and answer and abnormal error reporting, reducing the complexity of troubleshooting work. In addition, it also supports special training for enterprise-level code bases, better understands enterprise code and document data, and supports localized adaptation.

In the wave of artificial intelligence, the emergence of AI programmers is undoubtedly an important milestone in the technological revolution. From the exploration of logical reasoning and expert systems in the early 1950s to the widespread application of deep learning and machine learning today, the development background of AI technology is multifaceted. Over the past few decades, the scope of AI research and applications has continued to expand. From the early stages of initially focusing on solving logical problems and simulating human thinking, AI technology has now been involved in various fields, including health care, finance, transportation, etc. The role of AI programmers

The rise of artificial intelligence programmers marks a major breakthrough in artificial intelligence technology in the field of programming. It can not only reshape the ecosystem of the software development industry, but also have far-reaching significance for our understanding of the potential and boundaries of artificial intelligence. The rise of AI programming may change the way programmers work and the skill requirements, but it will not reduce the importance of the programmer profession.

Analysis of technical characteristics of AI programmers

AI programmers, such as Devin, Tongle Lingma and Princeton University’s SWE-agent, are redefining the boundaries of software engineering. These systems not only demonstrate the huge potential of AI in the field of programming, but also reveal the possible future directions of the technology industry. AI programmers can accelerate software development cycles and improve code quality through machine learning and automation technologies. The result can be a more efficient and smarter development process, as well as more creative and innovative development directions.

Devin: Fully Automatic AI Software Engineer

Devin is the world’s first fully automatic AI software engineer. Its emergence marks an important milestone in the field of AI software development. Devin has long-term reasoning and planning skills and is able to perform complex engineering tasks. This means that Devin is not only able to handle simple coding tasks, but also manages the entire project life cycle, from requirements gathering to design, coding, testing and deployment. The significance of Devin is that it can not only handle simple coding tasks, but also manage the entire project life cycle, from requirements collection to design, coding, testing and deployment, and can also undertake complex engineering tasks.

Picture

Picture: OpenDevin is a replica of Devin with the goal of sharing power with the open source community. Through community contributions and a commitment to open standards, OpenDevin hopes to refine and expand the capabilities of AI-assisted software engineering.

Devin’s core strength is its ability to recall relevant context, learn over time, and correct errors. This ability allows Devin to continuously improve himself as the project progresses, reducing the possibility of human error. In addition, Devin is equipped with commonly used tools for developers, such as shells, code editors, and browsers. These tools are integrated in a sandbox computing environment to ensure security and flexibility.

Real-time collaboration with users is also a major feature of Devin. It reports progress in real time, accepts feedback, and makes design choices with users as needed. This collaborative model not only improves development efficiency, but also enhances the adaptability and innovation of the project.

Overview of Devin’s technical characteristics and capabilities: Devin is an AI software engineer developed by the Cognition AI team. It has the ability to program independently and can complete complex engineering tasks independently. Devin’s core technical features include:

1. Long-term reasoning and planning: Devin is able to plan and execute complex engineering tasks that require thousands of decisions.
2. Context-sensitive memory: At every step, Devin is able to recall relevant context, learn and correct mistakes.
3. Integration of development tools: Devin is equipped with development tools such as shell, code editor and browser.
4. Real-time collaboration: Devin can report progress, receive feedback, and make design choices with users in real-time.
5. Autonomous repair and optimization: Devin can independently discover and fix bugs in the code, and can also train and fine-tune the AI ​​model.

「Tongyi Lingma」: Advanced AI programming assistant

Tongyi Lingma is an AI programming assistant launched by Alibaba Cloud. Its goal is to improve the efficiency and efficiency of software development. quality. It helps programmers reduce repetitive work by automating programming tasks, giving them more time to focus on innovation and solving complex problems.

Picture

Tongyi Lingma’s future goal is to write 20% of the company’s code so that programmers can Focus more on system design and core business development. At the same time, Tongyi Lingma is also launched on the Tongyi Qianwen APP, which is free and open to the public, allowing anyone to write and learn code on their mobile phones. For students without programming foundation, Tongyi Qianwen can also provide study suggestions and formulate study plans. This marks that AI programmers are not only assistants to programmers, but may become important players in the field of software development.

Overview of the technical features and capabilities of "Tongyi Lingma"

1. Automated programming: Tongyi Lingma can assist programmers in writing, reading, searching and optimizing code. It can automatically generate code, including line-level/function-level code, unit tests, and code comments, thereby improving programming efficiency.
2. Multi-language support: The tool has become proficient in more than 200 programming languages, breaking the boundaries of programming languages ​​and enabling programmers to write code across languages.
3. Intelligent Q&A: Tongyi Lingma provides the ability to develop intelligent Q&A, which can quickly answer programming-related questions and help solve programming problems.
4. Code optimization: It can deeply analyze the code and its context, quickly identify potential coding problems, and provide specific optimization suggestions.

Princeton Open Source AI Programmer: SWE-agent

SWE-agent launched by Princeton University is the first open source AI programmer based on GPT-4 technology. As an open source project, SWE-agent can automatically fix bugs in the GitHub repository, which is a huge contribution to the open source community. On the SWE-bench test set, SWE-agent successfully solved 12.29% of the problems, taking an average of only 93 seconds, showing an accuracy comparable to Devin.

The open source nature of SWE-agent means that it can quickly obtain feedback and improvements from the community, which accelerates the progress of AI in the field of software engineering. The success of SWE-agent also shows that AI programmers can not only improve development efficiency, but also improve code quality and reduce maintenance costs.

Figure: SWE benchmark performance

Technical characteristics of SWE-agent

Open source nature SWE-agent, as an open source project, means Its code, design, and functionality are all public, providing developers around the world the opportunity to participate and contribute. It quickly gained 1.6k stars and 109 Forks on GitHub, showing the high recognition and interest of the open source community in this technology.

1. Agent-Computer Interface (ACI) based on GPT-4 SWE-agent uses the design of Agent-Computer Interface (ACI) to make large models such as GPT-4 more convenient to browse the repository, view, Edit and execute code files. This design is similar to human-preferred UI design, improving the efficiency and quality of interaction with large models.

2. Ability to automatically fix bugs SWE-agent can automatically fix bugs in the GitHub repository. On the SWE-bench test set, it successfully solved 12.29% of the problems, taking an average of only 93 seconds, and its accuracy was comparable to the previously launched AI programmer Devin. This demonstrates SWE-agent's efficient and accurate capabilities in automating software engineering tasks.

3. Interaction with a dedicated terminal SWE-agent works by interacting with a dedicated terminal. It can open and search file contents, use automatic syntax checking, edit specific lines, and can also write and execute tests. This type of interaction provides AI programmers with more autonomy and flexibility.

4. Functional design SWE-agent contains features that the research team found very useful during the design of the agent-computer interface, including adding a linter that runs when an edit command is issued, and if the code syntax is not correct Correct, the editing command will not be allowed to pass. Additionally, agents are provided with a purpose-built file viewer and directory-wide string search commands, all designed to increase productivity and accuracy.

5. Agent workflow The workflow of SWE-agent is divided into two stages: "reasoning" and "evaluation". During the inference phase, SWE-agent handles issues reported on GitHub and generates a pull request (Pull Request) aimed at fixing the issue. During the evaluation phase, the generated pull request is evaluated to confirm whether it actually solves the reported issue.

The emergence of AI programmers such as Devin, Tongyi Lingma and SWE-agent indicates that the future of software development will be more automated, intelligent and collaborative. The technical characteristics and capabilities of these systems demonstrate AI How to improve development efficiency, reduce error rates, and promote innovation.

AI Programmer Testing Effect

Testing by Devin AI Programmer showed that it is capable of a variety of programming tasks, including using an API to obtain stock prices and creating software that allows users to play chess with large models website. Devin does a great job handling API keys, package errors, and hint words for playing chess with LLM. However, its feedback speed still needs to be improved.

Tongyi Lingma operates similarly to GitHub Copilot, and users can easily switch from one tool to another. Although it is not as good as GitHub Copilot in terms of code writing capabilities, it provides free use, advantages in specific fields, and a good operating interface and adaptability. According to personal testing and evaluation, Tongyi Lingma can get a score of 3.5~4 points (out of 5 points), and is regarded as an AI coding assistance tool worth trying.

The SWE-agent launched by Princeton University is comparable to Devin in its ability to automatically fix bugs in GitHub repositories, taking an average of only 93 seconds. This open source project quickly gained community recognition and interest on GitHub.

The testing effect of AI programmers is a key indicator to evaluate its performance

1. Testing efficiency and accuracy. AI programmers improve the efficiency of software testing through automation technology and can quickly identify and Bug fixes. They utilize machine learning algorithms to continuously learn and optimize to improve the accuracy and coverage of bug discovery. This continuous learning capability allows AI programmers to continuously improve its performance during the software development process.

2. Automated bug discovery. AI programmers perform well in automated bug discovery. They can conduct in-depth analysis of software code and test data to identify hidden and complex bugs. This not only improves the quality of testing, but also saves a lot of time and labor costs.

3. Continuous performance testing, AI programmers can perform continuous performance testing to ensure the stability and reliability of the software system. They ensure software quality by simulating different test scenarios and evaluating software performance under various conditions.

4. Functional and performance testing. In terms of functional and performance testing, AI programmers can automatically run unit tests, integration tests and performance tests to ensure code quality. These tests are often automated and can be performed before the code is deployed to production or test environments.

5. Improve work efficiency. AI programmers can also help traditional test engineers improve work efficiency, such as assisting in writing test documents, test cases and test reports. Although AI programmers cannot completely replace human test engineers in use case design, they can already provide effective support in some aspects.

Based on the above multiple indicators, multiple evaluation results show that AI programmers Devin, Tongyi Lingma and SWE-agent perform differently in the test of software engineering tasks, each with its own characteristics.

Devin

In the SWE-bench basic test, Devin was able to solve 13.86% of the problems without human assistance.

Devin’s tests show that it can perform multi-step planning and receive environmental feedback, and 72% of the passed tests take more than 10 minutes to complete, showing its iterative capabilities.

Tongyi Lingma

Tongyi Lingma performs best in assisting Java coding, but in terms of SQL and code error correction, iFlyCode may be a better choice.

Tongyi Lingma supports code explanations in 30 languages, and can answer questions and answers about knowledge in specific fields, such as Alibaba Cloud OSS-related questions.

SWE-agent

SWE-agent is an open source software engineering agent that converts GitHub issues into pull requests, solving 12.29% of the issues in the SWE-bench test set.

SWE-agent implements the Agent Computer Interface (ACI) and makes it easier for LM to browse the warehouse, view, edit and execute code files by designing simple LM center commands and feedback formats.

Comprehensive evaluation, Devin has a strong ability to solve problems without assistance, Tongyi Lingma has advantages in specific fields such as programming education and code explanation, and SWE-agent, as an open source agent, is good at solving problems Excellent performance on GitHub issues. Each tool has its unique advantages and applicable scenarios.

AI programmer technology expansion

The expansion of AI programmer technology is constantly promoting changes in the field of software development. From automated programming to intelligent programming tools, to optimization algorithms and automated testing, AI programmer technology is becoming an indispensable part of software development.

Automated Programming Application

The development of AI technology has caused many traditional programming tasks to be replaced by automation. For example, AI can be used to automatically generate code, conduct automated testing, optimize algorithms, etc. This means that programmers will no longer need to spend a lot of time doing repetitive programming tasks, but can focus on more creative and complex tasks.

The development of intelligent programming tools

The emergence of AI programmers has promoted the development of coding assistance tools. These tools can provide intelligent suggestions, automatic error correction, code optimization and other functions to help programmers improve development quality and efficiency. With the continuous development of technologies such as machine learning and deep learning, programmers can use these tools to improve development efficiency and create more colorful applications.

Contribution to optimizing algorithm performance

AI programmers have great potential in algorithm optimization. They can automatically optimize the performance, resource utilization and efficiency of the program through learning and evolutionary algorithms, thereby improving the operating efficiency and user experience of the program. In addition, AI can automatically give optimization suggestions by analyzing the running status and performance data of the code, helping programmers improve software performance and user experience.

Interdisciplinary ability

The rise of AI programmers has also prompted programmers to develop interdisciplinary abilities. The development of the AI ​​field involves multiple disciplines, such as computer science, mathematics, statistics, etc. Programmers need to have interdisciplinary knowledge and skills to better understand and apply AI technology. This interdisciplinary ability helps programmers integrate knowledge from different fields and find more comprehensive and effective solutions when dealing with complex problems.

These applications of AI technology not only improve the efficiency and quality of programming, but also provide programmers with a broader space for innovation and exploration. As AI technology continues to advance, future programmers will be better able to realize its potential, solve practical problems, and promote further development of computer science.

The impact of AI programmers on human programmers

With the rapid development of AI technology, the impact of AI programmers on actual programmers is multifaceted, including both challenges and opportunities. Programmers can take advantage of AI programmers to improve their competitiveness and productivity by continuously learning and adapting to new technologies.

The impact of automation and intelligence

The emergence of AI programmers allows certain aspects of programming work to be automated and intelligent. For example, AI programmers can automatically generate code, reducing the possibility of human error and improving development efficiency. This may change the working methods and responsibilities of traditional programmers, allowing them to focus more on higher-level design and innovation.

Improving efficiency and accuracy

AI programmers may have the ability to generate code more quickly and accurately. With the assistance of AI programmers, programmers can develop and test applications more efficiently, reduce errors and debugging time, and improve overall development efficiency.

Improvement of creativity and innovation capabilities

The emergence of AI programmers may release the creativity and innovation capabilities of programmers to a certain extent. AI can handle repetitive coding tasks, giving programmers more time to think and solve more complex problems, thereby enhancing innovation capabilities.

Intelligent Optimization and Tuning

AI programmers can perform performance analysis and optimization suggestions on the code, identify potential bottlenecks and propose improvement plans to make the code execution more efficient. In addition, AI-driven testing tools can automatically generate and execute unit tests and integration tests to ensure code quality and reduce regression errors.

Continuous integration and test automation

AI programmers’ technology can support complex decision-making and reasoning, automatically generate and execute tests, thereby improving software reliability and performance. This kind of automation not only improves the efficiency of testing, but also helps to detect and fix bugs at an early stage, reducing problems after the software is released.

Long-term impact on the career of programmers

Although the development of AI technology may lead to the reduction of some programming positions, it will also create new employment opportunities and development space. Traditional programmers can adapt to new technological development trends and remain competitive by continuously learning and improving their skills.

In short, the impact of AI programmers on actual programmers is profound and complex. Not only do they improve the efficiency and quality of programming, they may also change programmers' work content and career paths. In the future, programmers may need to focus more on innovation, design, and the ability to solve complex problems rather than just coding skills.

The last wordy words

AI programmers Devin, Tongyi Lingma and SWE-agent demonstrated the significant progress of artificial intelligence in the field of software development, improving programming efficiency and promoting technological innovation . The advantages of AI programmers are obvious. AI programmers can automatically generate code, reducing the time and energy of traditional manual coding. They can quickly learn large amounts of codes and patterns, improve coding efficiency, and reduce the occurrence of human errors. AI programmers continue to learn and evolve themselves through technologies such as deep learning and reinforcement learning. They can quickly adapt to new programming languages, frameworks, and technologies. Colleague AI programmers have large-scale task processing and parallel computing capabilities, as well as intelligent and automatic optimization capabilities.

But for now, AI programmers also have some limitations, such as insufficient creativity and innovation capabilities, making it difficult for AI programmers to carry out creative innovation and design. There are also limited contextual understanding and reasoning capabilities. AI programmers have limitations in understanding and processing complex contextual situations. They may not be able to accurately understand human intentions and domain-specific knowledge. Data dependence and generalization ability are also its shortcomings. The performance and effect of AI programmers are limited by the quality and coverage of training data, and their generalization ability may be insufficient when facing new fields or new scenarios.

On the positive side, the degree of automation of AI programmers will be further improved, and its explainability and transparency will also be enhanced. This will help programmers and relevant stakeholders better understand and control the behavior of AI programmers and improve the reliability and security of code. At the same time, AI programmers and human programmers will form a complementary partnership to jointly create more efficient, intelligent and innovative software solutions. In this process, programmers need to adapt to changes and form a win-win cooperative relationship with AI programmers through continuous learning, upgrading skills and innovative thinking. In this way, we can gain a foothold and play an important role in the AI ​​era, and jointly welcome the arrival of the AI ​​era. (END)

References:

1.https://news.opensauced.pizza/open-source-projects-that-are-gaining-steam-that-you-havet- heard-of/

2. https://www.php.cn/link/6def9cdc404151fd9c44071edea3d735

##

The above is the detailed content of Which AI programmer is the best? Explore the potential of Devin, Tongyi Lingma and SWE-agent. For more information, please follow other related articles on the PHP Chinese website!