Three PK games exposed the level of ChatGPT!

​Author|Xu Jiecheng

Reviewer|Yan Zheng

While Google and Microsoft are fiercely competing to seize the AI ​​search engine market, some programmers are Not interested in eating melon. Because they have realized in advance that when this search war comes to an end, the software giants will continue to carry ChatGPT or other artificial intelligence generation tools and stride towards the blue ocean of automatic coding. By then, not to mention being a loser, even your own job may be threatened.

In today's automatic coding field, the two most mature and well-known AIs are ChatGPT, which has been in the limelight recently, and Copilot, the AI ​​programming tool launched by Microsoft in June last year. , and these "two" are also the popular stars who are most expected to replace programmers at this stage. So for now, what are the coding capabilities of ChatGPT and Copilot? Will they really replace all junior and even some intermediate developers in the short term as rumored?

Recently, an American technical expert used multiple JavaScript function requirements to test the capabilities of ChatGPT and Copilot in data processing and algorithm generation. Next, let us use these results to understand the current real level of AI in coding, and then ask ourselves, will we really be replaced?

1. The JavaScript function accepts a variable number of arrays and returns the intersection

In the first test, the tester first asked ChatGPT and Copilot to generate a JavaScript function, with specific conditions For: Need to be able to accept a variable number of arrays and return their intersection.

OpenAI ChatGPT:

##Microsoft Copilot:

The function generated by ChatGPT assumes that providing less than an array is invalid. By using Set, ChatGPT ensures that there are no duplicates in the results. Intersection should be a set operation and duplicates should be removed. The Copilot code returns an array that may contain duplicates.

Neither ChatGPT nor Copilot sorts the original parameters in ascending order according to length. This is a trivial optimization, but it can bring about huge changes. If any argument has length 0, there is no intersection; it shortens the loop anyway because the maximum intersection is the same as the shortest array argument.

Subsequently, the testers asked ChatGPT and Copilot to improve the execution efficiency of the function.

OpenAI ChatGPT:

##Microsoft Copilot:

Faced with the above problem, Copilot generated the same code as the previous request. ChatGPT gave a different answer and added a comment saying that the function does not work on objects as expected, but this description is not accurate.

Then, the testers used the same method to check the operating efficiency and memory consumption of the code generated by the two fastest intersection libraries provided by ChatGPT and Copilt.

The code generated by ChatGPT takes up less CPU when executed, but the running efficiency is not ideal, and the code generated by Copilot Although the code uses less heap, its CPU usage and operating efficiency are poor.

In short, in this test, neither ChatGPT nor Copilot can generate efficient enough code; ChatGPT made wrong assumptions in this question; and the function generated by Copilot has parameters When duplicate values ​​are included, code is generated that does not produce a collection.

2. JavaScript function: Cartesian product

The second test requires ChatGPT and Copilot to complete a Cartesian product JavaScript function.

OpenAI ChatGPT:

##Microsoft Copilot:

Anyone familiar with Cartesian products will know that from the perspective of memory utilization and performance, the results generated by ChatGPT and Copilot are explosive. A simple implementation would consume a lot of RAM to store all combinations and not return the result until all combinations have been generated. The functions generated by ChatGPT and Copilot both have these shortcomings.

Subsequently, testers once again asked ChatGPT and Copilot to improve function efficiency.

OpenAI ChatGPT:

##Microsoft Copilot :

In response to this demand, ChatGPT’s performance is surprising. But in the overall function, ChatGPT made a serious mistake. yield [item,...result] is not inside the generator, but in a recursion. Copilot simply ignored the change in demand and returned the same results as before.

In terms of code running efficiency and memory consumption, the performance of ChatGPT and Copilot is as shown in the table below.

Overall, neither ChatGPT nor Copilot can generate the correct code for the Cartesian product function; ChatGPT will make a code that may be invalid. Assumptions, such as requiring two parameters; although the detection results show that the code generated by ChatGPT is more memory efficient, it does not run smoothly at all.

3. JavaScript function stores objects and original parameters

In the third round, the testers asked the two to generate JavaScript functions that can store objects and original parameters.

OpenAI ChatGPT:

##Microsoft Copilot:

In this regard, both ChatGPT and Copilot generated relatively inefficient codes. It is very efficient to convert strings first and then compare strings. Poor and consumes a lot of memory.

Although there are some JavaScript values ​​that cannot be converted to strings, such as Infinity and NaN. Unfortunately, the JavaScript JSON specification was defined before the era of data science and microservices, and these values ​​exist mainly so that when certain error conditions occur in the code, the program can also use specific values ​​to represent the results produced.

Finally, in order to verify the function efficiency, the tester compared the code generated by ChatGPT and Copilot with the commonly used caching tools nano-memoize and micro-memoize, and used the following code to generate the first 12 Fibonacci numbers.

# Among them, nano-memoize is the most efficient one , which is almost twice as efficient as the code generated by ChatGPT and Copilot, and the memory used is also the lowest, while the performance of micro-memoize can be said to be closely followed. Although Copilot performs well in terms of CPU utilization, overall, the performance of ChatGPT and Copilot in this test is still not enough to defeat a mature programmer.

4. Summary and prediction

Through these three tests, we can easily find that although the code generated using ChatGPT and Copilot is definitely Has a certain value. But for now, neither ChatGPT nor Copilot can generate sufficiently accurate and efficient code from simple task descriptions, and even in some cases, they make some very bad mistakes. After learning about this result, many developers also expressed that they felt that they could persist for a few more years.

For today's enterprises or programmers, if you want to use ChatGPT, Copilot or other code generation tools to help you complete some simple auxiliary coding tasks to speed up the build, then You can definitely get enough support. But if you want to rely on them to completely liberate R&D, then you may need to spend a lot of money to equip it with a strong debugging team.

However, even if this is the result, today we still cannot ignore the potential of AI in the field of automatic coding and the powerful software companies behind these systems. What is certain is that as the amount of training and technology maturity increases, future automatic coding tools will continue to expand their business data in different scenarios, and gradually try to solve some more professional and scenario-based practical tasks.

Finally, as to the question "Can AI replace programmers in the future?", perhaps the most reliable answer at present is the former head of Alibaba Israel Machine Vision Laboratory Itamar Friedman once said The prediction made in the interview - "In the next 10 to 20 years, artificial intelligence systems will make it possible for non-programmer creators to use natural language instructions to develop without errors, and our world will still need a lot of programmer, but his role may change in unpredictable ways."

Reference link:

https://medium.com/@anywhichway/chatgpt- vs-copilot-vs-programmers

https://github.com/anywhichway/nano-memoize

https://github.com/planttheidea/micro-memoize

The above is the detailed content of Three PK games exposed the level of ChatGPT!. For more information, please follow other related articles on the PHP Chinese website!