Stability AI open source 3B code generation model: can be completed and debugged

On Monday, Stability AI open sourced the small-volume pre-training model Stable Code Instruct 3B.

Stable Code Instruct 3B is an instruction-adapted coding language model (Code LM) based on Stable Code 3B. By providing natural language prompts, the model can be applied to a variety of tasks, including code generation, mathematical problems, and other tasks related to software engineering.

Stability AI claims that their model shows state-of-the-art performance at scale of 3B, outperforming larger scale models such as CodeLlama’s 7B Instruct, on software engineering related tasks , and even has the same performance as StarChat’s 15B model.

• Model: https://huggingface.co/stabilityai/stable- code-instruct-3b
• HuggingFace Trial: https://huggingface.co/spaces/stabilityai/stable-code-instruct-3b
• Stable Code Technical Report: https://static1.squarespace.com/static/6213c340453c3f502425776e/t/6601c5713150412edcd56f8e/1711392114564/Stable_Code_TechReport_release.pdf

Stable Code Instruct 3B has upgraded its code completion function and supports natural language interaction, aiming to improve the efficiency and intuitiveness of programming and software development tasks. Experimental results show that this model performs well in various coding-related tasks, outperforming competing models such as Codellama 7B Instruct and DeepSeek-Coder Instruct 1.3B.

Method introduction

Stable Code is based on Stable LM 3B. Stable Code is a causal pure decoder transformer, similar to the LLaMA architecture. The main differences from LLaMA are as follows:

• Position embedding, rotation position embedding is applied to the front of the head embedding dimension 25% to improve throughput;
• Normalization, LayerNorm with learned bias terms;
• Deviation, except key, query and Bias in value projection, Stable Code removes all bias terms from feedforward networks and multi-head self-attention layers.

The following table gives the sampling weight, epoch, category and other information of the pre-training corpus data set.

##According to Stack Overflow 2023 Developer Survey Report, Stable Code Instruct 3B Key Points Focus on languages ​​like Python, Javascript, Java, C, C++, and Go, which are the most popular and influential for developers of all kinds. While these languages ​​were selected as the focus of training, the model was also trained on other widely adopted languages ​​such as SQL, PHP, and Rust.

Stable Code Instruct 3B is powerful even for languages ​​that were not originally included in the training set (such as Lua) test performance. This proficiency likely stems from an understanding of underlying coding principles and the ability to adapt concepts in different programming environments by taking advantage of the inherent predictability of coding tasks.

Stable Code Instruct 3B is proficient not only in code generation, but also in FIM (Fill in the Middle) tasks, database queries, code translation, interpretation and creation. Its instructions are tuned to enable it to understand and act on nuanced instructions, facilitating a wide range of coding tasks beyond simple code completion, including mathematical understanding, logical reasoning, and processing complex technical descriptions surrounding software development.

Performance Evaluation

Compared with leading models such as Codellama 7B Instruct and DeepSeek-Coder Instruct 1.3B, Stable Code Instruct 3B performs better in a series of Demonstrated superior performance in coding tasks.

The research team also compared the three models on the Multi-PL benchmark. Despite having fewer parameters, Stable Code Instruct 3B significantly outperformed CodeLlama Instruct on all languages.

##Table 8 below shows the performance of several models on the FIM task:

Experimental testing shows that Stable Code Instruct 3B matches or exceeds other models in code completion accuracy, understanding of natural language instructions, and ability to span different programming languages.

Stable Code Instruct 3B’s parameter size and low hardware requirements make it accessible to a wide audience, empowering developers Ability to work more efficiently. It’s worth mentioning that Stable Code Instruct 3B is now available for commercial purposes with a Stability AI membership.

The above is the detailed content of Stability AI open source 3B code generation model: can be completed and debugged. For more information, please follow other related articles on the PHP Chinese website!