Stability AI releases stable code 3B model that runs locally and does not require a GPU

Stability AI recently released its first AI model in 2024, called Stable Code 3B. This model has 3 billion parameters and focuses on auxiliary code tasks.

Runs natively on a laptop without the need for a dedicated GPU, while still delivering competitive performance with large models like Meta’s CodeLLaMA 7B.

At the end of 2023, Stability AI began to promote the development of smaller, more compact, and more powerful models, such as the StableLM Zephyr 3B model for text generation.

In early 2024, Stability AI released an important language model called Stable Code 3B. In fact, its preview version, Stable Code Alpha 3B, was released as early as August last year. Since then, Stability AI has continued to improve the technology. This new version of Stable Code 3B is designed specifically for code completion and also has a variety of additional features.

Compared with CodeLLaMA 7b, Stable Code 3B is 60% smaller in size, but achieves performance comparable to the former on programming tasks.

Stable Code 3B achieves SOTA performance (compared to similarly sized models) on the MultiPL-E benchmark, such as Stable Code 3B in Python , performance is better than StarCoder on C, JavaScript, Java, PHP and Rust programming languages.

Research introduction

Stable Code 3B is trained based on Stable LM 3B. The number of training tokens is as high as 4 trillion. In addition, Stable Code also uses data specific to software engineering for training.

Stable Code 3B provides more features, performs well even across multiple languages, and also has other advantages, such as supporting FIM (Fill in the Middle, a new training techniques) function, and can also expand the context size. The basic Stable Code is trained on up to 16,384 token sequences and follows a similar approach to CodeLlama, that is, using Rotary Embeddings, which optionally allows modification of up to 1,000,000 rotation bases. The context length of the model is further extended to 100k tokens.

In terms of model architecture, the Stable Code 3B model is a pure decoder transformer, similar to the LLaMA architecture, with the following modifications:

• Positional embedding: Rotated positional embedding is applied to the first 25% of the head embedding dimensions to improve throughput;
• Tokenizer: Use A modified version of GPTNeoX Tokenizer.NeoX, adding special tokens to train FIM functions, such as , , etc.

Training

Training data set

The training data set of Stable Code 3B is composed of a filtered mixture of open source large-scale data sets provided on HuggingFace Hub, including Falcon RefinedWeb, CommitPackFT, Github Issues, and StarCoder, and is further supplemented with data from the field of mathematics.

Training Infrastructure

• ##Hardware: Stable Code 3B at Stability AI 256 NVIDIA A100 40GB GPUs are used on the cluster for training.
• Software: Stable Code 3B adopts a branch of gpt-neox, uses ZeRO-1 to train under 2D parallelism (data and tensor parallelism), and relies on flash-attention, SwiGLU ,FlashAttention-2’s rotation embedding kernel.

Finally, let’s take a look at the performance of Stable Code 3B:

A more detailed technical report on Stable Code 3B will be released later, so you can look forward to it.

The above is the detailed content of Stability AI releases stable code 3B model that runs locally and does not require a GPU. For more information, please follow other related articles on the PHP Chinese website!