Undergraduate students use 60 lines of code to teach you how to make a large GPT model by hand. The technical introduction is comparable to a tutorial.

Currently, large language models (LLM) are considered to be the direction of breakthroughs in artificial intelligence. People are trying to use them to do all kinds of complex things, such as question and answer, creation, mathematical reasoning, and writing code. The recent explosion of ChatGPT is the best example.

However, for machine learning practitioners, the threshold for large models is very high: because they are too large and difficult to train, this direction has been monopolized by large companies for a long time. However, recently, there are more and more ways to simplify the GPT model. In mid-January, former Tesla AI Senior Director Andrej Karpathy (now returning to OpenAI) released a complete tutorial on building a GPT model from scratch​​ ​. However, compared with the trained GPT and OpenAI's GPT-3, the size difference between the two is 10,000-1 million times. Recently, Jay Mody, a software engineering undergraduate student at McMaster University in Canada, implemented a GPT model from scratch with only 60 lines of code after importing the NumPy library. It's named PicoGPT. Not only that, he also loaded the trained GPT-2 model weights into his implementation and generated some text. 60 lines of code are shown below.

But to do this, you need to be familiar with Python and NumPy, and some Basic experience in training neural networks. The author stated that this blog aims to provide a simple, easy-to-understand and complete introduction to GPT. Therefore, the author only uses the already trained model weights to implement the forward pass code.

## Code address:

​https://github.com/jaymody/picoGPT/blob/29e78cc52b58ed2c1c483ffea2eb46ff6bdec785/gpt2_pico.py#L3-L58​

For this item Research, Andrej Karpathy gave four words: late but arrived. Back then, the minGPT and nanoGPT built by Karpathy required 300 lines of code.

It is worth mentioning that this tutorial is not completely zero-threshold. In order to let readers understand, the author first introduces what GPT is, its input, output and other contents, all in great detail.

## As for what GPT can do, the author gave several examples. It can write Email, summarize a book, give you instagram caption ideas, explain black holes to a 5 year old, write code in SQL, etc.

After reading this part carefully, you can roughly understand some basic knowledge of GPT. With this background information, the next step is how to set it up.

Project Introduction

Settings

##This chapter mainly introduces how to set up the encoder , hyperparameters and parameters.

##What you have to do is to clone the code base:

Then install the dependencies:

Note that if you are using an M1 Macbook, you need to change tensorflow to tensorflow-macos in requirements.txt before running pip install. Under this project, the files include encoder.py, utils.py, gpt2.py, gpt2_pico.py:

• ##encoder.py: Contains the code of OpenAI BPE Tokenizer, directly Taken from gpt-2 repo;
##utils.py: Contains code for downloading and loading GPT-2 model weights, tokenizers, and hyperparameters;
gpt2.py: Contains the GPT model and generated code, which can be run as a python script;
gpt2_pico.py: Same as gpt2.py, but with fewer lines of code.

gpt2.py needs to be implemented from scratch, so what you have to do is to delete gpt2.py first and create an empty file again:

##Then copy the following code into gpt2.py:

The above code contains 4 main parts:

gpt2 function is used to implement GPT this time Actual code;

• generate function implements the autoregressive decoding algorithm;
• main function;
• fire.Fire (main) Convert the file into a CLI application so that you can finally run the code: python gpt2.py "some prompt here".
The main function contains encode, hparams, and params parameters and executes the following code:

Then the necessary model and tokenizer files will be downloaded to the models/124M file.

After the setup is completed, the author begins to introduce some details of the encoder, hyperparameters, and parameters. Taking the encoder as an example, the encoder in this article is the same as the BPE tokenizer used by GPT-2. Here are some examples of text encoded and decoded by this encoder:

The actual token length This looks like:

Note that sometimes tokens are words (e.g. Not), sometimes they are words but preceded by a space (e.g. Ġall, Ġ represents a space), sometimes they are part of a word (e.g. capes is split into Ġcap and es), sometimes they are punctuation marks (e.g. .).

One benefit of BPE is that it can encode arbitrary strings, and if it encounters something that doesn't exist in the vocabulary, it will break it into substrings that it understands :

The more detailed content will not be repeated. Next, we will introduce the basic neural network. This part is even more basic, mainly including GELU, Softmax function, Layer Normalization and Linear.

There are code examples in each small section. For example, in the Linear section, the author shows the standard matrix multiplication offset:

The code for projecting a linear layer from one vector space to another vector space is as follows:

##GPT Architecture​

This part introduces the structure of GPT itself.

##Transformer structure is as follows:

Transformer uses only the decoder stack (right part of the diagram):

It should be noted that due to getting rid of the encoder, the middle cross-attention layer is also deleted. ​

At a high level, the GPT architecture has the following three parts:

Text position embedding;
Transformer decoder stack;
Project to vocabulary.

The code looks like this:

Code part screenshot

Next, each of the above three parts will be decomposed in more detail, and I will not go into details here.

The above is the author's implementation of GPT. The next step is to combine them together and run the code to get gpt2.py. Its entire content is only 120 lines of code (60 lines if you remove comments and whitespace).

The author tested the results by:

##The output results are as follows:

##As the author said: This experiment was successful.

This article only briefly introduces the overall process following the author's ideas. Friends who want to know more can refer to the original link.

Original link: https://jaykmody.com/blog/gpt-from-scratch/#basic-layers

The above is the detailed content of Undergraduate students use 60 lines of code to teach you how to make a large GPT model by hand. The technical introduction is comparable to a tutorial.. For more information, please follow other related articles on the PHP Chinese website!