How to create your own Shell with Python (Part 1)

I would like to know how a shell (like bash, csh, etc.) works internally. So in order to satisfy my curiosity, I used Python to implement a Shell called yosh (Your Own Shell). The concepts introduced in this article can be applied to other programming languages ​​as well.

(Tip: You can find the source code used in this blog post here, the code is released under the MIT license. On Mac OS X 10.11.5, I tested with Python 2.7.10 and 3.4.3 . It should run on other Unix-like environments, such as Cygwin on Linux and Windows)

Let's get started.

Step 0: Project Structure

For this project, I used the following project structure.

yosh_project
|-- yosh
   |-- __init__.py
   |-- shell.py

yosh_project is the project root directory (you can also simply name it yosh).

yosh is the package directory, and __init__.py can make it a package with the same directory name as the package (if you are not writing in Python, you can ignore it .)

shell.py is our main script file.

Step 1: Shell Loop

When you start a shell, it displays a command prompt and waits for your command input. After receiving the entered command and executing it (explained in detail later in the article), your shell will come back here and loop waiting for the next instruction.

In shell.py, we will start with a simple main function, which calls the shell_loop() function, as follows:

def shell_loop():
    # Start the loop here
def main():
    shell_loop()
if __name__ == "__main__":
    main()

Then, in shell_loop() In order to indicate whether the loop continues or stops, we use a status flag. At the beginning of the loop, our shell will display a command prompt and wait for command input to be read.

import sys
SHELL_STATUS_RUN = 1
SHELL_STATUS_STOP = 0
def shell_loop():
    status = SHELL_STATUS_RUN
    while status == SHELL_STATUS_RUN:
        ### 显示命令提示符
        sys.stdout.write('> ')
        sys.stdout.flush()
        ### 读取命令输入
        cmd = sys.stdin.readline()

After that, we split the command (tokenize) input and execute (we are about to implement the tokenize and execute functions).

So, our shell_loop() will look like this:

import sys
SHELL_STATUS_RUN = 1
SHELL_STATUS_STOP = 0
def shell_loop():
    status = SHELL_STATUS_RUN
    while status == SHELL_STATUS_RUN:
        ### 显示命令提示符
        sys.stdout.write('> ')
        sys.stdout.flush()
        ### 读取命令输入
        cmd = sys.stdin.readline()
        ### 切分命令输入
        cmd_tokens = tokenize(cmd)
        ### 执行该命令并获取新的状态
        status = execute(cmd_tokens)

This is our entire shell loop. If we start our shell using python shell.py it will display the command prompt. However, if we enter the command and press enter, it will throw an error because we haven't defined the tokenize function yet.

To exit the shell, try typing ctrl-c. Later I'll explain how to exit the shell gracefully.

Step 2: Command segmentation tokenize

When the user enters a command in our shell and presses the Enter key, the command will be a long character containing the command name and its parameters. string. Therefore, we have to shard the string (split a string into multiple tuples).

It seems simple at first glance. We may be able to use cmd.split() to split the input with spaces. It works for commands like ls -a my_folder because it splits the command into a list ['ls', '-a', 'my_folder'] so that we Then you can handle them easily.

However, there are also situations like echo "<span class="wp_keywordlink">Hello World</span>" or echo 'Hello World' where parameters are quoted in single or double quotes. . If we use cmd.spilt, we will get a list of 3 tokens ['echo', '"Hello', 'World"'] instead of a list of 2 tokens ['echo', 'Hello World'].

Fortunately, Python provides a library called shlex, which can help us split commands like magic. (Tip: We can also use regular expressions, but it's not the focus of this article.)

import sys
import shlex
...
def tokenize(string):
    return shlex.split(string)
...

We then send these tuples to the executing process.

Step 3: Execute

This is the core and interesting part of the shell. What exactly happens when the shell executes mkdir test_dir? (Tip: mkdir is an executable program with test_dir parameters, used to create a directory named test_dir.)

execvp is the first function required in this step. Before we explain what execvp does, let's see it in action.

import os
...
def execute(cmd_tokens):
    ### 执行命令
    os.execvp(cmd_tokens[0], cmd_tokens)
    ### 返回状态以告知在 shell_loop 中等待下一个命令
    return SHELL_STATUS_RUN
...

Try running our shell again and enter the mkdir test_dir command, then press the Enter key.

After we hit the Enter key, the problem is that our shell will exit directly instead of waiting for the next command. However, the directory is created correctly.

So, what does execvp actually do?

execvp 是系统调用 exec 的一个变体。第一个参数是程序名字。v 表示第二个参数是一个程序参数列表（参数数量可变）。p 表示将会使用环境变量 PATH 搜索给定的程序名字。在我们上一次的尝试中，它将会基于我们的 PATH 环境变量查找mkdir 程序。

（还有其他 exec 变体，比如 execv、execvpe、execl、execlp、execlpe；你可以 google 它们获取更多的信息。）

exec 会用即将运行的新进程替换调用进程的当前内存。在我们的例子中，我们的 shell 进程内存会被替换为 mkdir 程序。接着，mkdir 成为主进程并创建 test_dir 目录。最后该进程退出。

这里的重点在于我们的 shell 进程已经被 mkdir 进程所替换。这就是我们的 shell 消失且不会等待下一条命令的原因。

因此，我们需要其他的系统调用来解决问题：fork。

fork 会分配新的内存并拷贝当前进程到一个新的进程。我们称这个新的进程为子进程，调用者进程为父进程。然后，子进程内存会被替换为被执行的程序。因此，我们的 shell，也就是父进程，可以免受内存替换的危险。

让我们看看修改的代码。

...
def execute(cmd_tokens):
    ### 分叉一个子 shell 进程
    ### 如果当前进程是子进程，其 `pid` 被设置为 `0`
    ### 否则当前进程是父进程的话，`pid` 的值
    ### 是其子进程的进程 ID。
    pid = os.fork()
    if pid == 0:
    ### 子进程
        ### 用被 exec 调用的程序替换该子进程
        os.execvp(cmd_tokens[0], cmd_tokens)
    elif pid > 0:
    ### 父进程
        while True:
            ### 等待其子进程的响应状态（以进程 ID 来查找）
            wpid, status = os.waitpid(pid, 0)
            ### 当其子进程正常退出时
            ### 或者其被信号中断时，结束等待状态
            if os.WIFEXITED(status) or os.WIFSIGNALED(status):
                break
    ### 返回状态以告知在 shell_loop 中等待下一个命令
    return SHELL_STATUS_RUN
...

当我们的父进程调用 os.fork() 时，你可以想象所有的源代码被拷贝到了新的子进程。此时此刻，父进程和子进程看到的是相同的代码，且并行运行着。

如果运行的代码属于子进程，pid 将为 0。否则，如果运行的代码属于父进程，pid 将会是子进程的进程 id。

当 os.execvp 在子进程中被调用时，你可以想象子进程的所有源代码被替换为正被调用程序的代码。然而父进程的代码不会被改变。

当父进程完成等待子进程退出或终止时，它会返回一个状态，指示继续 shell 循环。

运行

现在，你可以尝试运行我们的 shell 并输入 mkdir test_dir2。它应该可以正确执行。我们的主 shell 进程仍然存在并等待下一条命令。尝试执行 ls，你可以看到已创建的目录。

但是，这里仍有一些问题。

第一，尝试执行 cd test_dir2，接着执行 ls。它应该会进入到一个空的 test_dir2 目录。然而，你将会看到目录并没有变为 test_dir2。

第二，我们仍然没有办法优雅地退出我们的 shell。

我们将会在下篇解决诸如此类的问题。

The above is the detailed content of How to create your own Shell with Python (Part 1). For more information, please follow other related articles on the PHP Chinese website!