AI virtual gestures to play airplane battle

Hello, everyone.

Let’s break down this small project and take you through the implementation step by step. At the end of the article, we will obtain the complete source code of the project.

1. Prepare the aircraft war program

Find a Python version of the aircraft war program on Github, install Pygame and run it.

The game operation is simple. In the upper right corner is the game pause/start button, which can be operated by clicking with the mouse.

The A, D, W, and S keys on the keyboard are used to control the movement direction of the aircraft, corresponding to left, right, up, and down respectively.

So our AI program must complete two core functions. First, recognize gestures; second, convert gestures into mouse and keyboard operations to control the game.

2. Recognize gestures

Here, we use opencv to read the video stream from the camera.

Send each frame in the video stream to the palm detection model in mediapipe to identify 21 key points of the palm.

In our project, only the index finger and middle finger are used, which are the 8th and 12th points on the left and right.

The core code is as follows:

ret, frame = cap.read()
results = hands.process(frame[:, :, ::-1])
if results.multi_hand_landmarks:
# 遍历每个手掌
for hand_landmarks in results.multi_hand_landmarks:
finger_axis_8 = hand_landmarks.landmark[8]
finger_axis_12 = hand_landmarks.landmark[12]

frame​ is each frame in the video stream, and hands is the palm detection model.

The parsed finger_axis_8 and finger_axis_12 objects store the x and y coordinates of the index finger and middle finger respectively.

3. Gesture control to pause the game

Calculate the distance between the coordinates of the index finger and middle finger. If it is greater than a certain threshold, move the mouse and click the pause game button.

Game pause

If the distance between the coordinates of the index finger and middle finger is less than a certain threshold, move the mouse and click the start game button

Game Start

Distance calculation is very simple. I won’t post the code here. The focus is on Python controlling the mouse.

I use the PyUserInput library, which provides two classes, PyMouse and PyKeyboard, to control the mouse and keyboard respectively.

When we want to use a Python program to control the pause and start of the game, we only need to move the mouse to the position of the button and perform a click operation.

# 定义鼠标对象
self.mouse = PyMouse()

def pause_or_start_game(self, dist):
"""
判断是否需要暂停（开始）游戏
:param dist:
:return:
"""
if (not self.is_pause and dist > 80) or (self.is_pause and dist < 80):
self.mouse.move(915, 125)
self.mouse.click(915, 125)

self.is_pause = not self.is_pause

The parameter dist of the pause_or_start_game​function is the distance between the index finger and the middle finger.

The coordinates of the pause/start button are (915, 125). The coordinates of each computer are different. You need to recalculate according to your actual situation.

The calculation idea is very simple. The game border size is (480, 700)​, and the game is started in the middle of the screen. As long as the size of the screen is obtained, the coordinates of the buttons can be roughly estimated. After calling PyMouse's move function, check and fine-tune it.

The move function of PyMouse is used to move the mouse position, and the click function is used to perform mouse click operations.

4. Gesture control of aircraft movement

Here, you need to calculate the movement direction and distance of the x coordinate and y coordinate of the index finger in two adjacent frames. This determines which of the keyboards A, D, W, and S is pressed.

Similarly, the direction and distance of movement are very simple and will not be discussed here. The focus is on the control of keyboard keys by the PyKeyboard module.

self.key_board = PyKeyboard()
# 按下按键
self.key_board.press_key(key)
# 停留一段时间
time.sleep(press_dwell)
# 释放按键
self.key_board.release_key(key)

Between the press_key and release_key functions, time.sleep(press_dwell) is called to control the duration of the key. If the key is pressed for a long time, the distance the aircraft moves will be long. On the contrary, if the key time is short, the distance the aircraft will move will be short.

So, the difficulty here is how to map the movement distance of the index finger to the duration of the key press.

I used the following code to measure it

for i in range(n):
kb.press_key('A')
time.sleep(0.05)
kb.release_key('A')

The fixed key duration is 0.05 seconds, and the test aircraft moves from the middle to the far left, which requires the smallest n.

The aircraft moves from the middle to the far left, and the moving distance is 240. Therefore, 240 / (n * 0.05) is the moving distance of the aircraft per second.

I measured n=7​, therefore, the distance the aircraft moves per second is 685.7142857.

As long as you calculate the movement distance of your index finger and divide it by 685.7142857, you can get the key duration of the keyboard.

The complete code for gesture control of aircraft movement is:

def press_key_board(self, direction, move_dist):
"""
将手指移动距离，换算为按键间隔，并执行按键操作
:param direction:移动方向
:param move_dist:移动距离
:return:
"""
dist_per_sec = 685.7142857
if direction == 'x':
key = 'A' if move_dist < 0 else 'D'
elif direction == 'y':
key = 'W' if move_dist < 0 else 'S'
else:
return

press_dwell = math.fabs(move_dist / dist_per_sec)
self.key_board.press_key(key)
time.sleep(press_dwell)
self.key_board.release_key(key)

The core part of the project has been explained, and the complete code has been sorted out. If you need it, just leave a message in the comment area.

After obtaining the code, first look at the running steps.txt.

If you think this article is useful to you, please click "Read it" to encourage me. I will continue to share excellent Python AI projects in the future.

The above is the detailed content of AI virtual gestures to play airplane battle. For more information, please follow other related articles on the PHP Chinese website!