Machine Learning Decision Tree Practical Exercise

Translator | Zhu Xianzhong

##Reviewer | Sun Shujuan

Decision tree in machine learning

Modern machine learning algorithms are changing our daily lives. For example, large language models like BERT are powering Google search, and GPT-3 is powering many high-level language applications.

On the other hand, building complex machine learning algorithms is much easier today than ever before. However, no matter how complex a machine learning algorithm may be, they all fall into one of the following learning categories:

• Supervised learning
• Unsupervised learning
• Semi-supervised learning
• Reinforcement learning

In fact, Decision trees are one of the oldest supervised machine learning algorithms and can solve a wide range of real-world problems. Research shows that the earliest invention of the decision tree algorithm can be traced back to 1963.

Next, let’s delve into the details of this algorithm and see why this type of algorithm is still so popular today.

What is a decision tree?

The decision tree algorithm is a popular supervised machine learning algorithm because of its relatively simple method of processing complex data sets. Decision trees get their name from their similarity to the structure of a tree; a tree structure consists of several components such as roots, branches, and leaves in the form of nodes and edges. They are used for decision analysis, much like an if-else based decision flow chart, where decisions will produce the desired predictions. Decision trees can learn these if-else decision rules to split the data set and finally generate a tree-like data model.

Decision trees have been used in the prediction of discrete results for classification problems and the prediction of continuous numerical results for regression problems. Over the years scientists have developed many different algorithms such as CART, C4.5 and ensemble algorithms such as random forests and gradient boosted trees.

Analyzing the various components of a decision tree

The goal of the decision tree algorithm is to predict the outcome of the input data set. The tree data set is divided into three forms: attributes, attribute values, and types to be predicted. As with any supervised learning algorithm, the data set is divided into two types: training set and test set. Among them, the training set defines the decision rules that the algorithm learns and applies to the test set.

Before we gather together the steps of the decision tree algorithm, let us first understand the components of the decision tree:

• Root Node: It is the starting node at the top of the decision tree and contains all attribute values. The root node is divided into decision nodes based on the decision rules learned by the algorithm.
• Branches: Branches are connectors between nodes that correspond to attribute values. In binary splitting, the branches represent true and false paths.
• Decision Node/Internal Node: Internal node is the decision node between the root node and leaf node, corresponding to the decision rule and its answer path. Nodes represent questions, and branches show paths to relevant answers based on those questions.
• Leaf nodes: Leaf nodes are terminal nodes that represent target predictions. These nodes will not be split further.

The following is a visual representation of a decision tree and its above components, the decision tree algorithm goes through the following steps to arrive at the desired prediction:

• The algorithm starts from the root node with all attribute values.
• The root node is divided into decision nodes based on the decision rules learned by the algorithm from the training set.
• Pass internal decision nodes through branches/edges based on the question and its answer path.
• Continue the previous steps until you reach a leaf node or all attributes are used.

In order to select the best attribute on each node, one of the following two attribute selection metrics will be used for splitting:

• ​​Gini coefficient​​（Gini index）Measurement of Gini impurity (Gini Impurity) to indicate the likelihood that the algorithm will misclassify a random class label.
• ​​Information gain​​Measures the improvement in entropy after segmentation to avoid predicting class 50/ 50 split. Entropy is a mathematical measure of the impurity in a given data sample. The chaotic state in the decision tree is represented by a partition close to 50/50 .
Flower classification case using decision tree algorithm

After understanding the above basic knowledge, let us start to implement an application case. In this article, we will implement a decision tree classification model in Python using Scikit Learning library.

A brief explanation about the data set

The data set for this tutorial is an iris data set. This dataset is already built into the Scikit open source library, so developers do not need to load it externally. The dataset includes a total of four iris attributes and corresponding attribute values, which will be input into the model to predict one of three types of iris flowers.

Attributes/Features in the dataset: sepal length, sepal width, petal length, petal width.
Predicted labels/flower types in the dataset: Setosis, Versicolor, Virginica.

#Next, a step-by-step code description of the decision tree classifier based on the python language will be given.

Import library

First, import the library required to implement the decision tree through the following piece of code.

import pandas as pd
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

Loading the iris data set

The following code shows the use of the load_iris function to load the iris data in the sklearn.dataset library stored in the data_set variable set. The next two lines of code will print the iris type and characteristic information.

data_set = load_iris()
print('Iris plant classes to predict: ', data_set.target_names)
print('Four features of iris plant: ', data_set.feature_names)

Separating attributes and tags

The following lines of code separate the characteristics and type information of the flower, and Store them in the corresponding variables. Among them, the shape[0] function is responsible for determining the number of attributes stored in the X_att variable; the total number of attribute values ​​in the data set is 150.

#提取花的特性和类型信息
X_att = data_set.data
y_label = data_set.target
print('数据集中总的样本数：', X_att.shape[0])

In fact, we can also create a visual table to display some attribute values ​​​​in the data set by adding the value in the X_att variable to the DataFrame function in the panda library. .

data_view=pd.DataFrame({
'sepal length':X_att[:,0],
'sepal width':X_att[:,1],
'petal length':X_att[:,2],
'petal width':X_att[:,3],
'species':y_label
})
data_view.head()

Split the data set

The following code shows the use of the train_test_split function to split the data set into two parts: a training set and a test set. Among them, the random_state parameter in this function is used to provide a random seed for the function to provide the same results for the given data set every time it is executed; test_size indicates the size of the test set; 0.25 indicates that the test data accounts for 25% after splitting. Training data accounts for 75%.

#数据集拆分为训练集和测试集两部分
X_att_train, X_att_test, y_label_train, y_label_test = train_test_split(X_att, y_label, random_state = 42, test_size = 0.25)

Applying the decision tree classification function

The following code creates a

​​ by using the DecisionTreeClassifier function Classification model​​to implement a decision tree, classification standard is set to "entropy"Way. This standard enables to set the attribute selection metric to (Information gain). The code then matches the model to our training set of attributes and labels.

#应用决策树分类器
clf_dt = DecisionTreeClassifier(criterion = 'entropy')
clf_dt.fit(X_att_train, y_label_train)

计算模型精度

下面的代码负责计算并打印决策树分类模型在训练集和测试集上的准确性。为了计算准确度分数，我们使用了predict函数。测试结果是：训练集和测试集的准确率分别为100%和94.7%。

print('Training data accuracy: ', accuracy_score(y_true=y_label_train, y_pred=clf_dt.predict(X_att_train)))
print('Test data accuracy: ', accuracy_score(y_true=y_label_test, y_pred=clf_dt.predict(X_att_test)))

真实世界中的决策树应用程序

当今社会，机器学习决策树在许多行业的决策过程中都得到广泛应用。其中，决策树的最常见应用首先是在金融和营销部门，例如可用于如下一些子领域：

• 贷款批准
• 支出管理
• 客户流失预测
• 新产品的可行性分析，等等。

如何改进决策树？

作为本文决策树主题讨论的总结，我们有充分的理由安全地假设：决策树的可解释性仍然很受欢迎。决策树之所以容易理解，是因为它们可以被人类以可视化方式展现并便于解释。因此，它们是解决机器学习问题的直观方法，同时也能够确保结果是可解释的。机器学习中的可解释性是我们过去讨论过的一个小话题，它也与即将到来的人工智能伦理主题存在密切联系。

与任何其他机器学习算法一样，决策树自然也可以加以改进，以避免过度拟合和出现过于偏向于优势预测类别。剪枝和ensembling技术是克服决策树算法缺点方案最常采用的方法。决策树尽管存在这些缺点，但仍然是决策分析算法的基础，并将在机器学习领域始终保持重要位置。

译者介绍

朱先忠，51CTO社区编辑，51CTO专家博客、讲师，潍坊一所高校计算机教师，自由编程界老兵一枚。

原文标题：​​An Introduction to Decision Trees for Machine Learning​​，作者：Stylianos Kampakis

The above is the detailed content of Machine Learning Decision Tree Practical Exercise. For more information, please follow other related articles on the PHP Chinese website!