Handling Outliers in Python - IQR Method

Introduction

Before uncovering any insights from real-world data, it is important to scrutinize your data to ensure that data is consistent and free from errors. However, Data can contain errors and some values may appear to differ from other values and these values are known as outliers. Outliers negatively impact data analysis leading to wrong insights which lead to poor decision making by stake holders. Therefore, dealing with outliers is a critical step in the data preprocessing stage in data science. In this article, we will asses different ways we can handle outliers.

Outliers

Outliers are data points that differ significantly from the majority of the data points in a dataset. They are values that fall outside the expected or usual range of values for a particular variable. outliers occur due to various reason for example, error during data entry, sampling errors. In machine learning outliers can cause your models to make incorrect predictions thus causing inaccurate predictions.

Detecting outliers in a dataset using Jupyter notebook

• Import python libraries

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
plt.style.use('ggplot')

• Load your csv file using pandas

df_house_price = pd.read_csv(r'C:\Users\Admin\Desktop\csv files\housePrice.csv')

• Check the first five rows of house prices data set to have a glimpse of your datafrane

df_house_price.head()

• Check for outliers in the price column by use of a box plot

sns.boxplot(df_house_price['Price'])
plt.title('Box plot showing outliers in prices')
plt.show()

• From the box plot visualization the price column has outlier values
• Now we have to come up with ways to handle these outlier values to ensure better decision making and ensure machine learning models make the correct prediction

IQR Method of handling outlier values

• IQR method means interquartile range measures the spread of the middle half of your data. It is the range for the middle 50% of your sample.

Steps for removing outliers using interquartile range

• Calculate the first quartile (Q1) which is 25% of the data and the third quartile (Q3) which is 75% of the data.

Q1 = df_house_price['Price'].quantile(0.25)
Q3 = df_house_price['Price'].quantile(0.75)

• compute the interquartile range

IQR = Q3 - Q1

• Determine the outlier boundaries.

lower_bound = Q1 - 1.5 * IQR

• Lower bound means any value below -5454375000.0 is an outlier

upper_bound = Q3 + 1.5 * IQR

• Upper bound means any value above 12872625000.0 is an outlier

• Remove outlier values in the price column
  

filt = (df_house_price['Price'] >= lower_bound) & (df_house_price['Price'] <= upper_bound)

df = df_house_price[filt]
df.head()

• Box plot After removing outliers

sns.boxplot(df['Price'])
plt.title('Box plot after removing outliers')
plt.show()

Different methods of handling outlier values

• Z-Score method
• Percentile Capping (Winsorizing)
• Trimming (Truncation)
• Imputation
• Clustering-Based Methods e.g DBSCAN

Conclusion

IQR method is simple and robust to outliers and does not depend on the normality assumption. The disadvantage is that it can only handle univariate data, and that it can remove valid data points if the data is skewed or has heavy tails.

Thank you
follow me on linked in and on github for more.

The above is the detailed content of Handling Outliers in Python - IQR Method. For more information, please follow other related articles on the PHP Chinese website!