Python Pandas - Dropping Missing Values

---

---

Missing data is a common issue when working with real-world datasets. The Python Pandas library provides an easy way for removing rows or columns that contain missing values (NaN or NaT) from a dataset using the dropna() method.

The dropna() method in Pandas is a useful tool to handle missing data by dropping rows or columns based on your specific requirements. In this tutorial, we will learn how to use dropna() to clean your dataset by dropping missing data based on various conditions.

The dropna() Method

The Pandas dropna() method allows you to remove missing values from a Pandas data structures such as, Series and DataFrame objects. It offers several options to customize how you drop rows or columns based on the presence of NaN values. This method returns a new Pandas object with missing data dropped or it returns None if inplace parameter is set to True.

Syntax

Following is the syntax −

DataFrame.dropna(*, axis=0, how=<no_default>, thresh=<no_default>, subset=None, inplace=False, ignore_index=False)

Where,

• axis: 0 or 'index' (default) to drop rows; 1 or 'columns' to drop columns.

• how: By default it is set to 'any', which drops that row or column if any missing values are present. If set to 'all', then it drops that row or column if all the missing values.

• thresh: Require a minimum number of non-NA values to retain the row or column.

• subset: List of specific columns (if dropping rows) or rows (if dropping columns) to consider.

• inplace: Modify the DataFrame in place (default is False).

• ignore_indexReset the index of the result (default is False).

Let's explore the how of the dropna() method drops the missing data based on various conditions.

Drop Rows with Any Missing Values

By default, the dropna() method removes rows where any missing values are present.

Example

The following example uses the dropna() method to drop the rows that have any missing values.

import pandas as pd
import numpy as np

dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], "Roll_number": [23, 45, np.nan, 18],
           "Major_Subject": ["Maths", "Physics", "Arts", "Political science"], "Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop the rows that have any missing values
df_cleaned = df.dropna()
print('\nResultant DataFrame after removing row:\n',df_cleaned)

Output

Following is the output of the above code −

Original DataFrame:
  Student_name  Roll_number      Major_Subject  Marks
1         Ajay         23.0              Maths   57.0
2      Krishna         45.0            Physics    NaN
3       Deepak          NaN               Arts   98.0
4        Swati         18.0  Political science    NaN

Resultant DataFrame after removing row:
   Student_name  Roll_number Major_Subject  Marks
1         Ajay         23.0         Maths   57.0

Drop Rows Where All Values Are Missing

To drop rows where all values are missing, then we need to set how='all' parameter to the dropna() method.

Example

The following example demonstrates how to drop the rows where all values are missing in a DataFrame.

import pandas as pd
import numpy as np

dataset = {"Student name": ["Ajay", np.nan, "Deepak", "Swati"], 
"Roll number": [23, np.nan, np.nan, 18],
"Major Subject": ["Maths", np.nan, "Arts", "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop rows where all values are missing
reslut = df.dropna(how='all')
print('\nResultant DataFrame after removing row:\n',reslut)

Output

Following is the output of the above code −

Original DataFrame:
  Student name  Roll number      Major Subject  Marks
1         Ajay         23.0              Maths   57.0
2          NaN          NaN                NaN    NaN
3       Deepak          NaN               Arts   98.0
4        Swati         18.0  Political science    NaN

Resultant DataFrame after removing row:
   Student name  Roll number      Major Subject  Marks
1         Ajay         23.0              Maths   57.0
3       Deepak          NaN               Arts   98.0
4        Swati         18.0  Political science    NaN

Keep Rows with a Minimum Number of Missing Values

The pandas dropan() method provides the thresh parameter to specify a minimum threshold of non-missing values for keeping rows with a minimum number of Non-Na values.

Example

This example demonstrates how to keep the rows the minimum number of missing values.

import pandas as pd
import numpy as np

dataset = {"Student name": ["Ajay", "Krishna", "Deepak", "Swati"], 
"Roll number": [23, np.nan, np.nan, 18],
"Major Subject": ["Maths", np.nan, "Arts", "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop the rows with a threshold 
result = df.dropna(thresh=2)
print('\nResultant DataFrame after removing row:\n',result)

Output

Following is the output of the above code −

Original DataFrame:
  Student name  Roll number      Major Subject  Marks
1         Ajay         23.0              Maths   57.0
2      Krishna          NaN                NaN    NaN
3       Deepak          NaN               Arts   98.0
4        Swati         18.0  Political science    NaN

Resultant DataFrame after removing row:
   Student name  Roll number      Major Subject  Marks
1         Ajay         23.0              Maths   57.0
3       Deepak          NaN               Arts   98.0
4        Swati         18.0  Political science    NaN

Drop Columns with Any Missing Values

To drop columns that contain any missing values, then we can use axis parameter of the dropna() method to select the columns.

Example

This example show how the dropna() method removes the entire column where any of the value is missing.

import pandas as pd
import numpy as np

dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], 
"Roll_number": [23, 45, np.nan, 18],
"Major_Subject": ["Maths", "Physics", "Arts", "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop column with any missing values
result = df.dropna(axis='columns')
print('\nResultant DataFrame after removing columns:\n',result)

Output

Following is the output of the above code −

Original DataFrame:
  Student_name  Roll_number      Major_Subject  Marks
1         Ajay         23.0              Maths   57.0
2      Krishna         45.0            Physics    NaN
3       Deepak          NaN               Arts   98.0
4        Swati         18.0  Political science    NaN

Resultant DataFrame after removing columns:
   Student_name      Major_Subject
1         Ajay              Maths
2      Krishna            Physics
3       Deepak               Arts
4        Swati  Political science

Drop Rows Based on Missing Data in Specific Columns

You can use the subset parameter of the drop() method to focus only on those particular columns while dropping rows where data is missing.

Example

This example shows how to remove the rows based on missing data present in the specific column using the subset parameter of the dropna() method.

import pandas as pd
import numpy as np

dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], 
"Roll_number": [23, 45, np.nan, 18],
"Major_Subject": ["Maths", "Physics", np.nan, "Political science"], 
"Marks": [57, np.nan, 98, np.nan]}

df = pd.DataFrame(dataset, index= [1, 2, 3, 4])
print("Original DataFrame:")
print(df)

# Drop Rows Based on Missing Data in Specific Columns
result = df.dropna(subset=['Roll_number', 'Major_Subject'])
print('\nResultant DataFrame after removing rows:\n',result)

Output

Following is the output of the above code −

Original DataFrame:
  Student_name  Roll_number      Major_Subject  Marks
1         Ajay         23.0              Maths   57.0
2      Krishna         45.0            Physics    NaN
3       Deepak          NaN                NaN   98.0
4        Swati         18.0  Political science    NaN

Resultant DataFrame after removing rows:
   Student_name  Roll_number      Major_Subject  Marks
1         Ajay         23.0              Maths   57.0
2      Krishna         45.0            Physics    NaN
4        Swati         18.0  Political science    NaN