Pandas

Pandas Missing Values

01 · What is a Missing Value? File · Missing Values · 1 of 8

What is a Missing Value?

A missing value means data is not available for a particular column/record.

In Pandas, missing values are represented as NaN (Not a Number).

02 · Detecting Missing Values File · Missing Values · 2 of 8

Detecting Missing Values

Python
df.isnull()
df.isna()
df.notnull()
df.notna()
isnull() / isna() → returns True where value is missing
notnull() / notna() → returns True where value is present
03 · Counting Missing Values File · Missing Values · 3 of 8

Counting Missing Values

Python
df.isnull().sum()          # missing values per column
df.isnull().sum().sum()    # total missing values
df.isna().sum()
True is treated as 1 and False as 0, so .sum() gives the count of missing values.
04 · Dropping Missing Values File · Missing Values · 4 of 8

Dropping Missing Values

Python
df.dropna()                 # drop rows with any missing value
df.dropna(axis=0)          # same as above (rows)
df.dropna(axis=1)          # drop columns with any missing value
df.dropna(how="any")       # drop if any value is missing
df.dropna(how="all")       # drop only if all values are missing
df.dropna(subset=["Age", "Salary"])
dropna() removes rows or columns that contain missing values.
Original DataFrame is not changed unless you use inplace=True.
05 · Filling Missing Values File · Missing Values · 5 of 8

Filling Missing Values

Python
df.fillna(0)                        # fill all NaN with 0
df["Age"].fillna(df["Age"].mean())
df["Salary"].fillna(df["Salary"].median())
df.fillna({"Age": 30, "Salary": 50000})
We can fill missing values with a constant, mean, median, or different values for different columns.
06 · Forward Fill (ffill) File · Missing Values · 6 of 8

Forward Fill (ffill)

Forward fill takes the previous valid value and fills the missing value with it.

Python
df["credit_score"].ffill()
df.fillna(method="ffill")   # older syntax
Output
0    780.0
1    720.0
2    690.0
3    690.0
4    810.0
5    700.0
6    650.0
7    650.0
Name: credit_score, dtype: float64
The missing value is replaced by the last known valid value above it.
07 · Backward Fill (bfill) File · Missing Values · 7 of 8

Backward Fill (bfill)

Backward fill takes the next valid value and fills the missing value with it.

Python
df["credit_score"].bfill()
Output
0    780.0
1    720.0
2    690.0
3    810.0
4    810.0
5    700.0
6    650.0
7      NaN
Name: credit_score, dtype: float64
The missing value is replaced by the next known valid value below it.
If there is no next value, it remains NaN.
08 · Summary File · Missing Values · 8 of 8

Summary

  • isnull() / isna() → detect missing values
  • isnull().sum() → count missing values
  • dropna() → remove rows/columns with missing values
  • fillna() → fill missing values with a value
  • ffill() → forward fill
  • bfill() → backward fill
Home Missing Values

Missing Values in Pandas

Detect, drop and fill missing data using isnull, dropna, fillna, ffill and bfill.

Select any topic from the left sidebar to begin.


Start Learning →