Duplicates are repeated rows (or repeated keys). Unique values are the distinct entries in a column. You need both when joining and when counting customers vs orders.

Goal

Flag, drop, and inspect duplicates, and summarize a column with value_counts.

Row duplicates

df = pd.DataFrame(
    {
        "name": ["Ada", "Alan", "Ada", "Grace", "Ada"],
        "city": ["Nairobi", "Mombasa", "Nairobi", "Kisumu", "Nakuru"],
        "score": [91, 88, 91, 95, 70],
    }
)
print(df)
print()
print(df.duplicated())
print()
print(df.duplicated(keep=False))
print()
print(df.drop_duplicates())

keep="first" (default) marks later copies True. keep=False marks every copy of a duplicated row.

Duplicates on a subset of columns

Same person, different city — still a duplicate name:

df = pd.DataFrame(
    {
        "name": ["Ada", "Alan", "Ada", "Grace", "Ada"],
        "city": ["Nairobi", "Mombasa", "Nairobi", "Kisumu", "Nakuru"],
        "score": [91, 88, 91, 95, 70],
    }
)
print(df.duplicated(subset=["name"], keep=False))
print()
print(df.drop_duplicates(subset=["name"], keep="first"))

Unique values

df = pd.DataFrame({"name": ["Ada", "Alan", "Ada", "Grace", "Ada"]})
print(df["name"].unique())
print("nunique:", df["name"].nunique())
print("nunique including NA:", df["name"].nunique(dropna=False))

`value_counts`

df = pd.DataFrame(
    {
        "name": ["Ada", "Alan", "Ada", "Grace", "Ada"],
        "city": ["Nairobi", "Mombasa", "Nairobi", "Kisumu", "Nakuru"],
    }
)
print(df["name"].value_counts())
print()
print(df["name"].value_counts(dropna=False))
print()
print(df.value_counts(["name", "city"]))

df.value_counts([...]) counts combinations.

Drop after you inspect

Always print(df.duplicated().sum()) before drop_duplicates. A surprising count means your key is wrong, not that pandas is wrong.

df = pd.DataFrame(
    {
        "name": ["Ada", "Alan", "Ada", "Grace", "Ada"],
        "city": ["Nairobi", "Mombasa", "Nairobi", "Kisumu", "Nakuru"],
        "score": [91, 88, 91, 95, 70],
    }
)
print("duplicate row count:", int(df.duplicated().sum()))
print("duplicate names:", int(df.duplicated(subset=["name"]).sum()))

You should see

The Files chapter’s messy.csv has a repeated Ada Kwon row. You will drop it in Practice.