Types and conversion

astype, to_numeric, convert_dtypes, and categorical columns.

dtypes decide what you can do. Object columns will not sum. Integers cannot hold NaN unless you use pandas’ nullable types. Convert early, after you have looked at a few values.

Goal

Turn text numbers into numbers, dates into datetimes, and labels into categories — without silently turning bad cells into NaN unless you asked.

See the problem

df = pd.DataFrame(
    {
        "product": ["A", "B", "A", "C"],
        "units": ["12", "7", "9", "n/a"],
        "price": [10.5, 22, 10.5, 31],
        "flag": [1, 0, 1, 0],
    }
)
print(df.dtypes)
print(df)
print()
print("sum of units column (strings concatenate):", df["units"].sum())

to_numeric

df = pd.DataFrame(
    {
        "product": ["A", "B", "A", "C"],
        "units": ["12", "7", "9", "n/a"],
        "price": [10.5, 22, 10.5, 31],
        "flag": [1, 0, 1, 0],
    }
)
df["units"] = pd.to_numeric(df["units"], errors="coerce")
print(df.dtypes)
print(df)
print("units NA after coerce:", int(df["units"].isna().sum()))
print(df.loc[df["units"].isna()])
  • errors="raise" (default) — explode on bad values
  • errors="coerce" — bad values become NaN
  • errors="ignore" — return the original (rarely what you want)

astype

df = pd.DataFrame(
    {
        "product": ["A", "B", "A", "C"],
        "flag": [1, 0, 1, 0],
    }
)
df["flag"] = df["flag"].astype(bool)
df["product"] = df["product"].astype("category")
print(df.dtypes)
print()
print(df["product"].cat.categories)
print(df)

Categories save memory on repeated labels and keep a stable order you can set with cat.set_categories.

convert_dtypes

Asks pandas to pick nullable dtypes (Int64, string, boolean).

df = pd.DataFrame(
    {
        "product": ["A", "B", "A", "C"],
        "units": ["12", "7", "9", "n/a"],
        "flag": [1, 0, 1, 0],
    }
)
print(df.convert_dtypes().dtypes)

Downcast (optional)

prices = pd.Series([10.5, 22.0, 31.0])
print(pd.to_numeric(prices, downcast="float"))

Useful on large frames; skip it on these toy tables.

Pitfall

"1,200" will not parse as 1200 until you strip commas: s.str.replace(",", "", regex=False) then to_numeric. The String chapter covers that.