pandas · difficulty ◆◆
pandas replace() — swap sentinels, labels and currency in one call
replace() is the line between a CSV export and usable data — it is how '-' becomes missing.
A CSV export almost never contains missing values. It contains '-', 'NULL' and '-999', and replace() is the one call that turns those sentinels into data you can compute on.
$ df.replace()What it does
replace() substitutes values inside a Series or DataFrame without changing its shape, index or column order. to_replace says what is looked for — a scalar, a list, a dict, a nested {column: {old: new}} mapping, or a regex when regex=True — and value says what takes its place, either one scalar for everything or a list that lines up pairwise. The two modes people forget are the useful ones: a dict on its own means "map these labels" (no value needed), and a list of sentinels plus one scalar value means "collapse all of these into that one value". Matching is exact per cell by default, so replace("a", "X") leaves "ab" alone. pandas 3.0 also stopped guessing: a non-dict-like to_replace without a value now raises ValueError instead of quietly falling back to filling (GH 33302).
Why it matters
Between the file and the analysis there is always a small set of values that lie about their meaning. An ERP export writes '-' for a missing quantity, a CRM writes 'n/a' for an unknown city, a logger writes -999.0 for a dead sensor, a finance system exports money as '$1,240.50'. Nothing downstream works until those are fixed: .isna() counts zero, .mean() averages -999 into your temperature series, .sum() concatenates strings. replace() is the smallest tool that fixes all four cases, and doing it once, right after the load, is what lets every later step — filter, groupby, pivot — assume honest values. It is also the cheapest way to keep a cleaning rule visible: the mapping sits in the call site, where the person who owns the report can read and review it.
Example
$ import pandas as pd
export = pd.DataFrame({
"order": ["SO-1001", "SO-1002", "SO-1003", "SO-1004"],
"city": ["Vienna", "Berlin", "n/a", "Rome"],
"units": ["12", "-", "40", "8"],
"note": ["", "late", "NULL", "ok"],
})
clean = export.replace(["n/a", "-", "NULL", ""], pd.NA)
print(clean.to_string(index=False))
print(clean.isna().sum().to_string()) order city units note
SO-1001 Vienna 12 NaN
SO-1002 Berlin NaN late
SO-1003 NaN 40 NaN
SO-1004 Rome 8 ok
order 0
city 1
units 1
note 2The frame-wide list form: every column is checked for the same four sentinels and each cell that matches becomes a real missing value. Note the detail in the printout — the str column shows NaN, not <NA>, because a pandas 3.0 string column stores its missing value as NaN, while the counts from .isna() are the same. Before this line, .isna().sum() reported 0 for city, units and note; now dropna(), fillna() and groupby(dropna=False) all see the truth.
$ import pandas as pd
tickets = pd.DataFrame({
"ticket": [501, 502, 503, 504, 505, 506],
"priority": ["P0", "HIGH", "high", "Med", "medium", "LOW"],
})
tickets["priority"] = tickets["priority"].replace({
"P0": "high", "HIGH": "high",
"Med": "medium",
"LOW": "low",
})
print(tickets["priority"].value_counts().to_string())priority
high 3
medium 2
low 1A dict with no value is a label map: exact matches only, applied to the frame, unknown labels untouched. 'P0' folded into 'high' without a second call, which is what you want when a support tool renamed its priorities three times and the export still carries all three spellings. Because matching is exact, 'HIGH' had to be listed next to 'P0' — one misspelt key and that row silently stays as it was.
$ import pandas as pd
invoices = pd.DataFrame({
"invoice": ["INV-2201", "INV-2202", "INV-2203"],
"amount": ["$1,240.50", "$980.00", "$2,015.75"],
})
invoices["amount"] = invoices["amount"].replace(r"[$,]", "", regex=True).astype("float64")
print(invoices.to_string(index=False))
print("total:", invoices["amount"].sum()) invoice amount
INV-2201 1240.50
INV-2202 980.00
INV-2203 2015.75
total: 4236.25The nested dict is the whole cleaning spec in one argument: region gets its own rules, stage gets different ones, and any other column is left exactly as it was. This is the form to keep in a report script — it reads like a config block and a reviewer can diff it. The traceback at the bottom is the 3.0 behaviour worth knowing: passing a list of values without a value used to silently *fill* instead (see the fun facts); now it is a ValueError that names the three legal shapes.
$ import pandas as pd
leads = pd.DataFrame({
"region": ["EMEA", "NAM", "EMEA", "APAC"],
"stage": ["Q", "Closed Won", "W", "Q"],
})
print(leads.replace({
"region": {"EMEA": "Europe", "NAM": "North America"},
"stage": {"Q": "qualified", "W": "won"},
}).to_string(index=False))
try:
leads["region"].replace(["EMEA", "NAM"])
except ValueError as err:
print("ValueError:", err) region stage
Europe qualified
North America Closed Won
Europe won
APAC qualified
ValueError: Series.replace must specify either 'value', a dict-like 'to_replace', or dict-like 'regex'.regex=True turns to_replace into a pattern and makes matching sub-string, which is what currency and measurement columns need. The character class [$,] is applied inside every cell — each '1,240.50' cell keeps its digits and loses the decoration. The .astype('float64') afterwards is not part of replace(); replace() rewrites text, it does not parse it, and the column is still str until something declares it numeric.
$ import pandas as pd
import numpy as np
readings = pd.DataFrame({
"sensor": ["T-01", "T-02", "T-03", "T-04"],
"temp_c": [21.4, -999.0, 22.0, -999.0],
})
fixed = readings.assign(
temp_nan = readings["temp_c"].replace(-999.0, np.nan),
temp_na = readings["temp_c"].replace(-999.0, pd.NA),
)
print(fixed.to_string(index=False))
print(fixed[["temp_nan", "temp_na"]].dtypes.to_string())
print("mean:", round(fixed["temp_nan"].mean(), 2), "| dtype kept:", fixed["temp_nan"].dtype)sensor temp_c temp_nan temp_na
T-01 21.4 21.4 21.4
T-02 -999.0 NaN <NA>
T-03 22.0 22.0 22.0
T-04 -999.0 NaN <NA>
temp_nan float64
temp_na object
mean: 21.7 | dtype kept: float64The numeric twin of example 1, and the dtype lesson: -999.0 into np.nan keeps the column float64 and .mean() then returns 21.7 from the two real readings, while the same replacement with pd.NA — pandas' own missing scalar — leaves the column as object, because object is the only thing that can hold <NA> next to a float. Both are 'correct'; only one keeps your arithmetic and your groupby working. Same trick in a nullable 'Int64' column keeps the dtype, and there pd.NA is the right choice.
$ import pandas as pd
df = pd.DataFrame({"sku": ["abc123", "xyz"], "qty": [1, 2]})
fixed = df.replace(r"\d+", 0, regex=True)
print("result:")
print(fixed.to_string())
print("the frame I called it on:")
print(df.to_string())
print("and its sku column still claims dtype:", df["sku"].dtype)result:
sku qty
0 0 1
1 xyz 2
the frame I called it on:
sku qty
0 0 1
1 xyz 2
and its sku column still claims dtype: strThis one is not a lesson about your data, it is a lesson about the method: on pandas 3.0.3, a regex replacement whose value is *not* a string rewrites the frame you called it on. The copy and the original print identically, and the sku column still reports dtype str while holding 0. It only happens on the extension-backed string columns (3.0's str dtype and the nullable 'string' dtype) — a numpy object column is untouched — and it only happens for non-string values, because that is the branch where the block is upcast to object before the regex pass writes into it. Reproduced on 3.0.3, 2026-10-10.
Common flags
- to_replace="-", value=pd.NA
- the scalar form, the one you reach for first when one specific junk value keeps showing up in a column.
- to_replace=["-", "n/a", "NULL", ""]
- list form plus one scalar value: every cell equal to any of them becomes that value, across the whole frame. This is the line that belongs at the top of a loader function.
- to_replace={"P0": "high", "HIGH": "high"}
- dict without a value = label map. Keys that never occur are silently ignored — no warning, no error, the column just stays dirty, so keep the map next to the data source that produced the labels.
- to_replace={"region": {"EMEA": "Europe"}}
- nested dict = per-column rules. All top-level values must be dicts (a mix raises TypeError), and columns not mentioned are untouched.
- regex=True
- the only switch that makes matching sub-string. Without it replace(r"[$,]", "") would look for a cell whose entire content is the literal text [$,]. With a list on both sides, the lists must be the same length or you get ValueError: Replacement lists must match in length.
- value="-" vs value=0
- with a string value the regex substitutes the matched part ('abc123' -> 'abc-'); with a non-string value it replaces the whole cell whenever the pattern matches anywhere ('abc123' -> 0). Two different operations behind one keyword — verified on 3.0.3.
- inplace=True
- since 3.0 it returns the object itself instead of None (GH 63207) and under Copy-on-Write it saves nothing. Reassign the result and forget the keyword exists.
History
0.8.0, June 29, 2012 — "a flexible replace method" arrives
There is no replace anywhere in the pandas 0.7.3 sdist — grep the tarball for 'def replace' and you get nothing. It lands in 0.8.0, whose whatsnew lists "Add flexible replace method for efficiently substituting values", and it lands twice, as two independent implementations: Series.replace(self, to_replace, value=None, method='pad', inplace=False, limit=None) at pandas/core/series.py:2153 and DataFrame.replace(self, to_replace, value=None, method='pad', axis=0, inplace=False, limit=None) at pandas/core/frame.py:2796. Look at those signatures for a second: the method shipped with a fill strategy as its default, because in 2012 replacing values and padding gaps were considered the same errand.
0.13.0, January 3, 2014 — one implementation, and a fallback that took 13 years to kill
The two versions drifted: the 0.12.0 sdist (July 24, 2013) shows DataFrame.replace with regex=False and a proper limit, while Series.replace still had no regex argument at all and sat at method='pad'. pandas 0.13.0 moved replace into NDFrame — pandas/core/generic.py:1957, "Series replace is now consistent with DataFrame" in the release notes — which is the single code path it still runs today. The last piece of the original design is only now gone: passing a list as to_replace with no value used to fall back to filling, i.e. a silent fillna, which was deprecated in 2.1.0 (Aug 30, 2023, GH 33302) and became the ValueError you saw above in 3.0.0, the same release that deleted method= and limit= (GH 53492).
Fun facts
Pros & cons
pros
- + Four cleaning jobs in one method — sentinel-to-NA across a frame, label maps, per-column rules, regex rewrites — with no change to shape, index or column order
- + Because it never reshapes anything, it composes: run it once after the load and every downstream filter, groupby and pivot gets to assume honest values
- + The nested-dict form turns a cleaning spec into data that can be read, reviewed and reused across scripts instead of hidden in a chain of conditional assignments
cons
- − Silent no-ops are the default failure mode: a dict key that never occurs, a pattern that matches nothing, an exact-match string you assumed was a substring — none of them raise
- − Type surprises come with the NA scalars: pd.NA into a float column gives object dtype, any new label on a categorical column is a TypeError
- − A non-string regex replacement can rewrite the original frame on 3.0.3 (reproduced above), so a cleaning call is not always as side-effect-free as Copy-on-Write promises
Takeaways
- 1Clean sentinels before anything else: df.replace(["-", "n/a", "NULL", ""], pd.NA) turns every fake missing value into a real one in a single call, and every isna()/dropna()/fillna() after it depends on having done so.
- 2Use the dict form for label cleanups and keep the mapping in a named constant — a dict is documentation that happens to execute, a chain of loc assignments is not.
- 3Check the dtype right after any replacement that introduces NA: np.nan keeps float64, pd.NA quietly leaves you with object.
- 4regex=True is for sub-string edits on text columns; for a numeric sentinel write replace(-999.0, np.nan) and leave regex off.
- 5Ignore any recipe that passes limit=, method=, or relies on inplace=True returning None — 3.0 removed the first two and made the third return the object (GH 53492, GH 63207).