pandas · difficulty ◆◆

pandas pipe() — chain your own functions into readable pipelines

Write the pipeline in the order you think it — clean, filter, total — and let pipe() hand the frame to each step.

You don't need a DSL, an operator or a library for readable pandas pipelines. The chainable step has been in the box since July 2015 — and it is six lines of source code.

fully AI-generated

2026-10-17 · 4 min read

$ df.pipe(clean)

0.16.2
added in pandas: 1 + *args/**kwargs
signature parameters: 5 classes
pipe methods in the API: 2.3.0 → 3.0.0
dispatch change

What it does

pipe() calls a function with your DataFrame as the first argument and returns whatever that function returns: df.pipe(clean) is clean(df). Extra positional and keyword arguments are passed straight through — df.pipe(scale, factor=2.5) becomes scale(df, factor=2.5) — so a step that needs configuration stays a one-liner. When your function wants the frame somewhere other than position one, give pipe a tuple: df.pipe((pd.merge, "left"), right=other, on="sku") injects the frame into that keyword. It is defined on NDFrame, so DataFrame and Series share it, and GroupBy, Resampler and Styler each have one too.

Why it matters

Every pandas report I have written starts the same way: read, clean three messes, filter, total by group. Without pipe that becomes a nest — total(filter(clean(read()))) — or a graveyard of reassigned temp names, where a reader has to hold four intermediate variables in their head and the order of steps is buried in bracket depth. pipe() puts the steps back in the order you think them, top to bottom, and it lets your own functions be steps: named, tested, reused across reports. It also solves the debugging problem — drop a function that prints the shape and returns the frame unchanged, and your chain logs its own progress without a single assignment.

Example

$ orders = pd.DataFrame({
    "order_id": ["SO-1041", "SO-1042", "SO-1043", "SO-1044", "SO-1045", "SO-1046"],
    "city": ["  utrecht", "ROTTERDAM ", "utrecht", "Eindhoven", "ROTTERDAM ", "  eindhoven"],
    "sku": ["VLV-20", "FLT-11", "VLV-20", "GSK-07", "FLT-11", "GSK-07"],
    "units": [120, 40, 310, 75, 220, 18],
    "unit_price": [4.5, 12.0, 4.5, 30.0, 12.0, 30.0],
})

def clean_city(frame):
    return frame.assign(city=frame["city"].str.strip().str.title())

orders.pipe(clean_city)[["order_id", "city", "units"]]

  order_id       city  units
0  SO-1041    Utrecht    120
1  SO-1042  Rotterdam     40
2  SO-1043    Utrecht    310
3  SO-1044  Eindhoven     75
4  SO-1045  Rotterdam    220
5  SO-1046  Eindhoven     18

The export came back with padded whitespace and screaming city names — pipe(clean_city) fixes the column and hands you a frame you can index immediately. Note the helper is not a lambda: a named function can be unit-tested once and reused in every report.

$ # continues from the orders frame and clean_city above
def revenue_by_city(frame, min_units):
    out = frame.assign(revenue=frame["units"] * frame["unit_price"])
    return out[out["units"] >= min_units].groupby("city")["revenue"].sum().sort_values(ascending=False)

orders.pipe(clean_city).pipe(revenue_by_city, min_units=75)

city
Rotterdam    2640.0
Eindhoven    2250.0
Utrecht      1935.0
Name: revenue, dtype: float64

A step with a parameter stays inline: min_units=75 is passed to the function, not to pipe. Reading left to right: clean the city names, then total revenue per city for orders of 75 units or more.

$ suppliers = pd.DataFrame({
    "sku": ["VLV-20", "FLT-11", "GSK-07"],
    "segment": ["valve", "filter", "gasket"],
    "lead_days": [21, 35, 14],
})
orders.pipe((pd.merge, "left"), right=suppliers, on="sku").head(4)

  order_id        city     sku  units  unit_price segment  lead_days
0  SO-1041     utrecht  VLV-20    120         4.5   valve         21
1  SO-1042  ROTTERDAM   FLT-11     40        12.0  filter         35
2  SO-1043     utrecht  VLV-20    310         4.5   valve         21
3  SO-1044   Eindhoven  GSK-07     75        30.0  gasket         14

The tuple form exists for exactly this: pd.merge wants the left frame in the first argument, but pipe would push suppliers there as a positional argument and you would get "merge() got multiple values for argument 'left'". Naming the target keyword — (pd.merge, "left") — puts the orders frame where merge expects it and lets you enrich mid-chain.

$ def audit(frame, label):
    print(f"[{label}] {len(frame)} rows x {frame.shape[1]} cols")
    return frame

(orders.pipe(clean_city)
       .pipe(audit, "cleaned")
       .pipe(pd.DataFrame.query, "units >= 75")
       .pipe(audit, "big orders")
       .pipe(lambda d: d.groupby("city")["units"].sum()))

[cleaned] 6 rows x 5 cols
[big orders] 4 rows x 5 cols
city
Eindhoven     75
Rotterdam    220
Utrecht      430
Name: units, dtype: int64

Two tricks in one line: pandas' own unbound methods are ordinary callables, so pd.DataFrame.query pipes like any helper; and audit returns the frame untouched, which turns it into a log step that does not break the chain. The last step returns a Series — pipe does not care what your function returns.

$ def scribble(frame):
    frame.loc[0, "site"] = "CHANGED"
    frame.loc[0, "tags"].append("MUTATED")
    return frame

sites = pd.DataFrame({"site": ["Yanbu", "Jubail"], "tags": [["a"], ["b"]]})
sites.pipe(scribble).iloc[0]["site"]
sites.iloc[0]["site"]
sites.iloc[0]["tags"]

'CHANGED'
'Yanbu'
['a', 'MUTATED']

Since Copy-on-Write became the default in pandas 3.0, pipe hands your function a shallow copy: assigning a scalar inside the function stays inside, and the caller's frame keeps "Yanbu". The shallow part matters — a list stored in a cell is shared, so appending to it leaked back into the original frame. Treat the copy as yours, but do not mutate objects you did not create.

Common flags

df.pipe(func): calls func(df) and returns whatever func returns — the frame goes in as the first positional argument
df.pipe(func, 2.5, mode="fast"): extra args and kwargs are forwarded to func, so configured steps stay one line
df.pipe((pd.merge, "left"), right=other, on="sku"): tuple form: inject the frame into a named keyword instead of position zero
df.pipe(pd.DataFrame.query, "units >= 75"): unbound pandas methods are plain callables — query, sort_values, groupby pipe fine
df.pipe(audit, "after clean"): a helper that prints and returns the frame unchanged is a readable in-chain log step
df.pipe(clean_city) # not pipe(lambda d: ...): named helpers are reusable and testable; lambdas hide the vocabulary pipe is supposed to give you
df.groupby("sku").pipe(top_n, 3): GroupBy, Resampler and Styler have pipe too, so a pipeline can drop below the 2-D level

History

pandas 0.16.2, July 2015 — and the release notes name the inspiration

The changelog entry is blunt: "A new pipe method", introduced with the sentence "The pipe method is inspired by unix pipes, which stream text through processes." The (callable, data_keyword) tuple form was not a later addition — it is in generic.py at the 0.16.2 tag, in the same release that shipped the method. R's magrittr %>% had made chaining fashionable a year earlier; pipe was pandas' way of letting Python functions join in without a new operator.

Design note: the family grew, and one of them was broken for years

pipe is defined once on NDFrame, which is why DataFrame and Series share identical semantics. GroupBy.pipe and Resampler.pipe followed, and Styler.pipe arrived in 0.24.0 (GH 23229) for styling chains. The Resampler version deserves a footnote: until 0.23.0 calls to pipe on a Resampler were diverted to the mean method, so ser.resample("D").pipe(f) quietly returned a mean instead of running f (GH 17905). And the dispatch line changed with Copy-on-Write — in 2.3.0 the source reads self.copy(deep=None) if using_copy_on_write() else self; in 3.0.0, where Copy-on-Write is the only mode, it is an unconditional self.copy(deep=False).

Under the hood

Under the hood: six lines in pandas.core.common.pipe

There is no pipeline machinery, no expression tree, no lazy evaluation — just dispatch. In pandas 3.0.3, DataFrame.pipe ends with common.pipe(self.copy(deep=False), func, *args, **kwargs), and common.pipe does this: if func is a tuple, split it into callable and target, raise ValueError if that target is already in kwargs ("both the pipe target and a keyword argument"), put the frame into kwargs[target], call it; otherwise call func(obj, *args, **kwargs) directly. Consequences worth knowing: nothing is validated, so a step that returns an int will only surface when the next .pipe() tries to use it; kwargs are handed over untouched, so a function that mutates an argument it received still does that; and the shallow self.copy(deep=False) is what makes the pre-3.0 idiom df.pipe(mutating_helper) safe for scalar assignment while leaving nested mutable cell objects shared.

Fun facts

Pros & cons

pros

+ Turns nested calls into left-to-right reading order — the same argument that made shell pipelines win decades ago.
+ Your own functions become first-class steps: named, documented, unit-testable, and reusable across reports.
+ Almost no machinery to trust: the implementation is dispatch plus a shallow copy, so the cost and behavior are predictable.

cons

− No validation anywhere: pipe will happily run a step that returns a scalar, and the error appears three steps later where the chain breaks.
− Lambdas in a chain quietly undo the point — you get the vertical layout of a pipeline with the readability of a nested call.
− No shared vocabulary: every team reinvents clean_rows / audit / enrich with different names, and newcomers must read the helpers before the pipeline means anything.

Takeaways

1If a step needs arguments, wrap it in a function and pipe it: df.pipe(scale, factor=2.5) beats unwinding five closing brackets.
2Keep a def audit(frame, label) that prints the shape and returns the frame; progress logging then lives inside the chain.
3pandas' own unbound methods are valid steps — df.pipe(pd.DataFrame.query, "units >= 75") reads like a shell pipe.
4Use the tuple form ((pd.merge, "left")) whenever your function wants the frame in a keyword that is not the first argument.
5pipe returns whatever the function returns, so put a scalar or Series step last and do not expect a DataFrame afterwards.

Related commands

archive entry — written and published by a scheduled job, never editorially reviewed. Kept for reference: out of the site’s search index, and no ads run here. the reviewed library →

← all learning