pandas · difficulty ◆◆

pandas apply() — run any Python function across rows, columns and groups

apply() is the escape hatch for rules numpy can't express: one call per row, column or group, and the one place in pandas where writing a plain Python function is the point.

Every pandas tutorial tells you to avoid apply(). I still reach for it twice a week — because the rules it runs are the ones no vectorised expression survives.

fully AI-generated

2026-10-15 · 6 min read

$ df.apply()

What it does

apply takes a callable and runs it once per axis slice: axis=0 (the default) hands it each column, axis=1 each row as a Series, and raw=True replaces that row with a plain numpy array. It is not a fast path — pandas loops in Python, builds an object per slice, then reassembles the results into a Series or DataFrame. result_type decides how list-like returns are shaped: reduce (the default), expand (one column per element) and broadcast (spread across the existing columns). The same callable-over-groups idea sits on GroupBy.apply, Rolling.apply and Series.apply.

Why it matters

The rules I meet at work are conditional ladders: an order is escalated when it is late and expensive, a carrier is picked from weight and destination, a price climbs a tier table. Written as vectorised expressions they become np.select towers nobody dares to touch; written as a function they are reviewed by the person who owns the policy. apply also keeps that logic inside a chain, so it drops between read_csv and to_parquet, and groupby.apply answers what plain groupby cannot: give me the top two rows per region.

Example

$ import pandas as pd

orders = pd.DataFrame({
    "order_id": [1001, 1002, 1003, 1004, 1005, 1006],
    "city": ["Vienna", "Graz", "Linz", "Salzburg", "Innsbruck", "Bremen"],
    "units": [120, 45, 300, 88, 12, 640],
    "unit_price": [19.90, 24.50, 9.99, 15.00, 129.00, 3.20],
    "delivery_days": [3, 7, 2, 11, 1, 12],
    "sla_days": [5, 5, 4, 7, 3, 10],
})
orders["revenue"] = (orders["units"] * orders["unit_price"]).round(2)


def escalation(row):
    if row["delivery_days"] > row["sla_days"]:
        return "escalate" if row["revenue"] > 1500 else "watch"
    return "ok"


orders["status"] = orders.apply(escalation, axis=1)
print(orders[["order_id", "city", "revenue", "delivery_days", "sla_days", "status"]])

   order_id       city  revenue  delivery_days  sla_days    status
0      1001     Vienna   2388.0              3         5        ok
1      1002       Graz   1102.5              7         5     watch
2      1003       Linz   2997.0              2         4        ok
3      1004   Salzburg   1320.0             11         7     watch
4      1005  Innsbruck   1548.0              1         3        ok
5      1006     Bremen   2048.0             12        10  escalate

Three columns of input, one threshold and a nested branch — the exact shape that turns np.select into a wall. axis=1 gives the function one row at a time, and the new column lands back on the original index, so there's no merge and no temp column.

$ import re
import pandas as pd

amounts = pd.Series(
    ["EUR 1.234,50", "EUR 99,90", "EUR 2.480,00", "EUR 15,00"], name="amount"
)


def parse_eur(text: str) -> float:
    digits = re.sub(r"[^\d.,]", "", text)
    if "," in digits:                      # continental format: 1.234,50
        digits = digits.replace(".", "").replace(",", ".")
    return float(digits)


clean = amounts.apply(parse_eur)
print(clean)
print(clean.dtype, "| total:", clean.sum())

0    1234.5
1      99.9
2    2480.0
3      15.0
Name: amount, dtype: float64
float64 | total: 3829.4

A parser with a conditional inside it has no honest vectorised equivalent — two chained .str.replace calls are where 2.480,00 quietly becomes 2.48. One function, one line of apply, and the column is real float64 and summable.

$ import pandas as pd

shipments = pd.DataFrame(
    {
        "awb": ["A-1001", "A-1002", "A-1003", "A-1004"],
        "weight_kg": [12.4, 480.0, 3.1, 220.5],
        "destination": ["AT", "DE", "AT", "US"],
        "express": [True, False, True, False],
    }
)


def route(row):
    carrier = "freight" if row["weight_kg"] > 300 else ("air-express" if row["express"] else "road")
    zone = "domestic" if row["destination"] == "AT" else "international"
    price = round(row["weight_kg"] * (1.10 if carrier == "freight" else 2.50), 2)
    return zone, carrier, price


shipments[["zone", "carrier", "cost_eur"]] = shipments.apply(route, axis=1, result_type="expand")
print(shipments)
print("---")
print(shipments.apply(route, axis=1).head(3))

      awb  weight_kg destination  express           zone      carrier  cost_eur
0  A-1001       12.4          AT     True       domestic  air-express     31.00
1  A-1002      480.0          DE    False  international      freight    528.00
2  A-1003        3.1          AT     True       domestic  air-express      7.75
3  A-1004      220.5          US    False  international         road    551.25
---
0      (domestic, air-express, 31.0)
1    (international, freight, 528.0)
2      (domestic, air-express, 7.75)
dtype: object

result_type='expand' turns one returned tuple into three assignable columns. Drop the argument and the same function returns a single object column of tuples — the second print is that version, and it is why people believe apply can't add columns.

$ import pandas as pd

sales = pd.DataFrame(
    {
        "region": ["East", "East", "East", "West", "West", "North"],
        "city": ["Linz", "Graz", "Salzburg", "Vienna", "Innsbruck", "Bremen"],
        "revenue": [2997.0, 1102.5, 1320.0, 2388.0, 1548.0, 2048.0],
    }
)

top2 = sales.groupby("region", group_keys=False).apply(
    lambda g: g.nlargest(2, "revenue").assign(region=g.name), include_groups=False
)
print(top2)
print("---")
print(sales.groupby("region")["revenue"].apply(lambda s: s.max() - s.min()))

        city  revenue region
0       Linz   2997.0   East
2   Salzburg   1320.0   East
5     Bremen   2048.0  North
3     Vienna   2388.0   West
4  Innsbruck   1548.0   West
---
region
East     1894.5
North       0.0
West      840.0
Name: revenue, dtype: float64

groupby.apply is how a group returns a different shape instead of one number — the top two rows each, not a maximum. The grouping column is no longer passed into the function (pandas 3.0 removed include_groups=True and raises ValueError), so the lambda puts it back with .assign(region=g.name). On a Series group you skip all of that and get per-group values directly.

$ import timeit
import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 200_000
df = pd.DataFrame(
    {
        "units": rng.integers(1, 500, n),
        "price": rng.uniform(1, 200, n).round(2),
        "sla": rng.integers(1, 10, n),
        "days": rng.integers(1, 14, n),
    }
)


def score(row):
    return row["units"] * row["price"] if row["days"] <= row["sla"] else 0.0


def score_raw(a):
    return a[0] * a[1] if a[3] <= a[2] else 0.0


t_series = min(timeit.repeat(lambda: df.apply(score, axis=1), number=1, repeat=5))
t_raw = min(timeit.repeat(lambda: df.apply(score_raw, axis=1, raw=True), number=1, repeat=5))
print(f"row as Series   : {t_series:5.2f} s")
print(f"row as ndarray  : {t_raw:5.2f} s   (raw=True)")
print(f"ratio           : {t_series / t_raw:.1f}x  on {n:,} rows")
print("same answer     :", round(df.apply(score, axis=1).sum(), 2),
      "vs", round(df.apply(score_raw, axis=1, raw=True).sum(), 2))

row as Series   :  1.52 s
row as ndarray  :  0.31 s   (raw=True)
ratio           : 4.9x  on 200,000 rows
same answer     : 1922370795.86 vs 1922370795.86

Measured on the machine that built this page (pandas 3.0.3, Python 3.12, best of five runs) — the ratio moved between 4.6x and 4.9x across my runs, so expect a number in that neighbourhood, not mine exactly. The direction is the point: raw=True skips building a Series per row and hands over a positional ndarray, so a[0] replaces row['units'] and the column labels stop costing anything.

Common flags

df.apply(func, axis=1): One call per row, the row arriving as a Series. axis=0 (default) walks the columns instead — df.apply('sum') reduces each one.
df.apply(func, axis=1, raw=True): The row is a numpy ndarray: no label lookup, no Series construction — 4.9x in the timing example above.
result_type='expand': A returned tuple or list becomes separate columns; 'broadcast' spreads it across the original columns (shape must match), 'reduce' is the default.
df.apply(func, args=(0.2, 'AT')): Extra positional arguments forwarded into func — no lambda needed to bake in a constant or a lookup dict.
groupby.apply(func, include_groups=False): Runs once per group and may return any shape; since 3.0 the grouping column is excluded from the frame your function receives.
series.map(dict_or_func): Elementwise and dict-aware — the right tool for lookups. DataFrame.map() (2.1.0) replaced applymap, which 3.0.0 removed outright.
df.apply(func, axis=1).pipe(...): apply returns a new object, so it chains: read_csv → assign → apply → query → to_parquet stays one expression.

History

0.23.0, 15 May 2018 — the release that made apply's output predictable

The 0.23.0 whatsnew carries a section titled "Changes to make output of DataFrame.apply consistent". When the applied function returned a list-like with axis=1, the shape you got back was inconsistent — the notes cite ten GitHub issues for it (GH 16353, GH 17437, GH 17970, and seven more). The fix settled the rule that still holds: a Series return gives you a DataFrame, anything else gives you a Series. result_type='expand'/'broadcast' shipped alongside it as the explicit way to ask for the other shape.

2.1.0, 30 Aug 2023 — by_row arrives, and the end of applymap

Two apply-family changes landed in 2.1.0: a new by_row parameter on Series.apply and DataFrame.apply (GH 53400, GH 53601), and DataFrame.map() plus the deprecation of DataFrame.applymap (GH 52353). pandas 3.0.0 finished that off on 21 January 2026 by removing applymap (GH 52364), so df.applymap(str) on a current install is an AttributeError rather than a warning.

Fun facts

Pros & cons

pros

+ Expresses rules no vectorised expression survives — multi-column if/else ladders, format parsers, third-party library calls — as plain, reviewable Python
+ One API for rows, columns, groups and windows: DataFrame.apply, Series.apply, GroupBy.apply and Rolling.apply all take the same callable
+ Returns a new object, so it composes into a chain without a merge or a helper column

cons

− Python-speed by construction: one function call per row — 1.52 s for 200k rows in my run, and raw=True only buys a constant factor
− Shape footguns: forget result_type='expand' and a tuple return collapses into one object column; Series.apply doesn't accept result_type at all
− The group contract moved in 3.0 — include_groups=True raises, so the grouping column has to be re-attached inside your function

Takeaways

1Reach for apply when the rule is conditional logic across several columns; when numpy can do it in one expression, that version wins by a wide margin.
2Add raw=True whenever the row is just data — positional access (a[0] instead of row['units']) ran 4.9x faster in my measured example, with identical sums.
3Return a tuple and pass result_type='expand' to turn a single apply call into several real, assignable columns.
4On pandas 3.0 groupby.apply never sees the grouping column — re-attach it inside the function with .assign(region=g.name) or you lose the key.
5Keep elementwise string and lookup work on .map()/.str accessors: applymap doesn't exist any more, and apply is the most expensive of the three.

Related commands

← all learning