Parquet remembers what CSV forgets: the types.
Every DataFrame you save to CSV loses its mind on the way out — Parquet is the format that doesn't.
read_parquet() loads a Parquet file (or a whole directory of them) into a DataFrame, and to_parquet() writes one. Parquet is columnar and binary: each column is stored, compressed and encoded on its own, so dtypes, datetimes, categoricals and the index survive the roundtrip untouched. Engine-wise pandas dispatches to pyarrow (default preference) or fastparquet via engine="auto".
CSV is a text format — everything is characters, and every read re-guesses types. With read_csv() you spend half your life in dtype= and parse_dates=; with Parquet those types are stored in the file, so a read is a memory copy, not a guess. Add columns= to read only some columns and filters= to skip row groups, and you pay only for the data you need — that is exactly how tools like DuckDB, Spark and polars treat the format too.
orders.to_parquet("orders.parquet")
back = pd.read_parquet("orders.parquet")
print(back.head(3))
print("same dtypes:", (back.dtypes == orders.dtypes).all(),
"| same values:", back.equals(orders))order_id region product units unit_price_eur order_date 0 1001 North valve 3 1199.0 2026-07-14 1 1002 South sensor 10 229.9 2026-07-15 2 1003 East valve 3 1199.0 2026-07-18 same dtypes: True | same values: True
A full roundtrip: int64 order IDs, float64 prices, datetime64[us] dates all come back exactly as written — the same frame through CSV would hand back order_date as strings and force you to re-parse.
# 8 columns on disk, 3 needed for the report:
ledger.to_parquet("ledger.parquet")
small = pd.read_parquet("ledger.parquet",
columns=["region", "units", "unit_price_eur"])
print(small)
print(f"file has {ledger.shape[1]} columns, read {small.shape[1]};",
f"memory {ledger.memory_usage(deep=True).sum()}",
f"-> {small.memory_usage(deep=True).sum()} bytes")region units unit_price_eur 0 North 3 1199.00 1 South 10 229.90 2 East 3 1199.00 3 West 25 89.50 4 North 8 229.90 5 East 4 59.99 6 South 3 1199.00 7 West 12 89.50 file has 8 columns, read 3; memory 756 -> 360 bytes
columns= is column pruning at read time — Parquet stores each column separately, so the other five are never decoded. On wide files this is the difference between loading 1 GB and 200 MB.
# Only decode rows for one region:
north = pd.read_parquet("ledger.parquet",
filters=[("region", "=", "North")])
print(north[["order_id", "region", "revenue"]])order_id region revenue 0 1001 North 3597.0 1 1005 North 1839.2
filters= (pandas ≥ 2.1) is predicate pushdown: pyarrow reads each row group's statistics, skips whole groups that can't match, and decodes only survivors. Syntax is (col, op, value) tuples; inner lists AND, outer list OR.
# One directory, one file per region:
ledger.to_parquet("by_region/", partition_cols=["region"])
for root, dirs, files in os.walk("by_region"):
for f in sorted(files):
print(os.path.join(root, f))
back = pd.read_parquet("by_region/")
print(back[back.region == "South"][["order_id", "region", "revenue"]])
print("rows:", len(back), "| regions:", sorted(back.region.unique()))by_region/region=South/fe91a020e80a41cfa768af4f3b7bb790-0.parquet by_region/region=West/fe91a020e80a41cfa768af4f3b7bb790-0.parquet by_region/region=East/fe91a020e80a41cfa768af4f3b7bb790-0.parquet by_region/region=North/fe91a020e80a41cfa768af4f3b7bb790-0.parquet order_id region revenue 4 1002 South 2299.0 5 1007 South 3597.0 rows: 8 | regions: ['East', 'North', 'South', 'West']
partition_cols (pyarrow engine, since 0.24) writes a Hive-style dataset: the region column moves into directory names and vanishes from each file's schema — read the dataset root and pandas reconstructs it. Low-cardinality columns only: one directory per distinct value.
# 100k sensor readings: same data, two formats.
# Files: CSV 3,287,818 B | Parquet 1,087,125 B — and the read:
t0 = time.perf_counter()
a = pd.read_parquet("wide.parquet")
t1 = time.perf_counter()
b = pd.read_csv("wide.csv")
t2 = time.perf_counter()
print(f"parquet read {1000*(t1-t0):.0f} ms | csv read {1000*(t2-t1):.0f} ms")
print("parquet ts:", a.ts.dtype, "| csv ts:", b.ts.dtype)parquet read 6 ms | csv read 67 ms parquet ts: datetime64[us] | csv ts: str
Same rows, same machine (pandas 3.0.5, pyarrow 25.0.1): 3x smaller file, ~11x faster read, and the timestamp column comes back typed instead of as strings — no parse_dates= needed. Your exact numbers will vary; the pattern won't.
| Flag | Meaning |
|---|---|
columns=[...] | Read only named columns — column pruning before anything is decoded; the cheapest speedup on wide files. |
filters=[[(col, op, val), ...]] | Predicate pushdown (pandas ≥ 2.1): skip row groups / partitions that can't match; inner list = AND, outer list = OR. |
engine="auto|pyarrow|fastparquet" | Backend choice; "auto" prefers pyarrow. Both must implement the identical Parquet spec, so files stay portable. |
dtype_backend="numpy_nullable"|"pyarrow" | Read with Arrow-backed or nullable dtypes — e.g. int64[pyarrow], large_string[pyarrow] — instead of NumPy defaults. |
to_parquet(partition_cols=[...]) | Write a partitioned dataset (pyarrow): one subdirectory per value, column encoded in Hive-style dir names. |
to_parquet(compression=...) | Per-file codec — snappy (default), gzip, zstd, brotli or "none"; smaller files cost extra CPU on write. |
to_parquet(index=None) | Whether to persist the index; the default None writes it only when it's more than a RangeIndex. |
storage_options={...} | Extra args for remote filesystems — S3 keys, tokens, endpoints — so the same call reads parquet straight off a bucket. |
read_parquet() and DataFrame.to_parquet() landed in pandas 0.21.0 (PyPI upload 2017-10-28), announced as "Integration with Apache Parquet" (GH issues 15838 and 17438). From day one the design was delegation: pandas implements none of the format, it dispatches to pyarrow or fastparquet and tries pyarrow first in engine="auto". Parquet itself was Twitter and Criteo's answer to columnar warehousing, open-sourced in 2013.
v0.24 (Jan 2019) added partition_cols to to_parquet (GH 23283). pandas 2.0 (Apr 2023) introduced dtype_backend and deprecated use_nullable_dtypes in favor of it. pandas 2.1 (Aug 2023) added the explicit filters= parameter to read_parquet (GH 53212) — before that, filtering meant dropping to the pyarrow API. pandas 3.0 added to_pandas_kwargs, a pass-through to pyarrow's Table.to_pandas.
pandas/io/parquet.py is a thin adapter: get_engine() resolves engine="auto" (option io.parquet.engine, else PyArrowImpl, else FastParquetImpl) and PyArrowImpl.read() just calls pyarrow.parquet.read_table, then arrow_table_to_pandas() rebuilds the frame with a zero-copy-ish cast. The speed story lives in the file layout: rows are split into row groups (default 1M rows, ~128 MB in Arrow's writer), and each column chunk inside carries min/max statistics in the footer. filters= evaluates your predicates against those statistics and skips whole row groups unread; columns= skips column chunks unread. Nothing is "queried" in your process — the format's metadata did the work before the read started.