CSV is what you send to people; Parquet is what you leave for yourself.
Your 800 KB DataFrame just shrank to 68 KB and now loads with the right columns pre-sorted — same file, no code changes.
df.to_parquet(path) serializes your DataFrame into the Parquet columnar format: values are grouped column by column, compressed, and written with the dtypes baked in. Call it with a directory plus partition_cols=['region'] and pandas lays out one subfolder per value — region=north/, region=south/ — which downstream readers can skip entirely when filtering.
CSV throws away your dtypes and makes every reader re-parse strings; Parquet preserves datetime64, floats and (when read back partitioned) categoricals exactly, so a write/read roundtrip satisfies df.equals() without a single converters argument. You also choose the codec: on my 200k-row test file, zstd produced 68 KB against 769 KB uncompressed. For anything that touches a data pipeline more than once, this is the format I reach for first.
df = pd.DataFrame({
"city": ["Berlin", "Munich", "Hamburg", "Cologne"],
"region": ["east", "south", "north", "west"],
"revenue": [1250.0, 980.5, 1520.25, 640.0],
"units": [42, 33, 51, 19],
})
df.to_parquet("/tmp/sales.parquet")
back = pd.read_parquet("/tmp/sales.parquet")
print(back.shape)
print(back.dtypes)
print(back.equals(df))(4, 4) city str region str revenue float64 units int64 dtype: object True
The roundtrip keeps dtypes and even the index intact — .equals() returns True with zero conversion arguments. Try that with a CSV and your datetime column comes back as object.
n = 200_000
big = pd.DataFrame({
"store": [f"store-{i % 250:03d}" for i in range(n)],
"product": [f"P{i % 40}" for i in range(n)],
"revenue": [round(i % 997 * 0.37, 2) for i in range(n)],
"units": [i % 60 for i in range(n)],
})
for comp in [None, "snappy", "gzip", "zstd"]:
big.to_parquet(f"/tmp/big_{comp}.parquet", compression=comp)
print(f"{comp!r:10} -> {os.path.getsize(f'/tmp/big_{comp}.parquet'):>7} bytes")None -> 769152 'snappy' -> 123279 'gzip' -> 72064 'zstd' -> 68301
Measured on my machine (pandas 3.0.3, pyarrow 25.0.1): zstd came out ~11× smaller than uncompressed and ~9× smaller than snappy on this repetitive synthetic data — your ratios depend on the data, so measure your own.
readings = pd.DataFrame({
"ts": pd.date_range("2026-01-01", periods=6, freq="D"),
"region": ["north", "north", "south", "south", "north", "south"],
"temp_c": [4.2, 3.8, 11.5, 12.1, 2.9, 13.0],
})
readings.to_parquet("/tmp/sensors", partition_cols=["region"])
# on disk:
# sensors/region=north/<uuid>.parquet
# sensors/region=south/<uuid>.parquet
south = pd.read_parquet("/tmp/sensors", filters=[("region", "=", "south")])
print(south)
print(south.dtypes)ts temp_c region 0 2026-01-03 11.5 south 1 2026-01-04 12.1 south 2 2026-01-06 13.0 south south dtypes: ts datetime64[us] temp_c float64 region category dtype: object
partition_cols builds one folder per region value; the filters=[...] read prunes whole directories instead of scanning rows — and the region column comes back as category for free.
| Flag | Meaning |
|---|---|
path=None | Write to a file; pass None (default) and the method returns the Parquet bytes instead of touching disk. |
engine='auto' | Picks pyarrow or fastparquet, whichever is importable; pin it when a server runs both. |
compression='snappy' | Codec per write: snappy is the fast default, gzip the safe classic, zstd the modern size/speed sweet spot. |
index=None | Let pandas decide whether the index is stored; set index=False to drop it and get a smaller file. |
partition_cols | List of low-cardinality columns to split into Hive-style directories (region=north/, region=south/...). |
storage_options | Extra credentials for remote paths — s3://, gs://, abfs:// — passed straight to the filesystem layer. |
Wes McKinney shipped to_parquet with the big IO layer rewrite in pandas 0.13 (January 2014), pitching Parquet as roughly 10× smaller and faster than CSV for large tables. The original implementation used its own Java-bridge writer; the format has outlived every engine that has driven it.
Modern pandas writes Parquet through engine libraries — pyarrow or fastparquet, selected by engine='auto'. That means format advances (zstd, dictionary pages, larger row groups) arrive with your engine upgrades, not with pandas releases.
A Parquet file stores data column-wise: each column chunk carries min/max statistics per row group, which is what lets a filtered read skip blocks it doesn't need — predicate pushdown without running a database. Each chunk is divided into pages that are compressed independently; and when path is None the same bytes are simply returned, which is how you stream a DataFrame into S3 or an HTTP response. One dtype surprise lives on the read side: pandas re-infers partition columns as category, so .equals() against your pre-partition frame fails even though every value matches.