Eighty percent of a clean DataFrame is decided at read_csv() time.
Your CSV never survives contact with read_csv() unchanged — and that's the point.
read_csv() reads delimited text into a DataFrame in one call. It sniffs headers, infers dtypes per column, splits NA detection from parsing, and hands you a table — with 44 parameters to steer every step. The defaults are good; the craft is knowing which four to touch.
The type coercion happens the moment the file is read: '1,199.00' becomes an object column, a stray blank in a units column floats everything to float64, dates stay strings unless you ask. Fixing types after the fact is rework; read_csv() lets you fix them at the door — then usecols trims the file down before the parser even builds the columns.
You received orders_q3.csv. Look at what inference did to it before you touch anything:
order_id int64 region str product str units float64 unit_price_eur str order_date str dtype: object
One blank cell turned units into float64, the thousands separator froze unit_price_eur as str, and dates stayed text. Three fixes, all at read time.
Right-size the dtypes so the DataFrame matches reality — IDs as int32, counts as float32:
order_id int32 region str product str units float32 unit_price_eur str order_date str dtype: object
dtype takes a dict {column: type}. int32 is half the memory of int64 for IDs, and pandas 3.0 backends the str columns on Arrow strings by default.
European exports use thousands separators, and you only need four columns with real dates. Strip, parse, trim — in one call:
order_id region unit_price_eur order_date 0 1001 North 1199.00 2026-07-14 1 1002 South 229.90 2026-07-15 2 1003 East 1199.00 2026-07-18 3 1004 West 89.50 2026-07-21 4 1005 North 229.90 2026-08-02 5 1006 East 59.99 2026-08-09 6 1007 South 1199.00 2026-08-27 7 1008 West 229.90 2026-09-05 datetime64[us]
converters run per cell and beat inference; parse_dates turns the column into datetime64[us]; usecols drops the other two columns before the parser builds them. Date span here: 53 days, 14 Jul to 5 Sep.
The file is 40 GB and your RAM is not. Stream it in chunks and accumulate:
13981.45
chunksize=3 yields an iterator of DataFrames; each chunk is garbage-collected after you use it, so memory stays flat no matter how big the file. The sum matches the one-shot read.
| Flag | Meaning |
|---|---|
sep / delimiter | Field separator; pass sep=None to let the C engine sniff it from the first rows. |
usecols | Read only named columns — saves parse time and memory before the DataFrame exists. |
dtype | Dict of {column: type}; wins over inference, ideal for int32 IDs or string columns that inference would mangle. |
parse_dates / date_format | Columns to parse as dates; pair with date_format to nail non-ISO layouts. |
converters | Dict of {column: function} applied per cell — the escape hatch when no parameter can clean the value. |
na_values / keep_default_na | Extra strings to treat as missing; keep_default_na=False stops 'NA' (Namibia) becoming NaN. |
na_filter / chunksize | na_filter=False skips NA scanning on clean files; chunksize=N streams big files as an iterator of DataFrames. |
read_csv has been in pandas since the 0.4 series in 2009, born as a Python wrapper around the matplotlib mlab.csv2rec loader.
0.10 (Dec 2012) moved parsing into a C engine by default, an order of magnitude faster and the shape you use today; 0.21 (Oct 2017) added engine='python' as the documented opt-in fallback.
The C engine is a tokenizer in pandas/_libs/parsers.pyx (Cython) feeding a C-level tokenizer derived from numpy's parser. It scans once, classifying each field as int, float, bool or string; the tokenizer's fast paths for common numeric formats decide your dtypes before any Python code runs. usecols, na_values and dtype are all resolved inside that single pass — which is why reading a 40 GB file with chunksize costs the same total parse time as one big read, but flat memory.