Polars — Lazy, Eager, and Expression API

The modern Pandas alternative. Rust core, multi-threaded, Arrow-native, lazy-optional. 5-30× faster than Pandas on typical workloads, with cleaner API for production pipelines.

Eager vs Lazy

Polars has two execution modes:

Eager — like Pandas

import polars as pl

df = pl.read_csv("orders.csv")
filtered = df.filter(pl.col("status") == "completed")
grouped = filtered.group_by("user_id").agg(pl.col("amount").sum())
sorted_df = grouped.sort("amount", descending=True)

Each operation runs immediately. Similar to Pandas semantics.

Lazy — query-planning

result = (
    # scan = lazy read
    pl.scan_csv("orders.csv")
    .filter(pl.col("status") == "completed")
    .group_by("user_id")
    .agg(pl.col("amount").sum())
    .sort("amount", descending=True)
    .collect()                          # execute now
)

scan_* returns a LazyFrame. Operations build a query plan; .collect() triggers execution with optimization (predicate pushdown, projection pushdown, common subexpression elimination).

For files: lazy reads only the columns you actually use and only the rows that pass filters — huge wins.

Rule: use lazy for file-based and pipeline workloads. Use eager for quick interactive exploration.

Expression API

Polars’ killer feature. Operations are expressions; expressions compose; the engine optimizes them.

df.with_columns(
    pl.col("amount").sum().over("user_id").alias("user_total"),
    (pl.col("amount") / pl.col("amount").sum().over("user_id")).alias("share"),
    pl.when(pl.col("amount") > 100).then("high").otherwise("low").alias("tier"),
)

Vs Pandas’ patchwork of .groupby().transform(), .apply(), np.where() — Polars has one consistent expression language.

Common expression operators

pl.col("a")                                  # column reference
pl.col("a", "b")                             # multiple columns
pl.col(pl.Int64)                             # all columns of a type
pl.col("^prefix.*$")                         # regex column selector

pl.col("amount").sum()                       # aggregate
pl.col("amount").mean()
pl.col("amount").quantile(0.95)
pl.col("amount").rolling_mean(window_size=7) # window function
# cumulative
pl.col("amount").cum_sum()

pl.col("name").str.contains("foo")           # string
pl.col("name").str.to_lowercase()
pl.col("ts").dt.year()                       # datetime
pl.col("ts").dt.weekday()

pl.when(...).then(...).otherwise(...)         # conditional
pl.struct(["a", "b"])                         # combine columns

Window functions

df.with_columns(
    pl.col("amount").sum().over("user_id").alias("user_total"),
    pl.col("amount").rank(method="dense").over("user_id").alias("rank_in_user"),
)

.over(...) is the Polars window function. Like SQL’s PARTITION BY. Same expression language; no separate API.

Reading data

# Eager
df = pl.read_parquet("file.parquet")
df = pl.read_csv("file.csv")
df = pl.read_json("file.json")
df = pl.read_database("SELECT * FROM users", connection_uri="postgres://...")

# Lazy (for files only)
ldf = pl.scan_parquet("file.parquet")
ldf = pl.scan_csv("file.csv")
ldf = pl.scan_ndjson("file.json")

Reading partitioned datasets:

ldf = pl.scan_parquet("s3://bucket/orders/*.parquet")     # glob
ldf = pl.scan_parquet("orders/", hive_partitioning=True)   # year=2026/month=5/...

hive_partitioning=True enables partition pruning — filter on year or month and only the matching partitions are read.

Joins

left.join(right, on="key", how="inner")        # default
left.join(right, on="key", how="left")
left.join(right, on="key", how="outer")
left.join(right, on="key", how="semi")          # rows in left that have a match in right
left.join(right, on="key", how="anti")          # rows in left with NO match in right
left.join(right, left_on="user_id", right_on="id", how="inner")

# Join by struct of columns
left.join(right, on=["a", "b"], how="inner")

semi and anti are common in Polars but verbose in Pandas (which requires isin or merge + filter).

Aggregations

df.group_by("user_id").agg(
    pl.col("amount").sum().alias("total"),
    pl.col("amount").mean().alias("avg"),
    pl.col("amount").count().alias("n"),
    pl.col("status").value_counts().alias("status_counts"),
    pl.col("ts").max().alias("last_seen"),
)

Multiple aggregations in one .agg() call. Each expression is independent — Polars runs them in parallel.

Streaming engine

For data larger than RAM:

result = (
    pl.scan_parquet("huge.parquet")
    .filter(pl.col("year") == 2026)
    .group_by("country")
    .agg(pl.col("revenue").sum())
    .collect(streaming=True)
)

Streams chunks through the query plan; aggregations spill to disk if needed. Not all operations support streaming (joins are the limitation); Polars warns when a step falls back to eager.

For 50 GB data on a 16 GB machine, this still works.

DataFrame vs Series

Polars distinguishes columns (Series) and tables (DataFrame) like Pandas, but with stricter semantics:

• Indices don’t exist by default (Polars is index-free, unlike Pandas’ RangeIndex / MultiIndex).
• Operations between Series respect order, not labels.
• No SettingWithCopyWarning — Polars is functional; transformations return new frames.

Comparison to Pandas

# Pandas
df.groupby("user_id")["amount"].agg(["sum", "mean"])

# Polars
df.group_by("user_id").agg([pl.col("amount").sum(), pl.col("amount").mean()])

# Pandas conditional
import numpy as np
df["tier"] = np.where(df["amount"] > 100, "high", "low")

# Polars
df = df.with_columns(pl.when(pl.col("amount") > 100).then(pl.lit("high")).otherwise(pl.lit("low")).alias("tier"))

# Pandas rolling
df["ma"] = df["price"].rolling(7).mean()

# Polars (eager)
df = df.with_columns(pl.col("price").rolling_mean(window_size=7).alias("ma"))

Roughly Pandas-equivalent operations. Migration mostly mechanical, but the expression API rewards rethinking operations into expressions.

Lazy execution plan inspection

ldf = (
    pl.scan_parquet("data.parquet")
    .filter(pl.col("status") == "completed")
    .group_by("user_id")
    .agg(pl.col("amount").sum())
)

print(ldf.explain())
# OPTIMIZED PLAN: shows pushdown of filter into the scan, etc.

Like SQL’s EXPLAIN. Confirms predicate pushdown happened.

ldf.explain(streaming=True) shows the streaming plan.

Arrow integration

Polars uses Apache Arrow as its in-memory format. Zero-copy interop with:

• PyArrow
• DuckDB
• numpy (for many types)
• Spark (via Arrow exchange)
• Many other Arrow-aware tools

import pyarrow as pa
arrow_table = pa.table({"a": [1, 2, 3]})
polars_df = pl.from_arrow(arrow_table)
arrow_back = polars_df.to_arrow()

For workflows that bridge Python ML tooling (pandas, scikit-learn) and big-data tools, Arrow as the lingua franca is huge.

Common gotchas

• pl.col vs pl.lit: pl.col("x") references a column; pl.lit("x") is a literal value. In when().then(pl.lit("high")), pl.lit is needed because "high" isn’t a column.
• with_columns returns a new frame. Polars is functional; reassign.
• Schema strict. Polars infers types and enforces them. A column inferred as Int64 won’t accept strings later.
• null vs NaN: Polars uses null (Arrow null). NaN is a float value; null is missing. Pandas conflates them (np.nan for everything missing); Polars distinguishes.
• group_by (Polars 1.0+) — was groupby in earlier versions. Old code uses both spellings.
• Streaming engine limitations — joins, some window operations don’t stream yet. Use eager for those.

When Polars beats Pandas

• Any production pipeline: faster, lower memory, lazy optimization.
• Reading large CSV / Parquet: predicate + projection pushdown.
• Joining large DataFrames: multi-threaded.
• Out-of-core data: streaming engine.
• Memory-constrained environments: more efficient layout.

When Pandas still wins

• Interactive notebooks where Pandas idioms are second nature.
• Integration with the wider Python ML ecosystem (most libraries accept Pandas first, Polars second).
• Plotting (matplotlib / seaborn integrate cleaner with Pandas).
• Tiny datasets where speed doesn’t matter.
• Code maintained by Pandas-fluent team.