Apache Parquet is the recommended file format used in every modern data platform, and for good reason. But what are those reasons? And would it really matter if you stuck with CSV?
The short answer is “YES”. The slightly longer answer is “Yes, because columns”.
The full answer is below. This post goes past the usual summary and into the file itself: what the bytes are arranged like, which encodings do the compression work, and the three separate mechanisms an engine uses to avoid reading your data. For the higher-level comparison against CSV and Avro, start with the Apache Parquet guide.
CSV and most traditional formats store data row by row, with every record written out in full before the next one begins. For a table with 200 columns and 500 million rows, a query that only needs three columns still has to read all 200 columns worth of data from disk to get them. The irrelevant 197 columns come along for the ride.
Parquet flips this. It stores data column by column: all values for transaction_date together, all values for revenue together, all values for customer_id together. A query that needs those three columns reads only those three columns from storage. The other 197 never leave the disk.
This is known as column pruning, and its impact compounds quickly on wide tables. Combined with predicate pushdown, which applies filters before data is read into memory rather than after, analytical engines end up reading a fraction of what a row-based format would force them to touch.
A Parquet file is not a stream you read from the front. It is a container you read from the back.
4-byte magic number "PAR1"
<Column 1 Chunk 1>
<Column 2 Chunk 1>
...
<Column N Chunk 1>
<Column 1 Chunk 2>
...
File Metadata
4-byte length of the file metadata (little endian)
4-byte magic number "PAR1"The metadata sits at the end rather than the beginning, and that is a deliberate design decision. A writer cannot know the final statistics, byte offsets or encodings for a column until it has finished writing that column, so putting the metadata last allows the whole file to be produced in a single pass without buffering everything or seeking backwards.
The cost lands on the reader, and it is a cost worth paying. A reader opens the file, seeks to the last eight bytes, reads the four-byte metadata length and the closing magic number, then jumps back by that length to read the footer. Only then does it know which byte ranges hold the columns it wants. On object storage this is typically two range requests before a single value is decoded.
The practical consequence shows up at scale. Every Parquet file carries this fixed overhead, so a query touching ten thousand small files pays ten thousand footer reads before it does any real work. That is the mechanism behind the small files problem, and it is why compaction matters so much in practice.
Parquet nests three levels of structure between the file and the values, and knowing which is which makes the tuning advice make sense.
A row group is a horizontal slice of the table, a set of complete rows. It is the unit of parallelism. One task reads one row group, which is why a file with a single enormous row group cannot be split across workers no matter how large it is.
Within a row group, each column occupies one contiguous column chunk. This is the unit that makes column pruning physically possible, because a reader can seek straight to the chunk it wants and ignore the byte ranges belonging to other columns.
Each column chunk is divided into pages, and the page is the smallest unit that can be compressed and encoded independently. Pages are treated as indivisible, so a reader cannot decode half a page.
The format specification recommends row groups of 512 MB to 1 GB and data pages of around 8 KB. Those row group numbers come from an era of HDFS block alignment, and most engines writing to object storage today target something smaller. Treat the specification as the reasoning rather than the setting: larger row groups give longer sequential reads and more write-side buffering, smaller ones give finer-grained skipping and better parallelism.
Compression ratios are the second area where columnar storage wins, and most of the win happens before a compression codec is involved at all. Storing values of one type together lets Parquet pick an encoding suited to the shape of that specific column.
Only after encoding does a compression codec run over the page. Parquet supports uncompressed, Snappy, GZIP, Brotli, ZSTD and LZ4_RAW, and the choice is per column chunk rather than per file. Snappy decompresses fast and compresses modestly, ZSTD gives a better ratio at more CPU, and GZIP sits between them with the widest compatibility. The older LZO and LZ4 codecs are deprecated in favour of LZ4_RAW.
Because the codec is set per column, a single file can use ZSTD on a large rarely filtered text column and Snappy on the columns that every query touches.
People often describe Parquet as having “statistics”, as if there were one mechanism. There are three, they live in different places, and they fail in different ways.
Each column chunk records minimum and maximum values plus a null count in the file metadata. An engine reads the footer, compares a predicate against those bounds, and skips entire row groups that provably cannot match. This is the cheapest check because the footer was going to be read anyway.
Chunk-level bounds only get you to a row group. Finer skipping used to require reading each page header in turn, which meant touching most of the column data to decide what not to read. The page index solves that with two structures stored near the footer: a ColumnIndex holding per-page bounds so a reader can locate pages matching a predicate, and an OffsetIndex mapping pages to row ranges so the corresponding rows can be skipped in every other column.
The payoff is large for selective queries. A single-row lookup on a sorted column can read one data page per column rather than the whole chunk. Readers that do no selective filtering can ignore the index entirely and pay nothing for it.
Min and max bounds only eliminate data when values are clustered. A column of random identifiers spread evenly across every row group gives every chunk roughly the same minimum and maximum, so no predicate eliminates anything and the statistics become overhead that never pays for itself.
This is the single most common reason teams conclude that Parquet statistics do not work for them. The format is doing its job. The data was written in an order that made the bounds useless. Sorting or clustering on the columns you actually filter by is what turns the mechanism on.
Parquet is a flat columnar format that nonetheless handles arrays, maps and nested structs. It does this with the Dremel model of definition and repetition levels.
Definition levels record how many optional fields along a column path are actually present for a given value. Repetition levels record at which repeated field in the path a value starts a new list. Between them, the original nested structure can be rebuilt from flat columns.
Nullity falls out of the same mechanism. Null values are not stored at all. A null is a definition level lower than the maximum, and because levels are run-length encoded, a column of a thousand consecutive nulls costs close to nothing. That is worth remembering before anyone proposes a sentinel value such as -1 or an empty string to “avoid nulls”. Sentinels are stored in full, nulls are not, and sentinels also pollute the min and max bounds that drive skipping.
Every file carries a footer, and every footer is a seek plus a read before any value is decoded. Streaming jobs that commit every few seconds produce thousands of files whose metadata overhead dwarfs their content.
One row group per file removes the ability to parallelise within that file. Very small row groups multiply metadata and shorten every sequential read. Both ends hurt.
Covered above, and worth repeating because it is usually diagnosed as a Parquet problem rather than a write-order problem.
A UUID column defeats dictionary encoding and produces useless bounds. If queries filter on it constantly, that is an argument for a different physical layout rather than for different Parquet settings.
Everything above is observable in about ten lines of Python. This prints the row group count, then the compression, encodings and bounds for every column in the first row group.
import pyarrow.parquet as pq
f = pq.ParquetFile("events.parquet")
md = f.metadata
print(md.num_rows, "rows in", md.num_row_groups, "row groups")
rg = md.row_group(0)
for i in range(rg.num_columns):
c = rg.column(i)
print(f"{c.path_in_schema:24} {c.compression:8} "
f"{str(c.encodings):28} "
f"min={c.statistics.min} max={c.statistics.max} "
f"nulls={c.statistics.null_count}")Three things are worth checking on a file from your own warehouse. Whether the encodings list mentions a dictionary, because its absence on a column you expected to repeat means cardinality defeated it. Whether the minimum and maximum for your most common filter column differ meaningfully between row groups, because if they do not, skipping is not happening. And whether the row group count is one, which means the file cannot be split across workers.
DuckDB answers the same questions in SQL if you would rather not leave a query editor:
SELECT path_in_schema, compression, encodings, stats_min, stats_max
FROM parquet_metadata('events.parquet')
WHERE row_group_id = 0;While Apache Iceberg is compatible with multiple file formats, including Avro and ORC, Parquet has become the default for Iceberg tables and for other table formats such as Delta Lake. When Dremio, Spark, Trino or DuckDB query an Iceberg table, they are reading Parquet files on object storage. The Iceberg layer adds metadata, transaction support and schema evolution, but Parquet is the physical format those engines scan.
The two layers do similar work at different scales, and they stack. Iceberg keeps per-file statistics in its manifest files, which lets an engine eliminate whole files before opening any of them. Parquet then eliminates row groups and pages inside the files that survive. The life of a read query walks through that full sequence.
Parquet is supported across effectively every analytics engine, language and cloud platform in production today, which is what makes it the default rather than merely a recommendation.
The fastest way to stop treating Parquet as a black box is to read a footer. Point PyArrow or DuckDB at a file you already have, print the row group count, the per-column encodings and the min and max values, and compare that against what your queries actually filter on.
Most of the time the mismatch is obvious within a minute, and it is almost always the same mismatch: the data was never sorted by the thing everyone queries.
Because a writer cannot know a column's final statistics, byte offsets and encodings until it has finished writing that column. Putting the metadata last allows the file to be written in one pass. The reader pays for it by seeking to the last eight bytes first, reading the metadata length, then jumping back to the footer.
A row group is a horizontal slice of the table and the unit of parallelism, so one task reads one row group. A page sits several levels below, inside a column chunk, and is the smallest unit that can be independently encoded and compressed. Pages are indivisible.
Almost always because the data is not sorted or clustered on the column being filtered. Minimum and maximum bounds only eliminate a row group when values are clustered. Randomly ordered data gives every chunk nearly identical bounds, so nothing can be ruled out.
Snappy when decompression speed matters most, ZSTD when storage and scan size matter more than CPU, GZIP when the widest compatibility is the priority. The codec is chosen per column chunk, so one file can mix them. LZO and the original LZ4 codec are deprecated in favour of LZ4_RAW.
No. Parquet does not store nulls at all. Nullity is carried in the run-length encoded definition levels, so a long run of nulls costs almost nothing. A sentinel such as -1 or an empty string is stored in full and also widens the min and max bounds that drive skipping.
With the Dremel model of definition and repetition levels. Definition levels record how many optional fields in a path are present, and repetition levels record where a new list begins. Together they let the nested structure be rebuilt from flat columns.
No. Parquet is a file format that describes how one file stores values. Iceberg is a table format that describes which files make up a table, plus schema evolution, snapshots and transactions. Iceberg tables normally contain Parquet files, so the two work together rather than competing.
For a deeper treatment of the table format layered above Parquet, download Apache Iceberg: The Definitive Guide by Tomer Shiran, Jason Hughes, and Alex Merced, free from Dremio.
To see these ideas applied in practice, explore how Dremio approaches the open data lakehouse.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
By unifying data from diverse sources, simplifying data operations, and providing powerful tools for data management, Dremio stands out as a comprehensive solution for modern data needs. Whether you are a data engineer, business analyst, or data scientist, harnessing the combined power of Dremio and Apache Iceberg will undoubtedly be a valuable asset in your data management toolkit.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
This blog helps you learn about table-driven access policies in Dremio Cloud and Dremio Software v24.1+.
