Skip to content
MINH VO A working notebook
by an engineer in Vietnam
Foundation9 min read

See how dictionary encoding handles cardinality

Compare repeated country codes and unique request IDs in real Parquet files, inspect dictionary metadata, and keep local array indices separate from stable identities.

Paper charts with an olive scatter plot, histogram, and data table on a beige desk

A column with three country codes and a column with thousands of unique request IDs can benefit very differently from dictionary encoding. The decision concerns repeated values inside a particular column chunk. It is not settled by the table’s row count or by switching a file-wide option on and assuming every column becomes smaller.

This article is for a reader who knows what a Parquet column and row group are. We will decode a small dictionary by hand, write three actual Parquet variants and compare their stored sizes while checking every recovered value. The examples use Python 3.12.3 and PyArrow 23.0.1 with synthetic data. They measure file and column-chunk lengths, not query latency or process memory.

Start with values, positions and a local dictionary

An Arrow dictionary array contains dictionary values and indices that select them. The following eight observations contain three distinct non-null country labels. A null observation remains null; it does not become a fourth country or the string "NULL".

import pyarrow as pa
import pyarrow.parquet as pq

values = pa.array(['VN', 'US', 'VN', None, 'DE', 'VN', 'US', 'DE'])
encoded = values.dictionary_encode()
print(encoded.dictionary.to_pylist())
print(encoded.indices.to_pylist())
assert encoded.dictionary_decode().equals(values)

The printed dictionary is ['VN', 'US', 'DE'], and the indices are [0, 1, 0, None, 2, 0, 1, 2]. Using zero-based positions, at position two, index zero refers back to VN. At position three, the observation is missing. The equality assertion verifies all eight values after decoding, including their order and null position.

The Arrow DictionaryArray reference describes this in-memory representation. The diagram shows its logical relationship, not the byte layout of a Parquet page or an Arrow validity bitmap. An ordinary integer column with the same numbers would not explain what those numbers mean without its dictionary.

Eight synthetic country observations map through local dictionary indices; the null observation remains missing rather than becoming a dictionary categoryView full-size image ↗

Count distinct values within the encoded scope

Cardinality here means the number of distinct non-null values in the selected column’s data. Repeated values can be stored once in a dictionary while many rows use shorter indices. If almost every value is unique, the dictionary must retain almost all the original values and the file also needs an index stream.

The Parquet encoding specification places the dictionary in a dictionary page for a column chunk. Data pages refer to its entries using run-length encoding and bit packing. Different chunks can have different dictionaries. A table-wide count of distinct customers therefore does not fully describe the dictionary economics of each row group.

Value length matters too. Saving repeated long strings can offset more dictionary overhead than saving repeated short strings. Ordering matters because a long run of the same index can be represented differently from alternating indices. Cardinality is a useful input to the decision, not a complete size formula.

For this experiment, keep ordering fixed. The country column cycles through VN, US and DE; the request column has one different ten-byte ASCII string per row. All values are present. The null-bearing eight-row example above explains semantics, while this second fixture keeps the stored-size comparison simpler.

Write the same table with three explicit settings

The Arrow writer options allow dictionary encoding to be disabled, enabled for selected columns or enabled generally. Here we use one row group containing 4,096 rows, no compression codec, statistics enabled and an explicit one-mebibyte dictionary-page limit. Every variant starts from the same ordinary string table.

row_count = 4096
table = pa.table({
    'country': [['VN', 'US', 'DE'][i % 3] for i in range(row_count)],
    'request_id': [f'req-{i:06d}' for i in range(row_count)],
})
settings = {'plain': False, 'country_only': ['country'], 'both': True}
payloads = {}
for label, dictionary_setting in settings.items():
    output = pa.BufferOutputStream()
    pq.write_table(table, output, compression='NONE',
                   row_group_size=row_count,
                   use_dictionary=dictionary_setting,
                   dictionary_pagesize_limit=1024 * 1024,
                   write_statistics=True)
    payload = output.getvalue()
    payloads[label] = payload
    assert pq.read_table(pa.BufferReader(payload)).equals(table)
    group = pq.ParquetFile(pa.BufferReader(payload)).metadata.row_group(0)
    print(label, len(payload),
          group.column(0).total_compressed_size,
          group.column(1).total_compressed_size)

The four printed fields are the setting name, complete file bytes, country column-chunk bytes and request-ID column-chunk bytes. These are actual lengths from this runtime:

SettingWhole fileCountry chunkRequest-ID chunk
plain82,56724,61957,403
country_only59,0621,10757,403
both65,2451,10763,574

Enabling the country dictionary reduces its chunk substantially. Enabling the unique request-ID dictionary increases that chunk from 57,403 to 63,574 bytes. The both file is still smaller than plain because the country saving exceeds the request-ID increase. It is larger than country_only. Inspecting only the whole-file comparison would hide the unfavorable change in one column.

The metadata property is named total_compressed_size even though these files use compression='NONE'. Here it reports the stored chunk size without codec compression. Dictionary encoding is still active in the selected columns. File length also includes the footer and other framing, so adding the two chunk lengths does not reconstruct the entire file.

The assertion compares the complete table after each write. A storage improvement would not be useful if it changed an identifier, moved a country to another row or dropped an observation. Retain the source construction, runtime and serialized artifact hashes with the result; a later writer can produce different bytes while preserving the same values.

Measured country and request-ID column-chunk sizes show country-only dictionary encoding saving space while encoding unique request IDs adds bytesView full-size image ↗

Explain the direction without pretending to predict every byte

For these ASCII strings, the PLAIN representation stores four length bytes followed by the string bytes. Ignoring page framing and other format details, the country values therefore require 4096 × (4 + 2) = 24,576 bytes when repeated directly. A three-entry dictionary stores 18 bytes of values. An ideal stream of 4,096 two-bit indices adds 1,024 bytes.

That simplified 1,042-byte total explains the opportunity; it does not predict the measured 1,107-byte chunk exactly. The actual format has its own headers, index encoding and level information. Do not present the ideal bit count as an observed file size.

For request IDs, the values alone require 4096 × (4 + 10) = 57,344 bytes. A dictionary with 4,096 distinct entries still needs those values, while an ideal twelve-bit index per row adds 6,144 bytes. This simplified accounting explains why the unique column can grow. It is not a universal rule that dictionary encoding is always bad above a particular cardinality, especially when codecs, longer strings or different row-group boundaries enter the comparison.

Read metadata without inventing page-level evidence

Inspect whether each column chunk has a dictionary-page offset and which encodings its metadata lists:

for label, payload in payloads.items():
    group = pq.ParquetFile(pa.BufferReader(payload)).metadata.row_group(0)
    print(label, [group.column(i).dictionary_page_offset is not None
                  for i in range(2)])
    print(label, [group.column(i).encodings for i in range(2)])

Dictionary-page presence is [False, False], [True, False] and [True, True] for the three settings. The selected columns list PLAIN, RLE and RLE_DICTIONARY. This confirms dictionary pages are present where requested in these files. It does not count how many data pages use each encoding.

In particular, PLAIN in that list is not proof of fallback: the specification itself uses PLAIN for dictionary-page entries. The format permits fallback when a dictionary becomes too large. The writer’s configured size limit is relevant to that behavior, but this tutorial does not decode individual page headers or claim to locate a fallback boundary. A page-level claim requires page-level evidence.

Keep the read representation separate from the stored encoding

A reader can construct an Arrow dictionary array even from a Parquet column stored without dictionary encoding. The Arrow read reference exposes read_dictionary for this choice on supported columns. Read the country column from the plain file this way:

read_back = pq.read_table(pa.BufferReader(payloads['plain']),
                          read_dictionary=['country'])
country = read_back.column('country').combine_chunks()
print(pa.types.is_dictionary(country.type), len(country.dictionary))
assert country.dictionary_decode().equals(table.column('country').combine_chunks())

other = pa.array(['US', 'VN', 'US']).dictionary_encode()
print(encoded.indices[0].as_py(), encoded.dictionary[0].as_py())
print(other.indices[0].as_py(), other.dictionary[0].as_py())

The first output is True 3: the returned country array has a dictionary type with three entries. The earlier metadata still says that the original file has no dictionary page for this column. A dictionary in memory is therefore not evidence that the writer used a dictionary on disk.

The last two lines print 0 VN and 0 US. Each index zero is correct within its own array. Comparing those raw indices across batches would incorrectly equate different countries. Decode the labels or use a separately defined stable identifier before joining analytical records. A local dictionary index is an implementation detail, not a business key.

Wes McKinney’s September 5, 2019 Arrow article explains the distinction between Parquet encoding and Arrow dictionary arrays, including changing chunk dictionaries and handling fallback. Its performance measurements compare historical Arrow versions. They are not timings for this fixture or a promise about a current workload.

Change one assumption and verify the result again

For an exercise, make every request ID equal to req-shared, rebuild all three variants and verify complete roundtrips. Predict the direction of the request-column size change before reading its metadata. The repeated value now offers a dictionary opportunity that the unique identifiers did not.

Then restore the unique requests and split the original table into smaller row groups. Count actual dictionary pages by checking each chunk’s dictionary-page offset, and inspect the resulting sizes. Additional chunks introduce separate dictionaries and framing. Keep that experiment distinct from the single-group baseline rather than explaining every change with cardinality alone.

Choose a setting from the columns and consumers you actually have. The smallest file in this example uses dictionaries selectively. A real decision also needs supported-reader tests and, when relevant, separate measurements of read time, write time and memory. The useful conclusion here is narrower: the same rows preserve their meaning under all three settings, while their columns have different measured storage costs.

Sources & further reading

  1. Apache Parquet encodings
  2. Apache Arrow 23.0 DictionaryArray
  3. Wes McKinney, Faster C++ Apache Parquet performance on dictionary-encoded string data coming in Apache Arrow 0.15, September 5 2019
  4. Apache Arrow 23.0 pyarrow.parquet.read_table
  5. Apache Arrow 23.0 pyarrow.parquet.write_table
← Back to the journal
All notes

Illustration

100%