What Is Parquet? Column-Oriented Storage Explained with Racing Data

When working with large datasets in Python, you may come across a file format called Parquet.

You know CSV, but what makes Parquet different? You may have heard the term “column-oriented” without having a clear picture of what it means.

Let’s use racing telemetry to explore the basics. We will create a Parquet file with Python, then compare file sizes and read times using real Gran Turismo 7 (GT7) data.

1. What is Parquet?

Parquet is a file format designed to store and read data efficiently.

Like CSV, it can hold tabular data. The main difference is how that data is stored.

For example, consider this telemetry from a racing car:

Time (s)	Speed (km/h)	Throttle (%)	Brake (%)
0.0	100	50	0
0.1	105	60	0
0.2	110	70	0
0.3	108	0	30

Speed, throttle position, and brake input are recorded at each point in time.

Real racing datasets can contain hundreds or even thousands of channels, including engine speed, temperatures, and suspension travel.

Parquet helps make large datasets like these more manageable.

2. What does column-oriented mean?

The key idea behind Parquet is column-oriented storage.

A CSV file usually records data one row at a time. The telemetry above would look like this:

Time,Speed,Throttle,Brake
0.0,100,50,0
0.1,105,60,0
0.2,110,70,0
0.3,108,0,30

Each row contains the time, speed, throttle, and brake values, in that order.

Parquet groups values from the same column together instead. A simplified picture looks like this:

Time     : 0.0, 0.1, 0.2, 0.3
Speed    : 100, 105, 110, 108
Throttle : 50, 60, 70, 0
Brake    : 0, 0, 0, 30

In other words, CSV groups data by row, while Parquet groups it by column.

The actual Parquet structure is more involved: it divides data into row groups and stores the columns within each group. For now, the simplified picture is enough to understand the basic idea.

3. Why does this suit racing data analysis?

Imagine, as an example, a telemetry dataset with 500 channels. The GT7 experiment later in this article uses 82 recorded columns.

Suppose you want to plot only speed and throttle.

With a typical CSV reader, processing the file involves scanning data for channels you will not use as well.

Parquet stores columns together, so a reader can efficiently load just the channels you need. It may greatly reduce the amount of data read, without loading all 500 channels.

Racing analysis rarely needs every channel for every task. You might compare speed, inspect braking, or look at a few suspension channels.

Column-oriented storage fits these workflows well.

Parquet is not always faster than CSV, though. The benefit depends on the dataset size, the reader, and the analysis you are doing.

4. Files often become smaller, too

Parquet also supports efficient compression.

Consider a sequence of brake inputs:

0, 0, 0, 0, 0, 0, 0, 0, 30, 50, 80

The value stays at zero while the driver is off the brake.

Keeping similar data together makes patterns such as repeated values easier to use when storing the data efficiently. Parquet supports encodings and compression methods that can take advantage of these characteristics.

As a result, Parquet files are often smaller than CSV files.

Smaller files can reduce storage requirements and the amount of data transferred when reading over a network.

5. How do you create a Parquet file?

Creating a Parquet file is simpler than the storage details might suggest. With Python, you can convert a CSV file in a few lines.

First, install the required libraries:

pip install pandas pyarrow

Then read the CSV and save it as Parquet:

import pandas as pd

df = pd.read_csv("telemetry.csv")
df.to_parquet("telemetry.parquet", index=False)

That’s it. This creates telemetry.parquet from telemetry.csv.

Here, we use PyArrow through pandas. PyArrow handles the column-oriented storage and compression, so you do not have to implement the conversion yourself.

You can also save a DataFrame created in Python directly as Parquet, without going through CSV first.

6. Reading a Parquet file

Reading the file is straightforward:

import pandas as pd

df = pd.read_parquet("telemetry.parquet")
print(df.head())

You can also take advantage of Parquet’s layout by reading only selected channels:

df = pd.read_parquet(
    "telemetry.parquet",
    columns=["Time", "Speed", "Throttle"]
)

The columns argument selects just the Time, Speed, and Throttle columns.

This becomes useful when a telemetry dataset contains many more channels than you need for a particular analysis.

7. Comparing real GT7 telemetry

So far, we have looked at the basic idea and the Python code. How much difference does it make with actual driving data?

We compared file sizes and read times using logs recorded in GT7.

The dataset

We combined 29 distinct files covering 11 tracks and 10 cars into a dataset of 205,002 rows. We did not repeat the same logs to inflate the dataset.

There are 82 recorded columns. We added one ID column to identify the source file, giving both the CSV and Parquet versions the same 83 columns. There are no missing values, and we verified that the values and data types match after reading.

The CSV used for comparison was saved from the original logs with consistent types and the added ID. Each file’s time values were preserved; we did not force them onto a continuous timeline.

File size and read time

We saved the same data as CSV and as Parquet using Snappy and ZSTD. Snappy and ZSTD are compression methods.

Format	Size (MiB)	All 83 columns (ms)	3 key columns (ms)
CSV	95.0	3,084.1	993.6
Parquet · Snappy	39.2	363.6	24.6
Parquet · ZSTD	30.7	424.1	27.5

Read times are the median of 15 measurements per condition. There are 1,000 ms in a second, and 1 MiB is 1,048,576 bytes.

The three key columns are game time (time_of_day_ms), speed (speed_kmh), and throttle (throttle_pct). We selected the same three columns with CSV’s usecols and Parquet’s columns. The time field is an in-game millisecond counter, so its name and unit differ from Time in the earlier illustrative example.

File size fell by about 59% with Snappy and 68% with ZSTD. In this test, Snappy had shorter read times for both all 83 columns and the three key columns. The smallest file was not necessarily the fastest to read.

What happens when you read fewer columns?

We also compared reading 3, 10, 25, 50, and 83 columns. We fixed the column selections before measuring: time, speed, and throttle first, then additional channels such as brake input and engine speed.

In this test, both CSV and Parquet took less time when fewer columns were selected.

Going from all 83 columns to three reduced CSV read time from about 3,084 ms to 994 ms. Snappy went from about 364 ms to 25 ms, and ZSTD from about 424 ms to 28 ms. These results illustrate why Parquet can suit repeated analysis of selected telemetry channels.

Test conditions and limitations

We used the usual pd.read_csv() settings for CSV and single-threaded PyArrow reads for Parquet. Measurements ran on the same PC’s local storage, after two warmup reads per condition, with the execution order randomized in each round. CSV-to-Parquet conversion time is not included in the read times.

Windows was restarted and unnecessary applications were closed before measuring. However, we did not control the operating system’s file cache or background activity, and available memory was limited. Treat these numbers as results for this dataset and environment.

The difference between formats also includes compression and numeric parsing. This experiment does not isolate the effect of column-oriented storage, and other datasets may produce different speed or compression results.

View detailed results by column count

Columns read	CSV median (ms)	Snappy median (ms)	ZSTD median (ms)
3	993.59	24.57	27.53
10	1,186.27	44.84	54.81
25	1,794.23	155.11	165.14
50	2,048.47	244.66	265.11
83	3,084.14	363.55	424.08

We compared CSV with Parquet (Snappy and ZSTD) using real GT7 data: 205,002 rows and 83 columns. Each format read only the selected columns. The table shows the median of 15 measurements per condition.

These results reflect this test’s conditions; they vary with the data and execution environment.

8. When should you use CSV or Parquet?

Parquet’s advantages do not make CSV obsolete.

Comparison	CSV	Parquet
Data representation	Text	Binary
Inspecting the contents	A text editor can open it	Requires a compatible tool
Reading selected columns	Depends on the method	Well suited to this
File size	Often larger	Often smaller
Common uses	Data exchange and quick checks	Storing and analyzing large datasets

CSV is convenient for checking a small dataset in Excel or sharing it with someone else.

Parquet is a strong option when you want to store large telemetry datasets and analyze them repeatedly.

In our GT7 test, it reduced storage size and made selected-column reads faster. One practical approach is to keep CSV for recording and sharing, then also convert data to Parquet for repeated analysis.

For an introduction to how GT7 data is received and the gt-telem Python library, see Can you capture GT7 telemetry? How receiving it with Python works. Once received data has been saved as CSV or another format, it can be converted to Parquet to suit the analysis.

Choose the format that fits the task.

Summary

We have explored how Parquet works, created files with Python, and compared formats using real GT7 telemetry.

The three main points are:

• Parquet is column-oriented: it stores values from the same column together.
• Efficient selected-column reads make it a good fit for analyzing telemetry with many channels.
• Converting CSV to Parquet takes only a few lines of Python.

For roughly 200,000 rows of GT7 logs, Parquet files were about 59–68% smaller than CSV, and selected-column reads showed a benefit. Results depend on the data and environment, so try it with the analysis you do most often.

Parquet may sound unfamiliar at first, but its basic idea is approachable. Converting a CSV you already have and comparing the file sizes and read behavior is a useful place to start.

References

For more detail on Parquet, these official documents are useful starting points:

• Apache Parquet: Overview — An introduction to Parquet’s main features.
• pandas: DataFrame.to_parquet() — Saving Parquet files and choosing compression settings.
• pandas: read_parquet() — Reading files and selecting columns.
• Apache Arrow: Reading and Writing the Apache Parquet Format — More detail on the PyArrow reader, writer, and compression options used underneath pandas.