Parallel Table.append

[bigluck]

Apache Iceberg version

main (development)

Please describe the bug 🐞

While doing some tests with the latest RC (v0.6.0rc5), I generated a ~6.7GB arrow table and appended it to a new table.

In terms of performances, I got similar results (writing to S3) on these 2 type of EC2 machines:

• c5ad.8xlarge 32 core, 64 ram, 10gbps nic -> wrote 1 parquet file of 2GB in 31s
• c5ad.16xlarge 64 core, 128 ram, 20gbps nic -> wrote 1 parquet file of 1.6GB in 28s

By using htop I notice that the code was only using a thread during the append operation, which means that it's not parallelizing the write operation.

[Fokko]

@bigluck Thanks for raising this. This is on my list to look into!

Parallelization of this is always hard since it is hard to exactly know how big the Parquet file will be. Efficient encoding and compression can vary a lot. However, just using one core does not make any sense.

[kevinjqliu]

Copy/paste my comments from Slack.

PyIceberg currently uses ParquetWriter's write_table function to write data files (see write_file) which I believe only uses 1 core (need to verify this)

ParquetWriter also has the write_dataset function for writing multiple files which has a use_threads argument to toggle and write in parallel.

https://arrow.apache.org/cookbook/r/reading-and-writing-data---multiple-files.html#write-partitioned-data---parquet
Describes using write_dataset to write multiple files

[kevinjqliu]

There's not a lot mentions of using Arrow to write a single file using multiple threads. The only thing I found was use_threads() here

I don't think this is exposed to writing a single file (write_table). But it is for writing multiple files (write_dataset)

[kevinjqliu]

Looks like we have to use a higher level API to force parallelism, i.e. write_dataset to write multiple parquet files using multiple threads

[kevinjqliu]

Looks like write_file already gets an iterator of WriteTasks.

Maybe we can use the ParquetWriter.write_dataset implementation to parallelize the data file writes.

[kevinjqliu]

Support encoding a single parquet file using multiple threads

Looks like the bottleneck might be encoding instead of IO

[kevinjqliu]

Related to #208 and #353

[zeddit]

I wonder if pyiceberg write will keep the row order when reading out later.
Why I ask this is because some engine like spark won't keep this, even specifying the sortOrder field.

[Fokko]

There's not a lot mentions of using Arrow to write a single file using multiple threads. The only thing I found was use_threads() here

This is for reading, so each core reads a separate row group.

Maybe we can use the ParquetWriter.write_dataset implementation to parallelize the data file writes.

How would we control the size of a file (or the number of records is a second best).

I wonder if we generate a fair amount of data, like @bigluck is doing:

import multiprocessing
import os
import time
import uuid
from functools import partial
from typing import Any

import pyarrow as pa
from faker import Faker
from pyiceberg.catalog import load_catalog


def generate_fake_rows(batch_size, i) -> list[dict[str, Any]]:
    fake = Faker()
    return [
        {
            **fake.profile(),
            'password': fake.sha256(),
            'device_id': fake.uuid4(),
            'device_ip': fake.ipv4(),
            'device_user_agent': fake.user_agent(),
            'phone': fake.phone_number(),
            'salary': fake.random_int(15_000, 250_000),
            'job_description': fake.sentence(nb_words=15),
            'cc_expire': fake.credit_card_expire(),
            'cc_number': fake.credit_card_number(),
            'cc_security_code': fake.credit_card_security_code(),
        }
        for _ in range(batch_size)
    ]


def generate_fake_table(num_records: int, batch_size: int) -> pa.Table:
    num_batches = num_records // batch_size

    generate_fake_rows_partial = partial(generate_fake_rows, batch_size)

    with multiprocessing.Pool(processes=multiprocessing.cpu_count() - 1) as pool:
        results = pool.map(generate_fake_rows_partial, range(num_batches))
    return pa.Table.from_pylist([row for batch in results for row in batch])

How many Arrow buffers do we get, and if we can smartly join these buffers (we already have a bin-packing algorithm in the code).

We could use the write_batch operation of ParquetWriter to write certain batches to a file.

[kevinjqliu]

I took the above code and did some investigation.
Here's the notebook to see it in action https://colab.research.google.com/drive/12O4ARckCwJqP2U6L4WxREZbyPv_AmWC2?usp=sharing

I'll summarize.

I used the above functions to generate 1 million records and save them in a feather file.
The file on disk is 447 MB in size.
Reading the feather file back to Python, the pyarrow.Table is 685.46 MB in size, using nbytes

We can use the to_batches function to chop the table into RecordBatchs, using the default setting.
This turns the table into 16 batches with around 40MB each.

Given this, it seems plausible to chunk the table into batches and use the write_batch function. We can set the default chunk to size 516MB or some recommended setting.

Note, to_batches is a "zero-copy" method, so there shouldn't be performance impact

[kevinjqliu]

It seems like there's an upper bound to the size of the RecordBatch produced by to_batches. I tried setting max_chunksize from 16 MB to 256 MB. All the batches produced are around 45MB in size.

I guess is this what you mean by bin-packing. We can bin-pack these batches into 512 MB parquet files.

[kevinjqliu]

Also, @bigluck, while running the code to generate the data using faker, I opened htop and saw that it was using 6 CPUs. I'm using a M1 Mac

[bigluck]

Thanks @kevinjqliu
Last week, I didn't test the code on my MBP; I did all the tests directly on the EC2 instance.

BTW it seems to use all the cores on my M2 Max:

[Fokko]

Do we know if this is for data generation, or also when writing? In the end, it would be good to be able to split the data into multiple files. The MacBooks have huge IO, so it might be that the CPU is the bottleneck and not the IO