Atrocious memory footprint and performance when reading many small files

[pronebird]

Describe the bug
Iterating over 1.5GB ZIP archive (with 947319 small files) causes the app to consume almost 512MB of memory on my machine. Listing files in ZIP with unzip -l on the other side takes 1.6M.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://www.sec.gov/search-filings/edgar-application-programming-interfaces
2. Download submissions.zip under "Bulk Data"
3. Run program:

let reader = std::fs::File::open("submissions.zip").unwrap();
let mut zip = ZipArchive::new(reader).unwrap()

let num_files = zip.len();
for i in 0..num_files {
   let file = zip.by_index(i).unwrap();
   println!("Read file: {}", file.name());
}

Expected behavior
Expected that memory footprint is somewhat small.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: macOS
• Version 15.7.2

Additional context
Add any other context about the problem here.

[Its-Just-Nans]

Hello

Thanks for the issue. Yeah it seems the crate is using a lot of memory (storing all files)

I'm curious, how do you check the memory usage of your program?

Btw I think you should check #487 (and why not trying it)

[remdalm]

Hi @Pr0methean. Thanks for maintaining this project. Much appreciated!
I wanted to ask, at a high level, whether this issue is something you might be open to addressing, or whether it’s likely out of scope for now.

@velocitysytems

[Pr0methean]

I'd like to get to it at some point, although open non-draft PRs are a higher priority.

[Its-Just-Nans]

I wanted to ask, at a high level, whether this issue is something you might be open to addressing, or whether it’s likely out of scope for now.

Hi

I think on my side #487 is still work in progress. I was focusing more on open issues recently but I hope to make the MR for fast, and low memory zip ready soon

[velocitysystems]

Hi @Pr0methean — we’re also interested in this issue and would be happy to help. Is there anything specific we can contribute? Do you already have a solution in mind, or would you be open to performance-related improvements from contributors?