Advanced data structures for very large datasets

[hdbeukel]

Let's discuss our ideas here for advanced data structures that allow to deal with very large datasets.

One option that came into my mind for 0/1/2 data would be to use two bitvectors (BitSet in Java) per individual. That could yield approximately a 4-fold reduction in memory usage as compared to using a full byte per value.

[xiekunwhy]

Can corehunter deal with large dataset(like WGS snp, 1000samples X 2M snps) efficiently now?

[hdbeukel]

I am not sure. A couple of 100k SNPs should work fine. But 1000 samples x 2M SNPs already requires at least 16 GB of RAM just to load the 0/1/2 allele score matrix in R (which uses 8 bytes per numeric value) and at least about 2 GB more in the Java Core Hunter library (which uses only 1 byte per 0/1/2 value). If you have access to a high-memory machine (I currently do not so I can't test) you might give it a try. We would be happy if you could share your findings here.

In any case, you will need to increase the default amount of memory assigned to the JVM. Instructions are provided on the website.

At the moment, the data structures used in Core Hunter are still simple implementations. We were able to largely reduce memory footprint in the latest version, mainly by being clever with Java data types, and by not precomputing certain values such as frequencies. More advanced data structures for very large datasets have not yet been implemented (nor are there any plans to add these in the near future). We are considering however to provide an option to load the data in Java only, instead of both in Java and in R (see corehunter/corehunter3-r#21). This would also make a big difference, especially for 0/1/2 marker data, which is stored inefficiently in R.

If you can't load the 0/1/2 matrix directly (format "biparental"), you could consider precomputing a distance matrix, which would reduce your 2 billion 0/1/2 data points to merely 1M double values, which fit in just a few MB. However, you can then only use the distance-based measures to sample your core, not allelic richness. If this limitation is no problem for you, precomputing a distance matrix once is likely the best solution.