DuckDB Metadata Index

[albert-du]

#73

Implements a DuckDB metadata index.

Worst case performance is better than the current SQLite index, but worse than if the SQLite index was better optimized.

poetry run python govscape/benchmarks/metadata_index_benchmark.py --documents 5000000 --queries 100
Index            Docs  Ingest(s)   Docs/s     Size  Scenario            Q/s    Lat(ms)
--------------------------------------------------------------------------------------
sqlite         5000000    16.6362   300550  886.5MB  no_filter        3220.3       0.31
                                                     domain_filter       2.9     344.46
                                                     date_filter      2451.8       0.41
                                                     all_filters         2.9     341.45
--------------------------------------------------------------------------------------
improved-sqlite 5000000     9.0737   551043  807.7MB  no_filter        9523.6       0.11
                                                      domain_filter   19925.2       0.05
                                                      date_filter     15548.3       0.06
                                                      all_filters     21146.7       0.05
--------------------------------------------------------------------------------------
duckdb         5000000     5.2445   953388  328.0MB  no_filter         108.0       9.26
                                                     domain_filter     172.2       5.81
                                                     date_filter       108.3       9.23
                                                     all_filters       188.5       5.30
--------------------------------------------------------------------------------------

I added a benchmark with the four possible query types:

• no_filter, searches for random pdf names
• domain_filter, pdf names and subdomain
• crawl_filter, pdf names and crawl ranges
• all_filters, pdf names, subdomains, and crawl ranges

I also cleaned up a duplicate .gitignore entry.

[kylebd99]

This is great. Cool to see that SQLite holds up when properly optimized. I removed the non-optimized SQLite version because it appears to be fully superseded.