Swap gzip with bzip2

[eric-mh]

Gzip occasionally has issues with the underlying zlib library. It's a bit difficult to reproduce, but errors look like:

...
  File "wordninja.py", line 80, in <module>
    DEFAULT_LANGUAGE_MODEL = LanguageModel(os.path.join(os.path.dirname(os.path.abspath(__file__)),'wordninja','wordninja_words.txt.gz'))
  File "wordninja.py", line 31, in __init__
    with gzip.open(word_file) as f:
  File "/usr/local/lib/python3.7/gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/usr/local/lib/python3.7/gzip.py", line 175, in __init__
    raw = _GzipReader(fileobj)
  File "/usr/local/lib/python3.7/gzip.py", line 380, in __init__
    wbits=-zlib.MAX_WBITS)
  File "/usr/local/lib/python3.7/_compression.py", line 53, in __init__
    self._decompressor = self._decomp_factory(**self._decomp_args)
ValueError: Invalid initialization option

Swapping out the gzip file with bzip should be a harmless option to prevent those

bzip generation steps:

gunzip wordninja_words.txt.gz
bzip2 wordninja_words.txt

gunzip test_lang.txt.gz
bzip2 test_lang.txt

tests:

python3 test.py   
......
----------------------------------------------------------------------
Ran 6 tests in 0.003s

OK

[keredson]

That's concerning. Did this happen to you? Is there an example somewhere?

This is complicated a bit because people can supply their own model files. We'd need to not break compatibility.

[eric-mh]

Yep, I can't figure out what's behind it though. It occasionally happens on some CentOS 8 hosts where everything else looks fine.

It boils down to this call sometimes throwing a value error and sometimes not:

zlib.decompressobj(-15)

That's a good call with compatibility, this change should really come with gzip support too.

[keredson]

since we have the filename, why not just check the extension?

[eric-mh]

Just in case there's any discrepancy between the file extension and its type. Although that really depends on what's the expected behavior if it's fed bad inputs.

Should it:

1. Accept the extension and just throw out errors if it fails to process?
2. Ignore the extension and just process it best it can?
3. Ignore the extension, but pop out a warning if the extension doesn't match?