Old ZNG input auto-detected as CSV #6695
Description
Test data is available at s3://brim-sampledata/fdns/fdns-a.zng.
$ super -version && super -dynamic -c "head 1" fdns-a.zng
Version: v0.2.0-12-gd68257f27
fdns-a.zng: record on line 3: wrong number of fields
Details
Repro is with super commit d68257f. This issue surfaced when an Autoperf run attempted and failed to load this old ZNG test data into a persistent super db.
It was my expectation that the input would have been recognized as an old BSUP version and rejected with this kind of error message that I can see if I explicitly specify BSUP as the input type.
$ super -version && super -dynamic -i bsup -c "head 1" fdns-a.zng
Version: v0.2.0-12-gd68257f27
fdns-a.zng: BSUP version mismatch: expected 1, found 0
However, that's not the error I saw, and I didn't get the more typical verbose dump of how it tried and failed at each of the input formats supported for auto-detect. So I manually made my way through each one and found the matching error message when I tried CSV.
$ super -version && super -dynamic -i csv -c "head 1" fdns-a.zng
Version: v0.2.0-12-gd68257f27
fdns-a.zng: record on line 3: wrong number of fields
The change in behavior came specifically at commit a5bdf1e when #6669 merged, as it worked ok at the prior commit.
$ super -version && super -dynamic -c "head 1" fdns-a.zng
Version: v0.1.0-53-g6e9da9bef
{name:"0.202.98.34.bc.googleusercontent.com",timestamp:2020-11-27T00:28:12Z,type:"a",value:34.98.202.0}
In conclusion, I respect there will probably always be corner cases that trip up the auto-detect, and I also recognize we probably can't invest excessively on graceful failures for all our prior format variations. However, the way a byte sequence here apparently caused it to "lock in" to CSV early might be worth a closer look.