Use simdutf8 for SIMD accelerated utf8 validation?

[jqnatividad]

Hi @BurntSushi ,
Was thinking of submitting a PR to use the simdutf8 crate for SIMD accelerated utf8 validation.

https://github.com/rusticstuff/simdutf8

The benchmarks look pretty good:
https://crates.io/crates/simdutf8

And its used by a lot of high-profile crates:
https://crates.io/crates/simdutf8/reverse_dependencies

I successfully used it in a rust-csv fork I was using earlier, but I was hoping to just keep using the release version of the csv crate.

Is this something that you would consider merging?

[BurntSushi]

In theory, I'd consider it. But in practice, probably not for a couple reasons:

1. I would need to see benchmarks justifying its usage in the context of this crate. This means that the benchmarks you linked to are pretty much meaningless here. They suggest there might be a speed improvement, but maybe not.
2. I would need to understand the use case for doing this instead of just using ByteRecord. The cheapest way to do UTF-8 validation is to avoid doing it at all.
3. My review bandwidth is pretty limited. For a dependency that uses a lot of unsafe, I would need to do a pretty thorough review of the project before I was willing to depend on it. As it stands, I can already tell that their dependency philosophy is probably not something I would agree with.

[jqnatividad]

Thanks for the quick reply!

I'll go ahead and draft a PR for your consideration with the reasons you listed in mind:

1. After drafting the PR, I'll run this crate's benchmarks to see if there's any appreciable performance improvement.
2. There is really no specific use case in mind. My primary motivation is to increase the speed of StringRecord which is more ergonomic than ByteRecord.
3. Though the simd_utf8 crate has fuzz testing, with the wide utilization of this crate - I understand and appreciate your dependency philosophy. At least the findings from point 1 should give you the metrics to see if its worth making time to review the PR.

[BurntSushi]

Wide utilization is a valuable signal. But it's not enough on its own for me personally. Especially for the csv crate, which itself enjoys wide use. csv depending on simdutf8 would further improve its reputability, so this is a step that should be taken very carefully. It has a lot of unsafe in it (by necessity) and it's easy to get that kind of thing wrong. People get SIMD stuff wrong all the time. Either I need to audit it before using it, or someone I trust needs to have audited it.

[jqnatividad]

Got it! And I should have been more clear - I was referring to the csv crate when I wrote about "wide utilization" above. :)

[BurntSushi]

Ah gotya. Makes sense.