Deterministic `NaN`

[penzn]

@sunfishcode proposed to add deterministic NaN behavior if this proposal adds a 'strict' mode.

I am curious to discuss this a bit, regardless of whether or not it will become part of the strict mode. Mainly I am curious if it has to become part of the spec as engines can restrict NaN patterns without violating the spec today.

[sunfishcode]

Engines could indeed just restrict NaN bitpatterns on their own today, but they could potentially make different choices in how they do so. An implementation on x86 might consider canonicalizing to a NaN which has the sign bit set, for example. The purpose of putting something in the spec would be to pick a single pattern so that all implementations that wish to be consistent with each other can have something specific to implement.

[Maratyszcza]

There are, in fact, two options for NaN normalization:

• All bits are set to 1. This option is the most efficient for canonicalization in software.
• Sign bit set to 0, the top mantissa bit set to 1, other mantissa bits set to 0. This option is particularly efficient on ARM64, as ARM64 supports an option in Floating-Point Control Register to normalize NaN to this representation.

[sunfishcode]

My proposal is for the second of those: sign bit 0, quiet bit 1, and the rest of the payload zero. This is consistent witha subset of Wasm's existing concept of a canonical NaN. In contrast, all-ones is not.

And in addition to "default NaN" mode on ARM, this is the behavior of RISC-V too. And it's consistent with what some popular toolchains do; for example, __builtin_nan("") in C produces this NaN.

[penzn]

All bits set to 1 is not canonical, though it is easy to produce in practice. Canonical pattern with sign bit 1 is produced by shifting that value by a constant amount.

My proposal is for the second of those: sign bit 0, quiet bit 1, and the rest of the payload zero. This is consistent with Wasm's existing concept of a canonical NaN.

Both sign 0 and sign 1 are canonical. I think you have measured it, and would be curious to see how you were able to enforce this pattern on x86. Unlike the other canonical pattern, it would be impossible to produce this variant form binary 11...1 by a single shift left, would require one more operation to just clear the sign bit (edit had some mention of unsigned shifts here, got confused).

Engines could indeed just restrict NaN bitpatterns on their own today, but they could potentially make different choices in how they do so. [...] The purpose of putting something in the spec would be to pick a single pattern so that all implementations that wish to be consistent with each other can have something specific to implement.

The reason for having a single NaN is full deterministic execution, which would be necessary some use cases, like code migration or crypto environments. As you outline in the slides, this has a cost. For a user that does not support the use cases that motivate single NaN it would be a cost for something they are not doing. Currently some standalone runtime can chose to do it because they want to enable code migration, and a different runtime can be supporting neither of the motivating use cases and would like to get some performance back. We should be OK with the latter choice, unless we want to intentionally eliminate that option.

[sunfishcode]

I implemented it on x86 using a load from a constant pool. It has a cost, and I'm not proposing anyone pay it that doesn't want to.

[sunfishcode]

The component model also has a concept of a canonical NaN, which matches the one I'm proposing here.

[penzn]

I implemented it on x86 using a load from a constant pool. It has a cost, and I'm not proposing anyone pay it that doesn't want to.

How would anyone avoid paying the cost if that this gets standardized?

[sunfishcode]

The rest of the presentation is about proposing a new relaxed-math mode. Canonicalization would not be required in that mode.