Figure out the ABI for the ibm-aix target (vec-default vs vec-extabi)

[RalfJung]

Disclaimer: I know little about PowerPC and even less about AIX. So maybe I'm wrong about all this. Someone please confirm whether any of this makes sense. :)

AIX on Power64 has two possible ABIs: vec-default and vec-extabi. In LLVM, the EnableAIXExtendedAltivecABI field controls which ABI is actually used. In Rust, we also have the abi field on each target which controls cfg(target_abi), but usually (and in particular in this case) doesn't itself have any effect on the ABI -- it merely informs the programmer which ABI the target spec claims to implement, but as target spec author one still has to manually arrange for that ABI to actually be used (in a usually architecture-specific manner, because ABIs are a mess so nothing is consistent).

The powerpc64-ibm-aix target got added in #102293, with cfg(target_abi) indicating "vec-extabi", but without setting EnableAIXExtendedAltivecABI, leaving it at its default value of false. In other words, the target claims via cfg(target_abi) to use vec-extabi, but actually it used vec-default. That seems inconsistent and we should fix it.

In #153784, the proposal was to fix this by actually setting EnableAIXExtendedAltivecABI in LLVM. However, @daltenty raised the concern that this might not actually be what the target maintainers want. So in #153830 I proposed to instead fix the problem by changing cfg(target_abi) to "vec-default", matching the (implied default) value of EnableAIXExtendedAltivecABI. But that clashes with logic in our inline assembly handling which rejects using the registers v20-v31 (vs52-vs63) when cfg(target_abi) is "vec-default" (#131341, #146949).

I don't know what the correct fix here is, so I figured I'd just open an issue, dump everything I know, ping all the right people, and let you all figure it out. :) In particular, it is confusing to me that the inline assembly logic looks at the ABI -- whether the registers are available and whether they get used by the ABI are usually orthogonal questions, and it would seem to me that inline asm should care about the former, not the latter. But I also don't know how availability of these registers is modeled on AIX/PowerPC; on other architectures, this is typically done via target features.

Cc @Gelbpunkt @daltenty @gilamn5tr @amy-kwan @ecnelises @pmur @taiki-e @Amanieu @workingjubilee

[taiki-e]

In particular, it is confusing to me that the inline assembly logic looks at the ABI -- whether the registers are available and whether they get used by the ABI are usually orthogonal questions, and it would seem to me that inline asm should care about the former, not the latter.

AIX documentation (AIX vector programming) says:

VR20:31	Reserved (default mode) Nonvolatile (extended ABI mode)	When the default Vector enabled mode is used, these registers are reserved and must not be used. In the extended ABI vector enabled mode, these registers are nonvolatile and their values are preserved across function calls

Inline assembly generally must not access such reserved registers.

---

But I also don't know how availability of these registers is modeled on AIX/PowerPC; on other architectures, this is typically done via target features.

Yeah, in other arches, we indeed refer to other things¹ even if that constraint actually stems from the ABI.

The reason the current AIX vector ABI handling in inline assembly refers to abi field is that when I asked the target maintainer at the time, they pointed me to that field (#131341 (comment)). (At the time, I didn’t know that the field was only used to set the cfg.)

That said, this doesn't appear to be managed as a feature in LLVM, so I'm not sure what other options might be available.

Footnotes

1. target_arch (aarch64 vs arm64ec, etc.), target_os (aarch64 x18, etc.), target_feature (sparc vs sparcv8+, rv{32,64}i vs rv{32,64}e, avr vs avrtiny, etc.) There is another powerpc code that refers to abi field, but it should be changed to refer to the target feature. ↩

[daltenty]

Thanks for looking into this, we've been meditating on this too from the maintainer side. I’m going on a bit of a long walk here, but I think there are some concerns when mixing the extvec and default ABIs that we should think carefully about for the target.
(I don't know a lot about how Rust wants to treats possibly unsafe things like variant ABIs like we have here on AIX, so any suggestions or clarifications are very welcome)

In general for the AIX ABI we ensure the registers remain reserved in the default ABI and LLVM refuses to use them unless the extend vector ABI is enabled.

Therefore:

• Calls from NonExt to Ext: are safe as the callee knows to save and restore the extended register set if used.
• Calls for Ext to Non-Ext: are safe as the callee cannot use the reserved registers

Unfortunately however certain system/library interfaces are necessarily sensitive to whether the extABI is in use at the whole program level. So the problematic/safety question really boils down to ensuring that the right interfaces are used (for example to save the execution context in setjmp or pthread).

Thus a long walk getting me back to what the doc says, mainly if any module in the program uses the wrong set of interfaces, the result may be unsafe.

On the C side, controlling which version of these interfaces are used is usually done at a TU level through the macros. Unfortunately all C compilers on the platform unfortunately do not do the extABI safe thing by default (i.e. set __AIXEXTABI). So that means you run the risk of calling into such from a Rust program if you link to C, even if the whole Rust side is correctly setup for the extabi. Further more, even in the Rust side in libc crate I see we link to the non-extended ABI (i.e. unsafe) signatures, so things are definitely not currently setup properly even on the Rust side to use the extended vector ABI.

To that end I'd conclude the extend vector ABI should be disabled for now for the AIX target in Rust (maybe we ought just rip the mode set entirely for simplicity if it's messing with the other infrastructure).

Before brining it back we need to think carefully both about a number of concerns raise here and in the attached issues. Namely:

a) how we model the ABI extended vector ABI in LLVM
b) migrating the rust ecosystem to the extVec safe OS interfaces
c) how to mitigate the issues caused by linking in existing non-extVec safe C code

[RalfJung]

@taiki-e

Inline assembly generally must not access such reserved registers.

Surely that just means "must leave them in the state that they found it in"? Using it as scratch space and restoring the original value should be fine?

@daltenty

(I don't know a lot about how Rust wants to treats possibly unsafe things like variant ABIs like we have here on AIX, so any suggestions or clarifications are very welcome)

Many targets have multiple ABIs. Some are pretty good at dealing with it (RISCV), some are messy. Looks like AIX is on the messy side. Generally the goal in Rust is to make it so that linking code built for the same target triple is sound, i.e., a given function signature and ABI string (like "C" for extern "C" fn) has to uniquely identify the exact convention used for this call. So for example, on RISCV there's two conventions for how to pass f32/f64 values: it can use float registers or integer registers. Those are mutually incompatible ABIs, so they must use different target triples to avoid causing UB when the two sides of a call don't use the same register for the same argument. We therefore have a range of RISCV target triples to represent this choice. For SIMD types things get a bit more complicated, especially with the "C" ABI: on x86-64, extern "C" fn(x: i32x8) -> i32x8 is meant to use AVX registers to pass the argument and return value. Those are only optionally available, so we have a special check where when you write or call such a function without the AVX target feature, we emit an error as without the AVX registers available we cannot perform this call. In contrast, for extern "Rust" fn(x: i32x8) -> i32x8, we pass argument and return value in memory to have a consistent ABI that's always the same for the target, no matter which target features are enabled.

What is an example of a function signature that would work differently between the two AIX ABIs? I assume it involves passing SIMD types as arguments/return value?

To that end I'd conclude the extend vector ABI should be disabled for now for the AIX target in Rust

Okay, could you file a PR that changes the ABI field then? This is a breaking change for any inline asm blocks on the target that use the extended registers, but from what you say those registers shouldn't be used anyway.