GCC, LLVM, related tooling and formats for the R5900

[frno7]

We talk about R5900 assemblers, compilers, related tooling, file formats such as ELF, etc.

GCC has supported the R5900 for decades, and now LLVM is about to join. Review #1 (comment) and llvm/llvm-project#176666 for some starting points!

[rickgaiser]

This is a nice place to have a conversation about designs, let me kick off with a few design decision I made so we can discuss (and I can change) them before it get to the point of a PR.

MMI datatypes

MMI uses the 128bit GRP registers. On GCC these can be used as TI mode. But on LLVM this gives many issues, becouse it then also needs all the TI mode operations implemented. Makes sense to me, r5900 does not really have TI mode, it only has 128bit vector registers.

• TImode: not supported
• V2DI: not supported
• V4SI: supported
• V8HI: supported
• V16QI: supported

Existing code (like ps2sdk) will need to change from using TI mode, to using one of the vector modes. Since it';s only used for things like pointers and data copying this will be easy to change (I already tested in a branch).

MMI ABI

The calling convention is the same, but extended from 64bit to 128bit. Saving restoring on the stack is also extended to 128bit. I have many questions about this. Like:

• Is it still n32 or should it be a new ABI?
• Most code only uses the low 32bit part of the registers, but the stack needs to save/restore 128bit, is that ok or too inefficient?

VU0 macro mode

I added the vu0 macro mode instructions optionally using the -mvu0 compile option. This way the vu0 can be used in both macro and micro mode. Supported modes:

• V4SF: supported

VU0 ABI

I extended the n32 ABI and took the same approach as the FPU registers, which are also 32 registers. Only VF0 is different between FPU and VU0 (for vu0 it's 0,0,0,1 fixed value). Again the question:

• Is it still n32 or should it be a new ABI?

ACC registers

This is where the ps2 performance starts to shine. I was not able to properly implement these in GCC, somehow each approach I took I got stuck. Leaving only inline assembly for GCC to use the ACC register.

PR plan for LLVM

• Phase1: basic CPU support (no FPU, no new instructions)
• Phase2: basic hard FPU support (no ACC)
• Phase3: Instruction scheduling

With phase 1 and 2 LLVM support will be sort of at the same level as GCC upstream. And with phase 3 it will be sort of at the same level as the custom GCC branch @fjtrujy and I have been maintaining on ps2dev for the past years.

References full feature LLVM:
https://github.com/rickgaiser/llvm-project/blob/EmotionEngine
https://github.com/rickgaiser/llvm-project/blob/EmotionEngine/PS2_EE_README.md

References full feature GCC:
https://gitlab.com/ps2max/toolchain/gcc/-/tree/vu0_autovec
https://gitlab.com/ps2max/toolchain/gcc/-/blob/vu0_autovec/PS2_EE_README.md

Reference ps2dev GCC:
https://github.com/ps2dev/gcc/commits/ee-v15.2.0/

[rickgaiser]

FAOD I think the MMI state ought to be a part of the context switched unconditionally; after all it requires the same number of machine instructions to manipulate and only 256 extra bytes, which shouldn't be a big deal with the hardware concerned I suppose unless you run thousands of tasks. And then the VPU0 context could be lazily switched with the aid of CP0.Status.CU2, similarly to how the FPU is currently handled (it's not clear to me how VPU1 is used).

I agree.

VPU0 can run in:

• macro mode as COP2
• micro mode as standalone vector processor with it's own instruction set (very similar to the cop2 instructions)

VPU1 can only run in micro mode.

VPU0 and VPU1 in micro mode are very similar to what modern GPU's call a programmable vertex processor. Micro mode is a backend of it's own, not MIPS.

[frno7]

@maciej-w-rozycki, sounds reasonable!

I attempted a ‘pure’ SQ/LQ 128-bit save/restore for MMI registers a few years ago, but it turned out that even with some struct and union trickery to fit and separate bits 31..0 from 127..32 it wasn’t that simple, because a couple of hundred places in the kernel that needed access to bits 31..0 (in 32-bit mode; in 64-bit mode it’d be bits 63..0) didn’t use register access macros (or any kind of abstraction) for it. So it became a fairly large patch with hundreds of tedious changes to deal with all places needing register access.

The other alternative I came up with was to leave the current register array as-is, and (additionally) save/restore all 128 bit registers in a new separate array (that isn’t accessed elsewhere in the kernel). That’d be functional and less intrusive patch-wise, but need somewhat more memory and be aesthetically unpleasing, of course. Other ideas? :-)

[frno7]

Maybe having Coccinelle fix all register accesses would easily meet the case, if one went far enough afield with it, huh?

[maciej-w-rozycki]

I attempted a ‘pure’ SQ/LQ 128-bit save/restore for MMI registers a few years ago, but it turned out that even with some struct and union trickery to fit and separate bits 31..0 from 127..32 it wasn’t that simple, because a couple of hundred places in the kernel that needed access to bits 31..0 (in 32-bit mode; in 64-bit mode it’d be bits 63..0) didn’t use register access macros (or any kind of abstraction) for it. So it became a fairly large patch with hundreds of tedious changes to deal with all places needing register access.

It should be fairly straightforward. The kernel needs to know of the MMI registers only for context switching, and for its own use it will only ever access the low-part 64-bit (or 32-bit) GPRs. And the EE is solely little-endian, so PT_Rxx macros will point at the right quadword-aligned data piece regardless of the width accessed (yay!).

So the bulk of the changes would be replacing some of the LONG_L/LONG_S macros in arch/mips/include/asm/stackframe.h (and possibly arch/mips/include/asm/stacktrace.h, which seems to be broken for not using the PT_Rxx macros) with abstraction macros expanding to either LQ/SQ (QUAD_L/QUAD_S could be defined too, for consistency) or LONG_L/LONG_S depending on whether R5900 support has been chosen in the kernel configuration or not.

Then accesses to ->regs from C code could be easily dealt with e.g. with the equivalent of:

typedef unsigned int __attribute__((mode(TI))) uint128_t;
typedef unsigned int __attribute__((mode(DI))) uint64_t;

struct pt_regs {
	union {
		uint128_t reg_mmi;
		uint64_t  reg_int;
	} rs[32];
	/* ... */
};
#define gprs(i) rs[i].reg_int

uint64_t get_reg(struct pt_regs *regs, unsigned int i)
{
	return regs->gprs(i);
}

-- avoiding the common (and painful!) overloading of regs between a function's parameter of the struct pt_regs type and the struct member (and with get_reg for illustration only: see what it assembles to). For non-R5900 configurations the reg_mmi member would be removed, keeping the effective layout of struct pt_regs in its current shape.

Yes, this would be a large mechanical change, preferably with an initial change making no functional difference that would add:

#define gprs(i) regs[i]

and then replace all ->regs[...] occurences with ->gprs(...). But it could be easily made with sed or suchlike, no fancy tools needed, and would be easy to verify, by making sure no change has been made to the machine code produced. Then the final change would add actual MMI context support and only redefine gprs accordingly, again easy to verify that no change has been made to the machine code produced in the non-R5900 configuration.

[frno7]

It should be fairly straightforward.

Thanks for the encouraging recipe! Noted in #5 (comment) in issue #5, and #14 (comment) in issue #14, for future reference.

[rickgaiser]

Phase 1 has been merged into LLVM! Yay!

So I'm moving on to phase2: basic hard FPU support (no ACC). I have the commit ready here:
llvm/llvm-project@6f3a693

However before making the PR I have a few questions that would be nice to discuss now:

R5900 specific instructions

The commit currently includes r5900 specific instructions: max.s, min.s and rsqrt.s. When emitting these, the resulting code will no longer be MIPS III compatible. Should these extra instructions be possible to enable/disable using an option?

Perhaps better to excude these extra instruction from the initial PR? As they are not absolutely needed but can lead to discussion.

No double, only single precision

r5900 seems to be the first to use only single precision.

IEEE 754 compatibility

How to handle the not supported features:

[maciej-w-rozycki]

Of course buggy user source code could still trigger misalignment in compilation, but that you can't really do anything about unless you assume the lack of alignment for all memory accesses, at a performance penalty. I do hope you don't mean this case though.

The bug report has a simple and correct C code example triggering the problem. As the maintainers explain in mentioned GCC bug 98627, the problem is that linux in m68k-linux-gcc implies m68020+, despite explicitly given the -march=68000 argument. Some kind of internal GCC target conflict. No warnings or errors are produced by GCC either; the generated executable simply crashes on execution.

And that's presumably on a plain M68000 CPU running no-MMU Linux, correct? Because regular Linux IIUC does require the M68020 along with its MMU, doesn't it? [Pardon my ignorance, I'm not that familiar with the m68k architecture and all the plenty of m68k chips I have are onboard my FDDI networking hardware.] So if you do need to produce plain M68000 user code for no-MMU Linux, then it's certainly a valid argument in favour to getting the bug fixed and I'd recommend to raise it while reopening the bug.

Conversely, for bare metal m68k-elf is indeed the correct configuration and the only bare-metal project *-linux compilers support, by a special exception so as not to be mean, especially in native environments, is the Linux kernel, so it's up to the policy set by the respective maintainers what they want to support beyond, if anything.

I'm assuming this is a linux ELF loader feature? GCC and LLVM set a flag in the ELF file so the loader can call prctl? AFL_EXT_5900 is not sufficient for this? Please let me know if I'm missing something here. I've mostly been running baremetal so I'm not too experienced with the ABI and ELF flags.

However the use of the FPU vs IEEE 754 emulation could well be subject to ELF file annotation and prctl(2).

@maciej-w-rozycki, what compiler option and ELF annotation would you recommend for enabling FPU hardware in Linux?

FAOD I'd consider AFL_EXT_5900 (and in the absence of MIPS_ABIFLAGS segment, EF_MIPS_MACH_5900) the proper way. A prctl(2) call might be useful for JIT environments; a JIT compiler could have been built for MIPS I and dynamically switch for its output to R5900 hard float where available.

[frno7]

And that's presumably on a plain M68000 CPU running no-MMU Linux, correct?

Plain 68000, yes, but not running Linux. But even if it was some flavour of Linux, it wouldn’t work anyway. The 68000 does provide address error exceptions (to handle the unaligned access generated by the compiler), which can be trapped, but it does not save enough processor state to resume the faulted instruction once the operating system has handled the exception.

The Apollo/Domain DN416 Unix workstation, for example, had dual 68000 processors and phased clocks to implement virtual memory—which a single 68000 is not capable of—by stopping one processor when there is a page fault and having the other processor handle the fault, then release the primary processor when the page fault was handled. Remember those machines? :-)

Because regular Linux IIUC does require the M68020 along with its MMU, doesn't it?

Yes, indeed.

Conversely, for bare metal m68k-elf is indeed the correct configuration and the only bare-metal project *-linux compilers support, by a special exception so as not to be mean, especially in native environments, is the Linux kernel, so it's up to the policy set by the respective maintainers what they want to support beyond, if anything.

Yeah. But an unassuming user with a m68k-linux-* compiler at hand may of course attempt -march=68000, which (mostly) kind-of works. So the argument is that the -march=68000 option doesn’t do what most users think it does. I struggle to come up with a reasonable use-case for its behaviour, really. But the compiler proceeds without warnings or errors.

So my conclusion, as mentioned, is that GCC options (such as -march in general) are difficult to understand and reason about. However, MIPS targets do seem to behave somewhat coherently, thankfully. Still, it’s a complex subject. :-)

[frno7]

@rickgaiser, is GCC (currently) restricting itself to the same subset of FPU instructions as basic LLVM FPU support? Are there any notable differences in FPU handling between GCC and LLVM? (Handling of comparison conditions, and rounding, for instance?)

@maciej-w-rozycki, is the Linux kernel expected to software emulate for IEEE 754 compliancy when compilers emit R5900-specific FPU instructions too? It won’t be able to carry on and software emulate autovectorised floating-point MMIs, since they are untrappable?

[rickgaiser]

@rickgaiser, is GCC (currently) restricting itself to the same subset of FPU instructions as basic LLVM FPU support? Are there any notable differences in FPU handling between GCC and LLVM? (Handling of comparison conditions, and rounding, for instance?)

In binutils:
https://github.com/ps2dev/binutils-gdb/blob/616e51fa6ed9d1e8f4bda7d0087fac30c5398aa9/opcodes/mips-opc.c#L801-L923
EE is excluded for many instructions, resulting in the same subset with 1 minor difference. Binutils maps:

• c.lt.s -> c.olt.s
• c.le.s -> c.ole.s

GCC does not seem to have special restrictions for unsupported instructions, instead binutils changes them.
LLVM doesn't do this and emits the supported instructions directly.

[frno7]

Thanks! Mentioning this comparison with GCC may assist reviewers of llvm/llvm-project#178942 and improve the likelihood of a merge.