One thing to try is using 24-bit waves instead of floating point. 1. See if fft can run with 32-bit or 64-bit ints 2. Try shifting 24-bit audio up to 32-bit (or higher) to avoid rounding errors