Upscaling to 1440p @ 165hz is buggy and runs slow.

[roninpawn]

Noahkra reported tests failing on a 2k screen @ 165hz running at an average of 29fps:

• Level Select not being detected (no run start event)
• Level complete dropping matches during screen-shake (false negatives)
• On-event text logs are often reporting 29fps, spiking, infrequently, as high as 144fps. cdn.discordapp.com/attachments/222120862503927818/678327524652089354/unknown.png

[THE PROBLEMS]

• Image detection is subject to the scaling mechanics used to account for different resolutions within the original aspect ratio. Up-scaling comparison images appears to be an additional burden of fidelity.
• On a 60hz monitor (60fps), the gap between frame grab events is .016 seconds; That's more than a tenth of a second. But at 165hz, the gap between frame grabs is .006s. That's not much of a window in which to do the processing and comparison.
• Additionally, at a 1920x1080 resolution, the test area as currently written in the clustertruck.rp file has a width of 670 pixels, and a height of 517 pixels. 670x517px is equal to a total area containing 346,390‬ pixels. But once scaled to 2560x1440, that area is now 1.333x larger. The width and height are now 893x689 -- and that area now contains 615,277 pixels. The extra 1/3rd up-scaling of width and height, results in a near doubling of the pixel volume that undergoes both processing and comparison.

[KNOWN FACTORS]

• At 1080p, slowest-frame processing and analysis time on my Ryzen 7 averages .001009s, spiking infrequently to .00200s. At the scale as written, processing routinely takes less than a thousandth of a second.

• As the screenshot method used by MSS is blocking - ie. it waits until there's a new frame delivered to take its shot, preventing linear code operation until it has acquired a new frame - it is difficult to test how long the ACTUAL screen grabbing operation takes. On my Ryzen, in the written resolution, timing before the call and after the call simply returns a .016-.017s avg time, because only a thousandth of a second is lost to processing, and all remaining time gets recorded as time spent blocked, while MSS waits for the next frame to come in.

[THE POTENTIAL CAUSES/RESOLUTIONS]

• Failed Matches

Since upscaling seems to be causing fidelity issues here -- what if we just just DON'T? Downscaling could use retesting with the latest test-file, but it's seemed fine in the past. So what if - on an up-scale situation - we downscale the screenshot, instead? I was trying to avoid adding that kind of additional, CPU intensive process, to every frame collected, but if the alternative is upscaling, maybe it's the best for it.

Alternatively, it's slightly possible that changing the scaling method could provide better results on upscaling. But I don't have much hope for that.

• FPS Losses

At 165hz, the .006s window between frames becomes meaningful, where at 60hz, 0.016s, it is not. In just six thousandths of a second, MSS has to acquire the screenshot, then processing and analysis have to do their thing, and we have to land back at the next MSS call before the incoming frame.

...to support 165fps operation, we'd have to do that. Which is NOT part of splitRP's goal-line. Steady, reliable 60fps operation IS.

Still though, two frames at 165hz take .012s delivering around 83fps. So if processing fails to complete within that first frame window consistently, we'd still be delivering eighty-some frames per second of analysis. And three frames at 165hz take .018s, or 55fps. So if processing took two whole frames to complete, we'd see a nearly-sixty frame rate.

But Noah's 29fps average constitutes .034s between each complete cycle. Or ~5.7 frame windows missed at 165hz. That's HUGE. And we see that on Noah's rig, splitRP is capable of running between 133-144fps. Which leaves me scratching my head a fair bit here.

If it were MSS getting bogged down, I'd expect it to be consistent. MSS has proven super-stable at taking the same size screenshot at very steady frame rates, and that's why I chose it as the library to build this on. IF that still holds true at 165hz 1440p, then the only remaining suspect is the processing. The doubled pixel volume is, perhaps, slowing things down a lot more than I'd anticipate. And because different test packs can require more or less processing, there's inconsistency.

If that's the case, then resizing the input screenshot (if the resize isn't too much load on it's own), could solve all concerns by enabling testing to happen AT the written pattern's intended scale. Additionally, frame-limiting is an intended feature that's been moved up on the to-do list to help in resolving this problem.

All that said, I have trouble buying it. It's just a lot of deviation to accept: 29fps usually/sometimes 144fps? Explaining it by either processing times or screenshot delays taxes my imagination.