I generally like their ideas very much. As a former user of a multi-gpu setup I particularly know that fps don't tell the whole story. In fact, they don't even tell half the story sometimes. I remember playing Crysis at 70-80fps (vsync=off) which stuttered big time and felt worse than 40fps without micro stuttering.
As per the metrics, they should definiitely normalize
everything, that would eliminate the problem of comparing runs of different lengths (or different amounts of frame times). Instead of counting the number of frame times larger than a given threshold, they would then report the proportion which is larger than 50ms. I find this transformation quite standard and straightforward, and it's a clear improvement IMO.
@BD vs. SB vs. NH
The results are pretty much as expected (sadly for AMD, one should note).
Here are two interesting things in the results:
1. In Skyrim, going from 1024 to 1280 keeps the fps constant, the same for 1680 to 1920.
But there's a difference between the upper resolutions and the lower ones:
(black arrow=no difference, red arrow=difference)
2. In Starcraft, on the other hand, there are neither "horizontal" nor "vertical" changes, or at least the upper resolutions are more similar to the lower ones than in our example before (Skyrim):
I suspect the aspect ratio makes the difference, since the upper resolutions are 4:3 and 5:4 whilst the lower ones are 16:10 and 16:9 respectively. The wide screen aspects require more rendering in the horizontal than in the vertical, compared to the 4:3 (and 5:4) ratio.
The explanation, why there is a difference in Skyrim and (almost) none in Starcraft is then, that
- there's not so much going on in the vertical in Skyrim: floor texture and sky texture, from which especially the latter is very simple to compute for the CPU. So, when you add more horizontal pixels, I would expect a much larger amount of CPU computations necessary than when adding vertical ones (just paint some more heaven and floor, to oversimplify).
- in Starcraft, since it's a top-down view, adding horizontal pixels and adding vertical ones should make (almost) no difference. It's far less asymmetric than Skyrim, since more terrain, buildings and units will be computed, regardless in what direction the image is expanded.