Tuesday, October 9th 2018

Intel's 9th Gen Core Gaming Benchmarks Flawed and Misleading

At its 9th Generation Core processor launch extravaganza earlier this week, Intel posted benchmark numbers to show just how superior its processors are to AMD 2nd generation Ryzen "Pinnacle Ridge." PC enthusiasts worth their salt were quick to point out that Intel's numbers are both flawed and misleading as they misrepresent both test setups - by optimizing Intel processors beyond their out-of-the-box performance, and by running AMD processors with sub-optimal settings.

Intel paid Principled Technologies, a third-party performance testing agency, to obtain performance numbers comparing the Core i9-9900K with the Ryzen 7 2700X across a spectrum of gaming benchmarks, instead of testing the two chips internally, and posting their test setup data in end-notes, as if to add a layer of credibility/deniability to their charade. The agency posted its numbers that were almost simultaneously re-posted PCGamesN, gleaming the headline "Up to 50% Faster than Ryzen at Gaming." You could fertilize the Sahara with this data.
Right off the bat, we see Principled Technologies use a sub-optimal memory configuration for the Ryzen 7 2700X machine, saddling it with a dual-rank memory with all four memory slots populated, and running at stock memory speeds with the motherboard BIOS determining "stable" memory timings. AMD processors compensate for dual-rank / 4-module setups by either restricting memory clocks or loosening up memory timings in the interest of stability. Principled Technologies incompetently set the Ryzen setup's memory clocks to 2933 MHz, leaving the motherboard BIOS to find extremely loose memory timings to stabilize the memory clock.

In stark contrast to this, for the Core i9-9900K machine, the testers simply flicked the XMP profile of the Corsair Vengeance RGB DDR4-3000 memory kit, which ended up running at not just higher clocks, but also tighter timings (which have been tested by Corsair on an Intel platform to obtain the XMP certificate). They reinforced the memory by adjusting the frequency manually. This gives the Intel platform a significant performance advantage against AMD. Ryzen processors are more memory-sensitive than Intel, as DRAM clocks are synchronized with other clock domains such as the InfinityFabric clock, which determines the data-rate of communication between the two Zen Compute Complex (CCX) components on the 8-core "Pinnacle Ridge" die.

As if that wasn't bad enough, the white paper reveals that some of the games were tested on the Ryzen machine with the "game mode" enabled via Ryzen Master. What this does is localise a game to just one of the two CCX units, essentially turning the 8-core chip to quad-core. The game mode is known to have a negative performance impact on games that can use more than 4 cores, or which are memory bandwidth intensive. This is truly below the belt from Intel.

The next part of its deception was testing both setups at 1080p on "Ashes of the Singularity" CPU benchmark with medium settings, to obtain extremely suspicious performance numbers. When HardwareUnboxed used similar settings to compare their Core i7-8700K with the Ryzen 7 2700X (using sane memory settings for both setups), the performance numbers obtained were very different, and don't bode well for the credibility of their i9-9900K numbers. Without the unfair advantage to the i9-9900K, the Ryzen 7 2700X yields up to 18% higher frame-rates than what Intel's numbers suggest. The story repeats (albeit to a smaller degree), with most other benchmarks posted by Intel. "Assassin's Creed Origins" is another benchmark where Principled Technologies numbers paint the Intel 8700K at 36% faster than the 2700X, while in reality, the 8700K is more like 8% faster.

Normally, performance numbers released by hardware manufacturers at launch are disregarded by consumers as hardware launches are almost always simultaneously followed by independent reviewers being allowed to post their benchmark numbers. Off late, however, there is a worrying trend of hardware manufacturers launching their products with reviewer NDAs expiring weeks later, letting them solicit pre-orders on the basis of questionable performance data. In this case, Intel's gilded numbers release almost 2 weeks before the review NDA, and the Core i9-9900K is up for pre-order, in some places even at $540.

We strongly recommend you to wait until you read performance reviews from multiple tech publications before basing your purchase decisions. It's a foregone conclusion that the i9-9900K will be faster than the 2700X, as the i7-8700K already trades blows with it despite having two fewer cores. However, the percentage-difference in performance, and the cost-performance numbers put out by Intel for the upcoming chip, are extremely questionable at this point.

Update 19:55 UTC:
Intel provided following statement to GamersNexus regarding the testing
We are deeply appreciative of the work of the reviewer community and expect that over the coming weeks additional testing will continue to show that the 9th Gen Intel Core i9-9900K is the world's best gaming processor. PT conducted this initial testing using systems running in spec, configured to show CPU performance and has published the configurations used. The data is consistent with what we have seen in our labs, and we look forward to seeing the results from additional third party testing in the coming weeks.
Sources: Intel Benchmark Results & Methodology, HardwareUnboxed (YouTube)
Add your own comment

76 Comments on Intel's 9th Gen Core Gaming Benchmarks Flawed and Misleading

#1
bug
Which is why I tend to disregard "leaked" benchmarks and benchmarks coming from other than a handful of websites.
Posted on Reply
#2
XiGMAKiD
Cherry picked facts used as marketing materials, nothing new here.
Posted on Reply
#3
kings
Nowadays it seems that people just want to rant about something. They are not doing well if they don't have something to rant about.

Of course Intel will show what works best for them, pick the best case scenario and hand-picked benchmarks/definitions to further enhance your product. Is not that what AMD, Nvidia or any other company does?

AMD not long ago paid a company to test the quality of graphics drivers between AMD and Nvidia, in an environment chosen by them, drivers chosen by them and even the cards themselves were chosen by them. And surprise surprise, the result was victorious for AMD, however, I did not see anyone outraged by it.

I don´t know why people still waste time on these manufacturers' results, full of juggling. Independent reviews will exist, as always!
Posted on Reply
#4
btarunr
Editor & Senior Moderator
"kings said:
I don´t know why people still waste time on these manufacturers' results, full of juggling. Independent reviews will exist, as always!
It becomes a problem when Intel posts these numbers 11 days before independent reviewers are allowed to post theirs, and solicits pre-orders. This is a big deal because Intel paid a third-party to obtain these numbers. If they'd only posted their "internal" numbers, nobody would have cared about them. They tried to pull off something bad, and got caught.
Posted on Reply
#5
XiGMAKiD
"btarunr said:
It becomes a problem when Intel posts these numbers 11 days before independent reviewers are allowed to post theirs, and solicits pre-orders.
This is the news worthy bits, not the trash marketing.
Posted on Reply
#6
kings
"btarunr said:
It becomes a problem when Intel posts these numbers 11 days before independent reviewers are allowed to post theirs, and solicits pre-orders.
Well, If people pre-order something, it's entirely their fault. People should never pre-order something without first knowing all the facts.
Posted on Reply
#7
Rahmat Sofyan
who still believing that AMD still a smart choice ?
Posted on Reply
#8
Camm
"Rahmat Sofyan said:
who still believing that AMD still a smart choice ?
It takes a special person to completely disregard the article they just read to continue being a partisan troll, but congratulations, do not pass go, and do not collect $200 in astro turf.
Posted on Reply
#9
iO
Funnily enough the benchmarks also show how awful the 9900X and 9980XE will be for gaming..
Posted on Reply
#10
bug
"iO said:
Funnily enough the benchmarks also show how awful the 9900X and 9980XE will be for gaming..
Do we really need benchmarks to tell us games still favor 4-6 fast cores above all else?
Posted on Reply
#11
iO
"bug said:
Do we really need benchmarks to tell us games still favor 4-6 fast cores above all else?
No. But it's interesting that Intel pays for benchmarks that throw their new HEDT chips under the bus just to make the 2700K look bad.
Posted on Reply
#14
jboydgolfer
i hope AMD builds a better chip , instead of a cheaper chip, soon. The "triple the core, at half the price" method , isnt working entirely. Many will still pay the higher price, for the stronger chip, if only because its what they have been doing for a decade. The market needs the challenge for manufacturers, and minus the competition, no one but the better chip maker profits. It looks like intel will win out another generation (if only by a smaller percentage than their paid for results have shown), hopefully next release is different, for all of our sake.
Posted on Reply
#16
bug
"iO said:
No. But it's interesting that Intel pays for benchmarks that throw their new HEDT chips under the bus just to make the 2700K look bad.
Not really. These benchmarks, supposedly leaked through semi-official channels are always cherry-picked. At best.
They might mislead the uninformed buyer, but which uninformed buyer googles for unreleased products to begin with?
Posted on Reply
#17
Durvelle27
"jboydgolfer said:
i hope AMD builds a better chip , instead of a cheaper chip, soon. The "triple the core, at half the price" method , isnt working entirely. Many will still pay the higher price, for the stronger chip, if only because its what they have been doing for a decade. The market needs the challenge for manufacturers, and minus the competition, no one but the better chip maker profits. It looks like intel will win out another generation (if only by a smaller percentage than their paid for results have shown), hopefully next release is different, for all of our sake.
Why does AMD need to build a better chip

It has be proven with even previous gen that intel on average is barely 10% faster than AMDs offering while having much higher clocks

Are people that naive to rave about a measly performance gain but a bigger hand in your pockets
Posted on Reply
#18
Vya Domus
Intel, paying third parties for skewed and misleading comparisons ?

I am absolutely shocked.
Posted on Reply
#19
Gasaraki
I don't see anything wrong with those graphs. They tested with multiple ram speeds for Ryzen, it looks balanced to me. Testing at 1080 is the standard for testing for CPU, etc. 3200MHz memory clocks for Ryzen is pretty standard.

"They reinforced the memory by adjusting the frequency manually. This gives the Intel platform a significant performance advantage against AMD."
-Memory frequency and timings have minimal effects on Intel processors. We all know this.

"The next part of its deception was testing both setups at 1080p on "Ashes of the Singularity" CPU benchmark with medium settings, to obtain extremely suspicious performance numbers"
-Didn't know testing Ashes at 1080 on medium is suspicious.

I looked at the pdf document. It doesn't look like they loosened any timings and Ryzen is actually running at a faster setting 2933 vs 2600 for Intel systems.
Posted on Reply
#20
lexluthermiester
"bug said:
Do we really need benchmarks to tell us games still favor 4-6 fast cores above all else?
No, we really don't.
"Durvelle27 said:
Why does AMD need to build a better chip
It has be proven with even previous gen that intel on average is barely 10% faster than AMDs offering while having much higher clocks
Are people that naive to rave about a measly performance gain but a bigger hand in your pockets
I'm inclined to agree. AMD is, at the moment, the better value and offers excellent performance/price ratio compared to Intel.
Posted on Reply
#21
bug
"Vya Domus said:
Intel, paying third parties for skewed and misleading comparisons ?

I am absolutely shocked.
It's not like AMD didn't "leak" those video processing benchmarks over and over before the Zen release, making it look like it will destroy Intel in IPC.
As I have said before, absolutely no reason to be shocked. Apply the mandatory grain of salt or disregard these "leaks" entirely. Goes a long way preserving one's sanity ;)
Posted on Reply
#22
Zifnab
"lexluthermiester said:
I'm inclined to agree. AMD is, at the moment, the better value and offers excellent performance/price ratio compared to Intel.
I love umbrella comments like this. For who? To me it's worth the extra money for that 10% (which is being gracious when my 8700k will run at 5.1 all day long) when it comes to gaming.
Posted on Reply
#24
Aldain
"bug said:
It's not like AMD didn't "leak" those video processing benchmarks over and over before the Zen release, making it look like it will destroy Intel in IPC.
As I have said before, absolutely no reason to be shocked. Apply the mandatory grain of salt or disregard these "leaks" entirely. Goes a long way preserving one's sanity ;)
actually that never happen.. thank you fr playing
Posted on Reply
#25
lexluthermiester
"Aldain said:
actually that never happen.. thank you fr playing
Actually it did, and those "leaks" got a lot of people excited about Ryzen.
Posted on Reply
Add your own comment