Tuesday, October 10th 2017

Principled Technologies' Response to Allegations of Horse Manure Data Disingenuous

Principled Technologies Wednesday published its first response to allegations of flawed and misleading "independent" comparison between the $319 AMD Ryzen 7 2700X and the 66% pricier $530 (pre-order price) Intel Core i9-9900K, which Intel used in its launch event to woo gamers and investors. In its response, the company elaborated on the reasons why it tested the AMD chip with memory and cooler settings reputed hardware reviewers found sub-optimal. "One goal of this study was to test the CPUs and their graphics subsystems, not the GPUs, so we ran the tests at the most common gaming resolution (62.06%), 1920×1080," reads the response, touting a foregone conclusion that gamers with $500 8-core processors still game at 1920 x 1080. We get that they, like every CPU reviewer, are trying to simulate a CPU-limited scenario, but to justify their settings with Steam Hardware Survey data as "the most common resolution," is a disingenuous argument.

We next see Principled Technologies justify the use of NH-U14S TR4-SP3 cooler on the Ryzen Threadripper 2990WX. Noctua, in its own TDP Guide for this cooler, states that 250W TDP (which also happens to be the TDP of the 2990WX), is the design limit of this air cooler, and goes as far as to mention that an additional NF-A15 fan, which is not included with the cooler, is recommended to be able to "increase Precision Boost headroom," implying that out of the box, the cooler is already bottlenecking the 2990WX. The Core i9-9980XE, on the other hand, has a rated TDP of 165W, and Noctua provides no additional guidance for 165W TDP Core X family processors, such as the Core i9-7980XE. Principled Technologies' reasoning for memory configuration proves they either continue to lack basic knowledge on AMD Ryzen memory controller limitations, or are deliberately disregarding it in an attempt to cripple AMD chips.
The Principled Technologies response contains no explanation as to why they used four memory modules, or a 4-module + dual-rank module configuration for mainstream desktop processors that feature dual-channel memory controllers. It has been common knowledge for close to two years now, that AMD Ryzen processors offer stunted performance when saddled with dual-rank memory modules, more so when all four memory slots of the motherboard are populated.

Simply mentioning the DRAM clock as 2933 MHz doesn't even paint half the picture. AMD platform motherboard BIOSes are programmed to automatically find stable memory timings for a given memory clock. Principled Technologies is quick to justify using the Wraith Prism cooler for the Ryzen 7 2700X "because AMD says so," but the same AMD document (Reviewer's Guide) that recommends Wraith Prism also provides a detailed account on the limitations of Ryzen's memory controllers. The controllers cannot handle 4-module + dual-rank configurations above DDR4-1866, 4-module single-rank (effectively dual-rank) above DDR4-2400, without severely loosening up memory timings to compensate for lack of stability.

2-module single-rank is not only an AMD-recommended configuration, but also 16 GB remains an overwhelmingly more popular memory amount than 32 GB vide Steam Hardware Survey (the same survey they used to justify 1080p game testing). In 2-module single-rank, a Ryzen processor will let you spool memory clocks all the way up to DDR4-3400, and memory modules exist that achieve that clock with reasonably tight timings.
It's good that Principled Technologies admitted the crime they committed testing the Ryzen 7 2700X with half its cores effectively disabled using the "game mode" toggle in Ryzen Master that no CPU reviewer tests with, and offer to re-test the processor. We wish that while they're at it, they also update their Ryzen Master utility to re-test the Ryzen Threadripper 2990WX with Dynamic Local Mode. The comparison with the re-badged Core i9-9980XE significantly will change.

Comparing a $319 processor with a 66% pricier $530 processor is disingenuous to begin with, and the only way you'll find out if the i9-9900K offers that much more performance is to wait for reviews from independent sources. So far there's none.
Add your own comment

22 Comments on Principled Technologies' Response to Allegations of Horse Manure Data Disingenuous

#1
eidairaman1
The Exiled Airman
So just ignore them
Posted on Reply
#2
champsilva
I use 4 memory sticks, just because!
Posted on Reply
#3
Darmok N Jalad
Seems like they should have just invited a few reputable tech sites over for some hands-on testing. Instead, they contract an obscure company with a bizarre and confusing website. Ironically, Principled Technologies promotes a book “Limit Your Greed,” which even says “nobody wins unless everybody wins.”

Reminds me of CTS-labs debacle. Go visit their website. Under available publications, it only has “AMDFlaws (3/12/2018) - Publication Summary Coming Soon...”
Posted on Reply
#4
Prima.Vera
Those idiots are getting too much advertisement recently.
Posted on Reply
#5
cadaveca
My name is Dave
Should be marked as editorial and not news. No matter how valid this is. :p
Posted on Reply
#6
mastrdrver
@btarunr PT is an Intel funded consortium much like Sysmark is.

tl;dr: Intel is required to say that benchmarks that favor their stuff unfairly with certain words per FCC settlement of AMD v Intel. Said phrase is found at the end of the PT document. It wasn't an accident. It was Intel up to their old shit again.
<div class="youtube-embed" data-id="3V8pEsjNa4Q"><img src="https://i.ytimg.com/vi/3V8pEsjNa4Q/hqdefault.jpg" /><div class="youtube-play"></div><a href="https://www.youtube.com/watch?v=3V8pEsjNa4Q" target="_blank" class="youtube-title"></a></div>
Posted on Reply
#7
Prima.Vera
Horse manure? :laugh::laugh::laugh:
That's the official word for "turd", or better..."bull shit" ? :roll:
Posted on Reply
#8
WikiFM
"One goal of this study was to test the CPUs and their graphics subsystems, not the GPUs, so we ran the tests at the most common gaming resolution (62.06%), 1920×1080," reads the response, touting a foregone conclusion that gamers with $500 8-core processors still game at 1920 x 1080.

Off topic using this as reference, if you play at any higher res than 1080p (specially 4K), then you don't need a $500 CPU since the GPU will be the bottleneck and having an expensive CPU wont give you more FPS right? As the res increases you should spend more in GPU and less in CPU right? Does it make any sense or I got it wrong?
Posted on Reply
#9
LDNL
They also wrote that they opted to use stock coolers on processors that had them included.

And since Intel processors doesn't ship with coolers anymore, they used a Noctua cooler lol.

Have a guess which processors were throttling because of thermal limits of a stock coolers.
Posted on Reply
#10
laszlo
now if they "re-make" the testing it will be foolish from their side to present the real numbers... i expect to see a small improvement only ; is the only way to keep credibility for them...
Posted on Reply
#11
londiste
Offtopic:
WikiFM said:
Off topic using this as reference, if you play at any higher res than 1080p (specially 4K), then you don't need a $500 CPU since the GPU will be the bottleneck and having an expensive CPU wont give you more FPS right? As the res increases you should spend more in GPU and less in CPU right? Does it make any sense or I got it wrong?
Oh, you will. I currently have an i5-8400 in my gaming box and it tends to occasionally bottleneck GTX 2080 at 1440p and Ultra Settings. There are even built-in benchmarks showing that - Shadow of Tomb Raider, Forza Horizon 4 and Gears of War 4 as some examples from what I tried recently. It is very game-specific though, most games tend to favor just 2-4 fast threads.
Posted on Reply
#12
Vayra86
WikiFM said:
"One goal of this study was to test the CPUs and their graphics subsystems, not the GPUs, so we ran the tests at the most common gaming resolution (62.06%), 1920×1080," reads the response, touting a foregone conclusion that gamers with $500 8-core processors still game at 1920 x 1080.

Off topic using this as reference, if you play at any higher res than 1080p (specially 4K), then you don't need a $500 CPU since the GPU will be the bottleneck and having an expensive CPU wont give you more FPS right? As the res increases you should spend more in GPU and less in CPU right? Does it make any sense or I got it wrong?
You got it exactly right, but even here on TPU and in many other places people persist in saying 1080p means you have an i3 and a 1060 or something.

Even Steam Survey still showing an overwhelming (!) amount of sub 1440p gamers is 'not relevant enough' somehow. And here's the kicker: on Steam Survey, many of the 'ultrawide' resolutions you see are people who use two separate screens side by side and only game on one ;) Its also completely not true that virtually every gaming oriented monitor out these days is first and foremost high refresh rate even before high resolution... :D

Oh well. Its an editorial disguised as news, let's just keep it at that I guess... The gist of it is still sound.
Posted on Reply
#13
oxidized
If i was them i'd go putting my head under the sand for good. They fkd up EVERYTHING and they still have the courage to even say a word. The old man interviewed by gamers nexus was utterly ridiculous, and he's one of the founders, i guess he's some kind of expert in damage control stuff, i can't imagine how good the others are at doing damage control, rofl.
Posted on Reply
#14
bug
I've been reading CPU benchmarks for 20 years and never have I seen a more finicky CPU than these Ryzens. Also, testing gaming performance in gaming mode being a downside is really out there.
Posted on Reply
#15
londiste
bug said:
I've been reading CPU benchmarks for 20 years and never have I seen a more finicky CPU than these Ryzens. Also, testing gaming performance in gaming mode being a downside is really out there.
Game Mode on 2700X is a dick move on PT part. This simply cannot be anything other than malicious compliance. They have enough experience and they should know precisely what it does.
Having Game Mode even available for 2700X is an interesting decision on AMD's part though.
Posted on Reply
#16
Vayra86
bug said:
I've been reading CPU benchmarks for 20 years and never have I seen a more finicky CPU than these Ryzens. Also, testing gaming performance in gaming mode being a downside is really out there.
Its typical AMD, add tons of options and features but forget to restrict them to the use cases where they shine. Total control in the hands of the user, even in obviously counterproductive situations. On the one hand, commendable, and on the other, incredibly stupid and above all: lazy.

londiste said:
Having Game Mode even available for 2700X is an interesting decision on AMD's part though.
Interesting... or a major, easily correctable mistake, of the type AMD has a long history of.
Posted on Reply
#17
rtwjunkie
PC Gaming Enthusiast
Darmok N Jalad said:
“Limit Your Greed,” which even says “nobody wins unless everybody wins.”
Awww, how cute. I guess their company leaders grew up in the “competition and hard work are evil” era, where “everybody gets a trophy.”
Posted on Reply
#18
WikiFM
Vayra86 said:
You got it exactly right, but even here on TPU and in many other places people persist in saying 1080p means you have an i3 and a 1060 or something.

Even Steam Survey still showing an overwhelming (!) amount of sub 1440p gamers is 'not relevant enough' somehow. And here's the kicker: on Steam Survey, many of the 'ultrawide' resolutions you see are people who use two separate screens side by side and only game on one ;) Its also completely not true that virtually every gaming oriented monitor out these days is first and foremost high refresh rate even before high resolution... :D
I can not say about developed countries, but in the third world most people play in 1080p with i3 and 1060 class because that is the highest they can afford.

I believe that a lot of gamers play in 60 Hz TVs (either 1080p or 4K) than in high refresh rates monitors, specially in the third world these monitors are not popular at all. Would be a nice idea that the next TPU poll were if you play above 60 Hz or not.

londiste said:
Offtopic:
Oh, you will. I currently have an i5-8400 in my gaming box and it tends to occasionally bottleneck GTX 2080 at 1440p and Ultra Settings. There are even built-in benchmarks showing that - Shadow of Tomb Raider, Forza Horizon 4 and Gears of War 4 as some examples from what I tried recently. It is very game-specific though, most games tend to favor just 2-4 fast threads.
2080 is the second fastest GPU and at 1440p is not a bottleneck for CPU sometimes.
Do you have a high refresh rate monitor? If not, what is the point of higher FPS.
Posted on Reply
#19
Slizzo
mastrdrver said:
@btarunr PT is an Intel funded consortium much like Sysmark is.

tl;dr: Intel is required to say that benchmarks that favor their stuff unfairly with certain words per FCC settlement of AMD v Intel. Said phrase is found at the end of the PT document. It wasn't an accident. It was Intel up to their old shit again.
<div class="youtube-embed" data-id="3V8pEsjNa4Q"><img src="https://i.ytimg.com/vi/3V8pEsjNa4Q/hqdefault.jpg" /><div class="youtube-play"></div><a href="https://www.youtube.com/watch?v=3V8pEsjNa4Q" target="_blank" class="youtube-title"></a></div>
Did you know that AMD has commissioned them for work in the past as well? While I can see and agree with everyone taking issue with their testing (under Intel's guidance mind you), it's hardly fair to also paint them as wholly funded by Intel when they are indeed an independent shop.
Posted on Reply
#20
bug
WikiFM said:
I can not say about developed countries, but in the third world most people play in 1080p with i3 and 1060 class because that is the highest they can afford.

I believe that a lot of gamers play in 60 Hz TVs (either 1080p or 4K) than in high refresh rates monitors, specially in the third world these monitors are not popular at all. Would be a nice idea that the next TPU poll were if you play above 60 Hz or not.



2080 is the second fastest GPU and at 1440p is not a bottleneck for CPU sometimes.
Do you have a high refresh rate monitor? If not, what is the point of higher FPS.
Yes, tech enthusiasts have a tendency to grow into a bubble where everything they read about is what people everywhere must be using. And everything is available at US pricing.
Posted on Reply
#21
dozenfury
PT seems like much more of a marketing firm than benchmarker. Even in their own advertising they make their service offering and niche pretty clear. Also, that they have done reviews in the past for AMD does not change this at all. It just means that AMD has used them for friendly marketing advertising/tests before too.

Most techie folks understand that just about any freshman college IT student could fairly easily run a benchmark. It's not rocket science. About the only hurdle to anyone starting their own benchmark company is just financial access to acquire (at least temporarily) the hardware to test. Setting up a PC to bios defaults, running a benchmark, and recording the results, is something I could have easily done at 13-14 years old as just a semi-techie kid. So the bar is pretty low. And with anything, when you hire what claims to be a 16 year professional corporation to perform what is a fairly simple task, they open themselves up to a lot of negative attention when they mess up. And there were at least 3 or 4 bad oversights/questionable decisions in this testing. Probably not a company I'll focus on when looking at benchmarks for my next CPU purchasing decision.

Intel shares some of this blame too. What it looks they were aiming to do in the first place is disappointing.
Posted on Reply
#22
mastrdrver
Slizzo said:
Did you know that AMD has commissioned them for work in the past as well? While I can see and agree with everyone taking issue with their testing (under Intel's guidance mind you), it's hardly fair to also paint them as wholly funded by Intel when they are indeed an independent shop.
They're about as independent as Sysmark is. In that Intel supplies the software engineers. AMD, nVidia, and VIA were all also part of Sysmark. We all know how biased Sysmark is, PT is nothing more then Sysmark with a new face.

Is PT owned by Intel? Of course not! They just supply PT with all the software engineers.
Posted on Reply
Add your own comment