GPU Test System Update May 2019

[londiste]

I'm a firm believer in quality over quantity.
Would 200%, without a doubt prefer cutting number of tests\games by half just to have a zen2 system on board.

In case of TPU quantity is its own quality. When it comes to GPUs having two platforms tested would not add much value. If Zen2 comes out and is best in gaming I am sure the platform will move to that.

CPU testing is a different discipline and would require much more than just 2 platforms and twice the tests to be useful. 4-6-8-12 cores/threads would probably need to be on the menu for that.

 

[Face2Face]

GN uses a smaller test selection, but they test much more rigorously, report more data, and choose their test titles with very stringent criteria - for example, you test Civilisation VI for GPU reviews. I would argue that's a pointless thing to do unless this is also your CPU test bench, since Civilisation VI is almost always CPU limited, not GPU. GN would also criticise the use of Civ VI as a benchmark in general, as they've seen patches for Civ VI impact performance dramatically in the past, making their previous test data inapplicable.

GN also reports 1% and 0.1% lows in framerate, but, beyond that, they generate frametime plots like these ones, that show in a much more significant manner, issues in specific games on specific hardware.

For example, in this test you can clearly see that the 1050Ti provided not only a lower overall framerate (longer frametimes), but it provided it less consistently overall, meaning more perceptible stutter.

View attachment 123413



They test F1 2018, Apex Legends, Sniper Elite 4, Far Cry 5, GTA V, and Shadow of the Tomb Raider, at 1080p and 1440 for all games and Sniper Elite at 4K because they've found it to be a repeatable and well optimised DX12 title.

Here's a video from them outlining their own updates to their testing methodology in the first segment:

I'm only seeing a 3-5 ms variance in frame times. That's not going to be noticeable. 10+ MS spikes are noticeable when you're running close or at 60 FPS.

 

[GlacierNine]

I'm only seeing a 3-5 ms variance in frame times. That's not going to be noticeable. 10+ MS spikes are noticeable when you're running close or at 60 FPS.

Humans are very much capable of perceiving things on that sort of timescale, but to some extent it is relative.

If I'm doing something passive, like listening to music, I can't necessarily tell small variations in sync between the musicians.

When I'm recording music, and therefore playing notes and hearing them through the studio monitors, I can perceive the difference between interface latencies of 8 and 4.5ms as a distinctly more "locked in" feel to the playing experience.

To that extent, I think it's entirely reasonable to expect that frametime variance of 3-5 ms would be perceptible when using higher refresh rate displays with low input lag.

 

[Face2Face]

To that extent, I think it's entirely reasonable to expect that frametime variance of 3-5 ms would be perceptible when using higher refresh rate displays with low input lag.

I'm not doubting statement above - at 144 FPS, a 3-5 MS FT variance would be more noticeable than if it was 60 FPS (like the GN plot above). I say this because I've tested this for myself many times with dozens of graphics cards. Also, they're pulling this data from the beginning of the render pipeline, before the OS and the driver. What is shown in a FT plot, isn't necessarily felt by the user. I believe GN also stated that a FT variance of 8 or more ms would be noticeable, but anything less you're unlikely to feel it.

 

[Veradun]

I am confused:


The following titles were removed:

• Deus Ex Mankind Divided: Uses DX12, but is simply getting too old (2016)
• Ghost Recon Wildlands: Too old (2017)
• Grand Theft Auto V: Too old (2015)

The full list of games is now: Ace Combat 7, Anno 1800, Assassin's Creed Odyssey, Battlefield V, Civilization VI (2016), Darksiders 3, Devil May Cry 5, Divinity Original Sin 2, F1 2018, Far Cry 5, Hitman 2, Metro Exodus, Monster Hunter World, Rage 2, Rainbow Six Siege (2015), Sekiro, Shadow of the Tomb Raider, Shadow of War, Strange Brigade, Witcher 3 (2015), and Wolfenstein 2.

How come a 2016 game and a 2017 game are too old while a 2016 and two 2015 games are not?

 

[londiste]

From couple of my posts earlier in the thread:

Witcher 3 is on CDPR's own REDEngine 3. Cyberpunk 2077 will be REDEngine 4 that will likely take Witcher 3's place in benchmarking.
Deus Ex Mankind Divided is on Dawn Engine which is modified Glacier Engine II - the same one that powers Hitman games, currently Hitman 2.

Ghost Recon Wildlands runs on Ubisoft's AnvilNext 2.0 that is already represented with 2 other games. AnvilNext 2.0 variability is rather large compared to other engines but the engine is not significant enough to warrant too many games with it in the lineup.

Before someone points out that Rainbow Six Siege is also on the same engine and is older - all this is true - but R6 Siege keeps being updated, is popular and is one of very few competitive FPS games that can use as much hardware as possible thrown at it. This is also one of the few games where I would seriously argue the FPS differences in 100-200 range and perhaps above do matter.

- Civilization, again, is unique game and engine and a long-running series with a definite large fanbase.

GTA V is kind of an outlier and a biased one at that. I suppose if/when RDR2 will get a PC release that will find its way into the games list. Witcher 3 is in a similar situation but seems to be a far more balanced benchmark.
I would be OK with dropping Civ VI, but not because it is not significant but because the stability of the benchmark results is in question.

 

[gridracedriver]

I am confused:


The following titles were removed:

• Deus Ex Mankind Divided: Uses DX12, but is simply getting too old (2016)
• Ghost Recon Wildlands: Too old (2017)
• Grand Theft Auto V: Too old (2015)

The full list of games is now: Ace Combat 7, Anno 1800, Assassin's Creed Odyssey, Battlefield V, Civilization VI (2016), Darksiders 3, Devil May Cry 5, Divinity Original Sin 2, F1 2018, Far Cry 5, Hitman 2, Metro Exodus, Monster Hunter World, Rage 2, Rainbow Six Siege (2015), Sekiro, Shadow of the Tomb Raider, Shadow of War, Strange Brigade, Witcher 3 (2015), and Wolfenstein 2.

How come a 2016 game and a 2017 game are too old while a 2016 and two 2015 games are not?

I agree and i have another question:
in addition to those you mentioned, at the beginning of 2019, these 8 games were widely used:

Res Evil 2
Forza Horizon 4
WWZ
The Division2
Anthem
Dirt rally 2
New Dawn
Apex legends
---
(mention) F1 2019 too soon
---

why these games are not selected and used by removing older games on the list?

Finally, In my opinion, games like CiV6, Anno, AC7, kill the final averages and do not give a real picture of today's situation.

Can anyone explain me?
thank you

 

[Vayra86]

I agree and i have another question:
in addition to those you mentioned, at the beginning of 2019, these 8 games were widely used:

Res Evil 2
Forza Horizon 4
WWZ
The Division2
Anthem
Dirt rally 2
New Dawn
Apex legends
---
(mention) F1 2019 too soon
---

why these games are not selected and used by removing older games on the list?

Finally, In my opinion, games like CiV6, Anno, AC7, kill the final averages and do not give a real picture of today's situation.

Can anyone explain me?
thank you

As stated by @londiste and I fully support that idea: W1zzard selects games based on engines. Its irrelevant what gets played the most or what is 'new' per say - what matters for a GPU review is that the review gives a fair picture on the performance of the card, in as wide a selection of different situations as possible. Civilization is in there because its a 4X / simulation that presents a high CPU load that directly impacts scrolling across the map for example. Its very easy to see even small FPS differences in actual play so it really matters that such games run well. If you read the small bits of text with each game in a review you will recognize a lot is being said about the engine in question, almost every time. The pattern with GPUs is that each architecture can handle different game engines in a specific way. And also: software follows hardware and vice versa. So its interesting to see how engines evolve and make better use of the GPU over time.

A review is not a pissing contest for brand A or B and despite what Youtubers want you to believe its not a 'Battle' either; its a buyer's advice. If the relative performance summaries matter to you, you should also care about why they show the number they do. Its irrelevant that game A/B/C 'skews the relative summary a bit' as long as you know its a game you wouldn't ever play, right? But at the same time, if I'm only reading about shooters and f2p's and a review includes Fortnite, CS GO, Dota 2 and Apex Legends and the new yearly sequel of each popular franchise, its just about 100% worthless to a large part of the audience, and it will show yesterdays' performance anyway because most engines are re-used every time with very minor upgrades.

 

[Veradun]

As stated by @londiste and I fully support that idea: W1zzard selects games based on engines. Its irrelevant what gets played the most [...] what matters for a GPU review is that the review gives a fair picture on the performance of the card, in as wide a selection of different situations as possible.

Not when you review has a "performance summary" chart that use the arithmetic mean between unpopular and extremely popular games

 

[Vayra86]

Not when you review has a "performance summary" chart that use the arithmetic mean between unpopular and extremely popular games

I don't follow your logic there.

 

[londiste]

in addition to those you mentioned, at the beginning of 2019, these 8 games were widely used:

The games need to stay popular and relevant for hopefully a year. The intent is to keep the same list for as much as possible. Out of the games you listed there is a good chance that half or more are going to quickly go out of style.

Finally, In my opinion, games like CiV6, Anno, AC7, kill the final averages and do not give a real picture of today's situation.

What exactly do you mean by killing the final averages and not giving a real picture?

 

[gridracedriver]

As stated by @londiste and I fully support that idea: W1zzard selects games based on engines. Its irrelevant what gets played the most or what is 'new' per say - what matters for a GPU review is that the review gives a fair picture on the performance of the card, in as wide a selection of different situations as possible. Civilization is in there because its a 4X / simulation that presents a high CPU load that directly impacts scrolling across the map for example. Its very easy to see even small FPS differences in actual play so it really matters that such games run well. If you read the small bits of text with each game in a review you will recognize a lot is being said about the engine in question, almost every time. The pattern with GPUs is that each architecture can handle different game engines in a specific way. And also: software follows hardware and vice versa. So its interesting to see how engines evolve and make better use of the GPU over time.

A review is not a pissing contest for brand A or B and despite what Youtubers want you to believe its not a 'Battle' either; its a buyer's advice. If the relative performance summaries matter to you, you should also care about why they show the number they do. Its irrelevant that game A/B/C 'skews the relative summary a bit' as long as you know its a game you wouldn't ever play, right? But at the same time, if I'm only reading about shooters and f2p's and a review includes Fortnite, CS GO, Dota 2 and Apex Legends and the new yearly sequel of each popular franchise, its just about 100% worthless to a large part of the audience, and it will show yesterdays' performance anyway because most engines are re-used every time with very minor upgrades.

first of all I apologize for my poor English

I'm not saying that some games released in 2019 use new engines, but it is undeniable that some of the games I mentioned go well and better than older ones on amd cards.
I do not mean that they should be taken because they are better with amd, but if I buy a card today it is to play games released in the last year at most, and not 3-4 years ago, or to do + 100fps in CiV6, when 60 they are already sufficient, being a strategic turn-based one.

Finally the problem is that to evaluate a card and therefore the purchase, the user looks at the table of averages and not the single game.

At least in Italy, I am not abroad, it is full of "dissing" for this, taking as a reference the average without looking at that in the last titles released things seem to go in another direction like DX12 or Vulkan, which is not that of dx11 or UE4 biased ...

 

[Veradun]

I don't follow your logic there.

I'll give you some time to think about it.

 

[gridracedriver]

The games need to stay popular and relevant for hopefully a year. The intent is to keep the same list for as much as possible. Out of the games you listed there is a good chance that half or more are going to quickly go out of style.
What exactly do you mean by killing the final averages and not giving a real picture?

at least 1 year?
so for example I buy 1 card in Q1 2019 to see how it goes with the 2017 titles?

I mean that there are games like the turn-based strategists that do not need to do +60 fps, so it's true that nvidia do double the fps, but it's not even necessary, they are not FPS TPS, and this invalidates the final averages .

AC7 is a classic example of UE4 biased, a stock engine that runs bad on amd.

 

[londiste]

"Better than older ones on amd cards" is a bad argument for including games in a list of games for GPU review.

Looking at your list of games:
- RE2 and Division 2 are games bundled with AMD hardware. WWZ was developed with help from AMD and is heavily promoted by AMD.
- RE Engine (RE2) is already represented by Devil May Cry.
- Forza Horizon 4 results are variable on the settings (MSAA, if I remember correctly).
- Dirt Rally is on EGO Engine and it is probably better represented by the yearly edition of F1 and F1 2019 will likely replace F1 2018 in the list.
- New Dawn is the new iteration of Dunia Engine and will probably replace Far Cry 5 in the list.
- Relevance of Anthem is arguable and Frostbite is better represented by current iteration of Battlefield.
- We will have to see about Apex Legends. Its popularily is based on it being free. Whether it remains popular is unknown.

Why not Sinking City, Three Kingdoms or Plague Tale?

AC7 is a classic example of UE4 biased, a stock engine that runs bad on amd.

Choice of AC7 is much more due to it being a current representative of a genre than Unreal Engine.

 

[Vayra86]

I'll give you some time to think about it.

An arithmetic mean of the games used in the review is just that. You are the one making more out of that - your perception is the problem. And an arrogant response like that one is the wrong way to have a conversation or make your point.

So try again...

 

[gridracedriver]

"Better than older ones on amd cards" is a bad argument for including games in a list of games for GPU review.

Looking at your list of games:
- RE2 (and Devil May Cry on the same engine) and Division 2 are games bundled with AMD hardware. WWZ was developed with help from AMD and is heavily promoted by AMD.
- Forza Horizon 4 results are variable on the settings (MSAA, if I remember correctly).
- Dirt Rally is on EGO Engine and it is probably better represented by the yearly edition of F1 and F1 2019 will likely replace F1 2018 in the list.
- New Dawn is the new iteration of Dunia Engine and will probably replace Far Cry 5 in the list.
- Relevance of Anthem is arguable and Frostbite is better represented by current iteration of Battlefield.
- We will have to see about Apex Legends. Its popularily is based on it being free. Whether it remains popular is unknown.

Why not Sinking City, Three Kingdoms or Plague Tale?

Choice of AC7 is much more due to it being a current representative of a genre than Unreal Engine.

because they are games I've never heard, let's include them why not!

AC7 really got bad grades, this should also be taken into account in the reviews, for me it doesn't deserve to be tested for that too.

 

[Vayra86]

at least 1 year?
so for example I buy 1 card in Q1 2019 to see how it goes with the 2017 titles?

I mean that there are games like the turn-based strategists that do not need to do +60 fps, so it's true that nvidia do double the fps, but it's not even necessary, they are not FPS TPS, and this invalidates the final averages .

AC7 is a classic example of UE4 biased, a stock engine that runs bad on amd.

TBS doesn't need 60+ FPS? I really do like 100+ on my high refresh panel for any isometric game. Scrolling is awesome that way.

Don't forget a review gets read by a broad variety of people with different use cases. Many people play 'popular games', but a whole lot of people also play more obscure games and genres.

Isn't it a win if a review can be useful for all of those readers?

 

[gridracedriver]

TBS doesn't need 60+ FPS? I really do like 100+ on my high refresh panel for any isometric game. Scrolling is awesome that way.

don't you think this makes the final average incorrect and not properly weighted?

ps. I replied to the first post but is awaiting moderation

 

[Vayra86]

don't you think this makes the final average incorrect and not properly weighted?

ps. I replied to the first post but is awaiting moderation

The final average is correct if you read the review and know what games are included in it. Because then you also know that if you do NOT play Civilization, the Radeon VII is a fine card for you - but if you DO play Civilization, its a horrible card to buy. In addition, it highlights that Radeon VII is not quite as consistent as other cards, which can be a problem going forward as well.

In the end, thát is the purpose of a review: buyer's advice. Do you really buy your GPU based on one relative perf summary? Many people play specific types of games and will want to know how that type of game works out.

Last, about the game's age - Civilization VI's latest release was an expansion dated 14 Feb 2019 and it is historically a very popular game - one that gets played alot in LAN and offline. There is no question that it is still relevant today.

 

[Veradun]

An arithmetic mean of the games used in the review is just that. You are the one making more out of that - your perception is the problem. And an arrogant response like that one is the wrong way to have a conversation or make your point.

So try again...

Let's start from the beginning, then.

What's the purpose of a TPU review?

 

[Vayra86]

Let's start from the beginning, then.

What's the purpose of a TPU review?

Mr 14 posts and counting, I'm not going down this road with you - and its your loss not mine, I've explained why older games are in the selection, and that is all it is.

 

[gridracedriver]

The final average is correct if you read the review and know what games are included in it. Because then you also know that if you do NOT play Civilization, the Radeon VII is a fine card for you - but if you DO play Civilization, its a horrible card to buy. In addition, it highlights that Radeon VII is not quite as consistent as other cards, which can be a problem going forward as well.

In the end, thát is the purpose of a review: buyer's advice. Do you really buy your GPU based on one relative perf summary? Many people play specific types of games and will want to know how that type of game works out.

Last, about the game's age - Civilization VI's latest release was an expansion dated 14 Feb 2019.

Are not all games included in the final average?
VII was an example, like all the line up amd

I agree that based on the titles and the games we see that gpu choose, but this is not the case, at least in Italy.

And in Italy the average tpu is very popular...

 

[Veradun]

Mr 14 posts and counting, I'm not going down this road with you - and its your loss not mine, I've explained why older games are in the selection, and that is all it is.

I wonder why

 

[Vayra86]

I wonder why

I'll give you that one - because you are asking rhetorical questions, and questions that I have already answered way ahead of you. Now its your turn to connect the dots - good luck.