TPU Math Check

[oZ65]

Oops. Just noticed I fucked up. Was being hasty since I had more pressing things to do at work
Will change it accordingly when I get home ...

As I think I've solved (=IF(C2>B2;C2/B2;2-B2/C2))
https://docs.google.com/spreadsheets/d/1I425L4JgHyJY0bxneoT3qqoxGtvh9amY5rcWVBNUx6g/edit?usp=sharing

 

[HTC]

That's my main concern, people will complain about cherry-picking, especially less well informed readers.

Here's an extreme example how a game can be excluded twice, if outliers were to be excluded: doom.

In OGL, it's performance on AMD is so bad that it can very likely be the one excluded. On vulkan, it's performance is so good ... that it's likely to be excluded. on nVidia's side the same game will probably be left on the final score because it's performance should not be an outlier with either API.

So long as there is ample information explaining how the results get achieved, i don't think exclusion of outliers is a problem.

 

[W1zzard]

Yup, and people won't be aware which games were included because so many individual cards but one summary graph and four of them

 

[Aquinus]

I think the problem is that people aren't considering what the percentages are relative to.

Lets consider the first item, Far Cry Primal:
RX 480: 58.1
GTX 1060: 64.7

111.3957 was the result @W1zzard got.

If we're going to compare the GTX 1060 against the RX 480, you need to take the FPS difference between the two compared to the GTX 1060.
That means, the difference would be (GTX FPS - RX FPS), then you would divide by the GTX frame rate to get a percentage based on the GTX's result.

(64.7 - 58.1) / 64.7 = 0.102
--- Or 10.2%, or as it's being reported on the graph, 110.2%, since 10.2% represents the gain or loss relative to the GTX 1060.

If we're going to compare the RX 480 against the GTX 1060, you essentially swap the GTX and RX values to be compared to the RX 480.
That means that the difference for this car would be (RX FPS - GTX FPS), then like above, you would divide the frame rate the value you're comparing against, which would be the RX.

(58.1 - 64.7) / 58.1 = -0.1136
--- Or -11.36% or, as being reported on the graph 88.64% (100% + (-11.36%) which represents the gain or loss relative to the RX 480.

The problem with W1zz's logic is that he's basically saying, "Let's get the difference between the two relative to the RX 480 but, merely flipping the sign on the percent difference to reflect a gain for the GTX". While this does show a difference, it's still a difference relative to whatever the initial calculation was before, which means that 11.36% gain is relative to the 58.1 FPS achieved on the RX 480 NOT the 64.7 FPS achieved by the GTX 1060.

What people need to realize is that the GTX 1060 (in the case of Far Cry Primal,) might have been 11.36% faster than the RX 480 but, that also means that the RX 480 is 10.2% slower than the GTX 1060 because the values used in the divisor changes.

This works well when strictly comparing two cards but, it doesn't work so well when you're trying to compare several different cards because the divisor is constantly changing to match the card it is being compared against so every value only means something between those two cards, so the comparison between two sets of different cards (even if one card is the same,) doesn't mean a whole lot.

I would argue that something based off of 60 FPS (the baseline for smooth gameplay,) would yield better results because the scale is consistent for all cards that may be compared, where 100% (or 0% gain/loss) would indicate an average of 60 FPS is maintained, >100% means that over 60 FPS was maintained and < 100% means under 60 FPS was maintained. This has the virtue of the divisor being a constant between all cards, so comparisons between any set of cards will be consistent.

That's my main concern, people will complain about cherry-picking, especially less well informed readers.

I think having more relative numbers and explaining exactly what they represent and how they were derived would make more sense. I would surmise that the average person doesn't realize that a GTX 1060 being X amount faster than a RX 480 does not mean that the RX 480 is X amount slower than the GTX 1060 because percentages change when what you're comparing against changes.

Either way, this is what I came up with. I think W1zz's numbers are correct and that it's more of people's understanding of them that's adding confusion.

Personally, I like the idea of calculating numbers relative to a constant like 60 FPS so it describes how much more playable a game is instead of how much faster a GPU is in comparison. What good are two GPUs if they both can't play a game smoothly?

 

[HD64G]

You need to average the percentage differences, otherwise you are favouring benchmarks that run high FPS.

Bench 1: 1000 FPS card 1, 1100 FPS card 2
Bench 2: 100 FPS card 1, 105 FPS card 2

1100 vs 1205. card2 = 109.5% - wrong, should be 107.5%

Agreed. The mean number of SUM(1060/480 FPS in each game) divided by the number of games tested is the only correct method to have the average diff between them.

Just done the whole review FPS with that method and GTX1060 is 12,5% faster than RX480 in average. Goes to 11,5% when Warcraft is out of the calculation.

My only complaint is Doom's and Warhammer's and Hitman's (DX12) absense.

 

[Aquinus]

Agreed. The mean number of SUM(1060/480 FPS in each game) divided by the number of games tested is the only correct method to have the average diff between them.

I think using 60 FPS as a baseline tells you a little more about gameplay as you could designate percentage ranges that describe smoother results. Any relative comparison is going to leave out some kind of information, the question is what kind of information are you trying to capture?

 

[HD64G]

I think using 60 FPS as a baseline tells you a little more about gameplay as you could designate percentage ranges that describe smoother results. Any relative comparison is going to leave out some kind of information, the question is what kind of information are you trying to capture?

I think that the most objective comparison is to average the difference between the GPUs compared using one of them as the base. Especially when you have many games and many GPUs in the equation as @W1zzard has.

 

[Aquinus]

I think that the most objective comparison is to average the difference between the GPUs compared using one of them as the base. Especially when you have many games and many GPUs in the equation as @W1zzard has.

The most objective way would be to not even touch percentages and stick with FPS numbers, difference or otherwise but, using percentages has a way of obscuring values unless your values started as percentages.

 

[the54thvoid]

@W1zzard - tomorrow I will be eating German sausage in Glasgow at a bierhalle. In that respect, I am one with you and I command you to not be swayed by nonsense. Use the geometric average by all means but leave it at that. The more complex you try to make it, the worse things will be. Keep it simple. It's your website god dammit.

 

[laszlo]

I think the problem is that people aren't considering what the percentages are relative to.

Lets consider the first item, Far Cry Primal:
RX 480: 58.1
GTX 1060: 64.7

111.3957 was the result @W1zzard got.

If we're going to compare the GTX 1060 against the RX 480, you need to take the FPS difference between the two compared to the GTX 1060.
That means, the difference would be (GTX FPS - RX FPS), then you would divide by the GTX frame rate to get a percentage based on the GTX's result.

(64.7 - 58.1) / 64.7 = 0.102
--- Or 10.2%, or as it's being reported on the graph, 110.2%, since 10.2% represents the gain or loss relative to the GTX 1060.

If we're going to compare the RX 480 against the GTX 1060, you essentially swap the GTX and RX values to be compared to the RX 480.
That means that the difference for this car would be (RX FPS - GTX FPS), then like above, you would divide the frame rate the value you're comparing against, which would be the RX.

(58.1 - 64.7) / 58.1 = -0.1136
--- Or -11.36% or, as being reported on the graph 88.64% (100% + (-11.36%) which represents the gain or loss relative to the RX 480.

The problem with W1zz's logic is that he's basically saying, "Let's get the difference between the two relative to the RX 480 but, merely flipping the sign on the percent difference to reflect a gain for the GTX". While this does show a difference, it's still a difference relative to whatever the initial calculation was before, which means that 11.36% gain is relative to the 58.1 FPS achieved on the RX 480 NOT the 64.7 FPS achieved by the GTX 1060.

What people need to realize is that the GTX 1060 (in the case of Far Cry Primal,) might have been 11.36% faster than the RX 480 but, that also means that the RX 480 is 10.2% slower than the GTX 1060 because the values used in the divisor changes.

This works well when strictly comparing two cards but, it doesn't work so well when you're trying to compare several different cards because the divisor is constantly changing to match the card it is being compared against so every value only means something between those two cards, so the comparison between two sets of different cards (even if one card is the same,) doesn't mean a whole lot.

I would argue that something based off of 60 FPS (the baseline for smooth gameplay,) would yield better results because the scale is consistent for all cards that may be compared, where 100% (or 0% gain/loss) would indicate an average of 60 FPS is maintained, >100% means that over 60 FPS was maintained and < 100% means under 60 FPS was maintained. This has the virtue of the divisor being a constant between all cards, so comparisons between any set of cards will be consistent.

I think having more relative numbers and explaining exactly what they represent and how they were derived would make more sense. I would surmise that the average person doesn't realize that a GTX 1060 being X amount faster than a RX 480 does not mean that the RX 480 is X amount slower than the GTX 1060 because percentages change when what you're comparing against changes.

Either way, this is what I came up with. I think W1zz's numbers are correct and that it's more of people's understanding of them that's adding confusion.

View attachment 77186

Personally, I like the idea of calculating numbers relative to a constant like 60 FPS so it describes how much more playable a game is instead of how much faster a GPU is in comparison. What good are two GPUs if they both can't play a game smoothly?

i agree to calculate the % to a fix baseline otherwise average of averages between any card give false numbers

now problem is we don't have an implemented standard which shall be mandatory in this field of reviewing ; basically all reviewers can calculate as they think is correct however the obtained testing results, for same hardware, show discrepancies which are hard to explained

ideal will be to have a specially developed bench soft(which shall be impartially), coded in all available currently used api's and cards tested&compared/each api basically no real game tests; this method will show exactly differences between each competitor; we'll see the differences and know the cause also( if one is lazy to improve architecture or other to write good drivers ); this may force them to improve(really??) and we, end-user will gain(go to work more if you want to gain...)

unfortunately as i see both of current major gpu producer don't really care about us as both are greedy;one is selling high priced product,which blow competition, and other mid-priced products lacking of good drivers

 

[Dethroy]

That's my main concern, people will complain about cherry-picking, especially less well informed readers.

You can't be everybody's darling. Someone is gonna complain no matter what. But switching to the geometric mean for future reviews would be a welcome change




Edit:

I always thought it would be awesome and helpful if the percentages would adjust relatively to the card you mouse over.

The reason why we don't use such a system is because I want people to be able to right click the image, get the URL, so they can easily repost it.

I see where you are coming from. But this would address what Aquinos was trying to explain in his long post #54.

 

[erocker]

IMO, just do what you've been doing for the past 12 years reviewing on this site. No need for change at all.

 

[HD64G]

The most objective way would be to not even touch percentages and stick with FPS numbers, difference or otherwise but, using percentages has a way of obscuring values unless your values started as percentages.

It can't be fair to give the low FPS of Crysis 3 lower importance in the result to the Warcraft's 100s of FPS result. This is THE unfair. Percentage diff for each result keeps all results equal in order to get a fair average too. Math is logic and fairness by itself when used well. Consclusions and interpretations are the not always objective ones.

 

[the54thvoid]

Consclusions and interpretations are the not always objective ones.

I would suggest the reactions of the reader base and entrenched viewpoints are not always objective. The views of @W1zzard tend to be very fair, they only tend to seem unfair if it's not what you want to hear.

Close the thread @W1zzard? - you've got your feedback.

 

[Dethroy]

I would suggest the reactions of the reader base and entrenched viewpoints are not always objective. The views of @W1zzard tend to be very fair, they only tend to seem unfair if it's not what you want to hear.

Close the thread @W1zzard? - you've got your feedback.

i actually thought this thread was civil and productive or at least somewhat thought provoking. That was until some people dragged it a bit on the subjective side of things...

 

[the54thvoid]

i actually thought this thread was civil and productive or at least somewhat thought provoking. That was until some people dragged it a bit on the subjective side of things...

Oh, it is civil but it's now flogging a statistical horse. And the post above my previous one has already side swiped at W1zz's conclusions.

 

[HD64G]

I would suggest the reactions of the reader base and entrenched viewpoints are not always objective. The views of @W1zzard tend to be very fair, they only tend to seem unfair if it's not what you want to hear.

Close the thread @W1zzard? - you've got your feedback.

My point about consclusions and interpretations are general and not pointing to @W1zzard 's. Just comparing opinions to math results being ALWAYS fair when used well.

 

[Aquinus]

Percentage diff for each result keeps all results equal in order to get a fair average too.

There is no such thing is a fair average when you complain about the outliers like WoW against an average. If people are complaining about outliers, give them the standard deviation and tell them to stop whining.

 

[HD64G]

There is no such thing is a fair average when you complain about the outliers like WoW against an average. If people are complaining about outliers, give them the standard deviation and tell them to stop whining.

I am talking about the only matehmatically correct way to get the average performance of different GPUs in a set of games. @W1zzard's method of getting the average is the correct one and this is what the thread is about afterall.

 

[Dethroy]

I am talking about the only matehmatically correct way to get the average performance of different GPUs in a set of games. @W1zzard's method of getting the average is the correct one and this is what the thread is about afterall.

Then prove that this is wrong ...

[...]
Assume we have benchmarks of 2 games:
1) RX480: 50 fps GTX1060: 100fps
2) RX480: 100fps GTX1060: 50 fps

Intuitively we would rate both cards equal, right?

Now lets look how TPU calculates:

RX480 GTX1060 100*GTX/RX
50 100 200
100 50 50

arith mean: 125
geom mean: 100

 

[birdie]

A lot of solutions given in this article are outright crazy.

It should be all normalized to the card being tested without swapping signs or anything, preferably using the geometric mean.

Let card A be baseline. Then every result of the card B should be calculated as result = FPS(CardB)/FPS(CardA).

The overall result of any other card then will be:

End of conversation.

The arithmetic mean doesn't cut it because when the baseline is compared to a very different much stronger card, the arithmetic mean will give a bigger more unrealistic number (read advantage to that card).

The attached spreadsheet can be trivially extended to include as many cards/games as you need.

 

[HD64G]

Then prove that this is wrong ...

GPU-RX480-GTX1060

Game1 FPS-50-100
Game2 FPS-100-50

Performance of GTX1060 with RX480 as a base

Game1 performance for GTX1060: 100/50= 2
Game2 performance for GTX1060: 50/100= 0,5

Average % performance of GTX1060 if we put RX480 as a base of 100 is (2+0,5)/2=1,25 which is wrong indeed as the 2 GPUs are equal in practice.

So, direct comparison between 2 GPUs is better to be done just by putting another GPU, faster than both as a base.

Supposing now that we have an even better GPU named X with 200 FPS in both of the games and we use that as the base our calculations go as:

Game1 for 1060: 100/200=0,5
Game2 for 1060: 50/200=025

Game1 for 480: 50/200=0,25
Game2 for 480: 100/200=0,5

When we use the X GPU as a base of 100, we calculate the % performance of 480 and 1060 and we get:

GTX1060: 100*[(0,5+0,25)/2]=37,5
RX480: 100*[(0,25+0,5)/2]=37,5

So, RX480=GTX1060 in performance.

But we need the best possible GPU to put as a base in order to have correct and independent results in average not influenced from neither of the 2 compared ones.

Geometrical mean is just lowering the differences and is used for other type of measurements and needs. Not useful for this type of comparisons that just need an average to show how much faster in % is a GPU vs another one.

UPDATE after the following post of @marios15 where he was absolutely correct about my mistake in the 1st part of my calculations.

 

[marios15]

GPU-RX480-GTX1060

Game1 FPS-50-100
Game2 FPS-100-50

Arithmetical Mean with 100 as a base

RX480: (50+100)/100=75
GTX1060: (100+50)/100=75

Performance of GTX1060 with RX480 as a base

75/75=100

RX480=GTX1060 in performance.

Geometrical mean is just lowering the differences and is used for other type of measurements and needs. Not useful for this type of comparisons that just need an average to show how much faster in % is a GPU vs another one.

For real?
First of all you are using fps targets.

If you want to see how close to 100fps, one card is you can do it with:
Method 1 - Average FPS in numbers.
(FPS1+FPS2+FPS3+FPS4)/GAMECOUNT = AVG GAME FPS

This is bad since the average of (400,5,5,200) is 152.5, which is 52.5% above 100, but it doesn't make any sense since games 2 and 3 only had 5fps so this method can't be used accurately.

In your case that would give us 75% for both cards meaning that they're equal, but there would be no card at 100%, and you would have a graph with a range from 20% to 300%, assuming all cards are included.

Method 2 - Arithmetic mean difference in percentages
[(FPS1/100) + (FPS2/100).....+(FPSn/100)]/ n * 100% = AVG PERCENTAGE DIFFERENCE from 100fps
This is good for comparing how much slower or faster than 100FPS and that's the problem, as it opens a whole different discussion.
When some games are rpg/rts/racing, does it matter if they hit 100fps?
Do you read a review of a new product, to see how relative to 100fps its performance is in games?
What if the product is cheap low-end?
What if all games are below/above 100fps?

You are comparing a gpu's performance to a certain FPS number which could mean nothing in one game and everything in another.
What you should be doing is this:

Let card A be baseline. Then every result of the card B should be calculated as result = FPS(CardB)/FPS(CardA)
View attachment 77244

End of conversation.

@W1z

I know that what i am about to ask can be a lot of work, but the measuring method i use in my engineering classes, is the geometric mean, including deviation.
So currently an arithmetic mean is used, but could a geometric mean + deviation in future summary graphs be used, for ex. 110%±15%, since this would show one card is 10% faster on average, but sometimes 5% slower and sometimes 25% faster.

Also, i assume you're doing multiple runs, so why not include the deviation between runs, this would show how consistent performance is on a card, like 74|80|85 fps or -6|80|+5

 

[P4-630]

LOL, getting a headache reading all this

W1zz has already decided it seems some posts back.

 

[CAPSLOCKSTUCK]

I was just about to request a head shot.