1. Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Benchmarking secrets

Discussion in 'Graphics Cards' started by Mussels, Feb 10, 2008.

?

Was it worth me writing this?

  1. Yes

    25 vote(s)
    86.2%
  2. No

    4 vote(s)
    13.8%
  1. Mussels

    Mussels Moderprator Staff Member

    Joined:
    Oct 6, 2004
    Messages:
    43,547 (10.93/day)
    Thanks Received:
    10,718
    Location:
    Australalalalalaia.
    I wrote up a brief summary (inspired by another thread, will find link later) of why benchmarks can be misleading, and first hand experience is key to a video cards performance.

    I hope this can be stickied or make an article here at some point, enjoy

    *********************************************************

    This is a short, brief guide to explain the flaw behind benchmark comparisons of video cards.


    Most benchmarks (excluding the 3Dmark series) get the results based on the minimum and maximum FPS (frames per second)

    This was ok back in the day as no one saw anything wrong with more performance as an end goal - but its flawed.
    Let me give you an example:

    Video card one gives 10FPS as a minimum with 100FPS as a maximum - an average between these is 55.
    Video card two gives 20 FPS minimum, with 80FPS maximum - an average between these is 50.

    In the average review, card 1 is the winner by 5 FPS or 10%. People would buy this card thinking its faster.

    But wait a sec, what about the minimum FPS? 20FPS is slightly laggy, but 10FPS is half that speed, and totally unplayable. Some would correctly argue this card is in fact slower! for the best gameplay card 2 is in fact the winner.


    Some might find this revelation enough, but theres even more to it - what was the FPS doing between these amounts?

    was that minimum of 10 FPS a once off, or was the score poor the whole way through the test, with one huge boost at some point?


    As a basic breakdown, consider the following:
    Think of this as an FPS timeline, spaced 10 seconds apart

    20 25 30 25 30 25 10 10 12 15 35 40 100

    There is a 10 and a 100 in there, but most of them are below 30 FPS (the minimum framerate most gamers are happy with)

    So which card is faster? How do you tell which is the faster card, do we examine minimum, maximum, or average frame rate when deciding what to buy?

    The simple answer: none. Benchmarks need to be updated, and the slightly old game F.E.A.R (First encounter assault recon) had an in-game benchmark with the answer.

    This benchmark listed:
    Minimum FPS
    Average FPS
    Maximum FPS
    Thats all three - but which one to look at?

    How about the other inclusion?
    It showed a percentage of Frames below a threshold.

    What if benchmarks were like this?

    % of frames below 30FPS
    % of frames below 50FPS
    % of frames below 75FPS
    % of frames below 100FPS
    Its far better, because we can see where the card lies in overall performance - does it generally go above 50FPS for smooth gameplay? Does it lag briefly, or is it smooth consistent results? say 95% of all results were above 50 FPS but below 75FPS - thats perfectly fine for all but the fussiest gamers

    If a card gets 1% of frames above 100FPS, but 80% of them are below 30 - that card could have beaten the old system, but will be shown to be crap for most gaming here.



    This is a brief writeup, and only deals with Frames per second. Updates/extensions may come out at a later date, feel free to leave a comment if you want something expanded upon.
     
    Last edited: Feb 10, 2008
  2. candle_86 New Member

    Joined:
    Dec 28, 2006
    Messages:
    3,916 (1.24/day)
    Thanks Received:
    233
    good write up, i tend to care more about minimum than anything and look for reviews with minimum.


    I base it on minimum/average instead of average/high.
     
  3. xfire

    xfire

    Joined:
    Nov 22, 2007
    Messages:
    1,397 (0.49/day)
    Thanks Received:
    193
    Location:
    Hyderabad,India
    remember Bta's suggestion of including video tutorials. Doesn't it make better sense now?
    The reviewers don't need good cams as there are screen capture utilities that record videos.
    This makes for a better experience of how it performes.
    Moreover extra points should be given for cards that don't dip below 30fps.
     
  4. tzitzibp

    tzitzibp New Member

    Joined:
    Nov 17, 2007
    Messages:
    2,225 (0.78/day)
    Thanks Received:
    324
    Location:
    Larisa, Greece
    Mussels has a good point.... benchmarks are designed to measure peaks and average and produce a score that is in fact a simple calculation and not representative of the true gfx output over time!

     
  5. Mussels

    Mussels Moderprator Staff Member

    Joined:
    Oct 6, 2004
    Messages:
    43,547 (10.93/day)
    Thanks Received:
    10,718
    Location:
    Australalalalalaia.
    ^ your quote broke.

    Simplified methods are good, as long as they are accurate.

    I think better FPS would be 60 instead of 50, that being the common Vsync refresh rate many people are locked to.

    <25 (unplayable/borderline)
    25-30 (acceptable)
    30-60 (great)
    60-100 (win!)

    I just wanted this put up because i'm sick of explaining to people how min and max FPS both are irrelevant and useless - every last argument you see online is always about one or the other, and rarely do people even care if the games playable - just how it benches to show off their e-penis.
     
    Last edited: Feb 10, 2008
  6. tzitzibp

    tzitzibp New Member

    Joined:
    Nov 17, 2007
    Messages:
    2,225 (0.78/day)
    Thanks Received:
    324
    Location:
    Larisa, Greece
    I agree with you, friend. I was only describing the pure mathematical method of measuring performance in any case and just applied it to the gfx card output issue. i.e. benchmarking....


    I 'll try to produce a theoretical fps / time graph to show you what I meen.
     
  7. Mussels

    Mussels Moderprator Staff Member

    Joined:
    Oct 6, 2004
    Messages:
    43,547 (10.93/day)
    Thanks Received:
    10,718
    Location:
    Australalalalalaia.
    Go for it. The better people understand this, the better it is for all of us. If we get pics, we could manage to get front page ;)
     
  8. btarunr

    btarunr Editor & Senior Moderator Staff Member

    Joined:
    Oct 9, 2007
    Messages:
    30,521 (10.58/day)
    Thanks Received:
    14,793
    Location:
    Hyderabad, India
    Don't certain benchmark sites use charts/graphs instead? By observing the charts you can pretty much make out how much consistently a card is performing.
     
  9. tzitzibp

    tzitzibp New Member

    Joined:
    Nov 17, 2007
    Messages:
    2,225 (0.78/day)
    Thanks Received:
    324
    Location:
    Larisa, Greece
    yes they do.... e.g. world in conflict bechmark. But the graph stays a presentation and is not trasformed to an actual measurement. That is the problem...

    I can produce (soon to be on) a similar presentation, but is really hard to calculate the actual product of it...
     
  10. tzitzibp

    tzitzibp New Member

    Joined:
    Nov 17, 2007
    Messages:
    2,225 (0.78/day)
    Thanks Received:
    324
    Location:
    Larisa, Greece
    OK, give me about 5 mins...:cool:
     
  11. xfire

    xfire

    Joined:
    Nov 22, 2007
    Messages:
    1,397 (0.49/day)
    Thanks Received:
    193
    Location:
    Hyderabad,India
    I also don't agree with how they test it with highest possible config though wizzard uses midrange most others use the best they can find but sometimes a faster card might perform bad when given a slow cpu when compared to a slightly slower GPU.
     
  12. tzitzibp

    tzitzibp New Member

    Joined:
    Nov 17, 2007
    Messages:
    2,225 (0.78/day)
    Thanks Received:
    324
    Location:
    Larisa, Greece
    ok.. check the graph below and imagine is a 2 minutes benchmark...

    the area under the fps line is the actual result and can be translated as a total benchmark score.
     

    Attached Files:

    • fps.pdf
      File size:
      21 KB
      Views:
      83
  13. DanTheBanjoman Señor Moderator

    Joined:
    May 20, 2004
    Messages:
    10,553 (2.56/day)
    Thanks Received:
    1,383
    Do you have a source that says the math in 3Dmark is this simple (and flawed)? As your example is horrible math. Besides you claim "most" benchmarks work this way.
     
  14. xfire

    xfire

    Joined:
    Nov 22, 2007
    Messages:
    1,397 (0.49/day)
    Thanks Received:
    193
    Location:
    Hyderabad,India
  15. tzitzibp

    tzitzibp New Member

    Joined:
    Nov 17, 2007
    Messages:
    2,225 (0.78/day)
    Thanks Received:
    324
    Location:
    Larisa, Greece
    I never mentioned 3dmark..... But do you have a sourse that says otherwise?

    My math example is fine....

    Never said that.... but I believe so!
    ------------------------------------------------------------------------------------
    all I 'm trying to show here is the fact that is better to take into account the whole result than
    just the peaks and produce an average from that.
     
  16. DanTheBanjoman Señor Moderator

    Joined:
    May 20, 2004
    Messages:
    10,553 (2.56/day)
    Thanks Received:
    1,383
    Tzitzibp,

    You never said the things I claimed you said? Odd, interesting that Mussels did though. Perhaps I wasn't talking to you? The fact that you didn't say the things I mentioned must have given you a clue...
     
  17. Mussels

    Mussels Moderprator Staff Member

    Joined:
    Oct 6, 2004
    Messages:
    43,547 (10.93/day)
    Thanks Received:
    10,718
    Location:
    Australalalalalaia.
    thats why its here to be cleared up first :) 3dmarks math is based on total amount of frames and irrelevant to gaming speed, but part of the score is definately about max FPS.

    Most GPU benchmarks do have only a max FPS, a min and an average - company of heroes, crysis, lost planet - they all have a built in benchmark and thats all they show. Company of heroes is a great example as it gave me over 80FPS average in the tests - but FRAPS in game hovered around 20-30 in most areas.

    3dmark was a bad example.. i dont know why i even mentioned that, as it DOESNT give min/max results. I'll remove it now. In all honesty i only spent 5 min writing that up, mistakes are bound to appear.

    just to clear this up:

    *MY* math *is* horrible. Thats where you guys come in and help sort this out.
     
  18. DanTheBanjoman Señor Moderator

    Joined:
    May 20, 2004
    Messages:
    10,553 (2.56/day)
    Thanks Received:
    1,383
    Well, without any facts about how any benchmark calculates scores there really isn't much we can say.

    Many game benchmarks simply measure how fast a computer can display an x amount of frames or how many frames can be rendered in x time. This says plenty about a system, as the test is identical every time a computer rendering more frames is actually faster. However a single benchmark could never be conclusive.
     
  19. Mussels

    Mussels Moderprator Staff Member

    Joined:
    Oct 6, 2004
    Messages:
    43,547 (10.93/day)
    Thanks Received:
    10,718
    Location:
    Australalalalalaia.
    i gave examples of games, and this is about the ones that give min/max/avg FPS. I gave reasons why they can be flawed, and ideas about a better system - is there a problem with this dan? Why dont you think i'm correct? (aside from the now fixed 3dmark screwup)
     
  20. tzitzibp

    tzitzibp New Member

    Joined:
    Nov 17, 2007
    Messages:
    2,225 (0.78/day)
    Thanks Received:
    324
    Location:
    Larisa, Greece
    ok
     
  21. wolf

    wolf Performance Enthusiast

    Joined:
    May 7, 2007
    Messages:
    5,547 (1.82/day)
    Thanks Received:
    847
    this is how i usually judge my gaming experience, for instance, if i generally game at around 100fps on a game and it dips to 80 thats fine, but if it dips to 25 thats quite poor, and just bacause the 100 is there alot the average will be high, however this is nye on perfect imo.

    % of frames @ <25 (unplayable/borderline)
    % of frames @ 25-30 (acceptable)
    % of frames @ 30-60 (great)
    % of frames @ 60-100 (win!)

    and like you said mussels, fear did this well, even though it was only 3 categories (under 25, 25-40 and 40+ i believe) it still gave a really good idea of where you were at.

    good post, well worth brining this point to the attention of a large audience of gamers/tweakers.
     
  22. jpierce55

    Joined:
    Oct 7, 2006
    Messages:
    1,335 (0.41/day)
    Thanks Received:
    91
    I am in agreement on so what about max fps..... 60fps (correct?) is supposed to be the threshold that we can no longer see the difference in speeds so what if it hits 150, but when it dips lower than 30fps it is an issue.
     
  23. Mussels

    Mussels Moderprator Staff Member

    Joined:
    Oct 6, 2004
    Messages:
    43,547 (10.93/day)
    Thanks Received:
    10,718
    Location:
    Australalalalalaia.
    yes. the biggest concern is not IF it dips below 30, but HOW OFTEN.
     
  24. trog100

    Joined:
    Dec 18, 2005
    Messages:
    4,422 (1.25/day)
    Thanks Received:
    238
    the cpu tends to play a part in how the average is made up.. the faster cpu tends to boost the average a little but it does it by boosting the highs and not the lows..

    so in a sense the grafix card controls the lower part of the average and the cpu the higher part..

    back to the old fashioned balanced system argument here..

    trog
     
  25. wolf

    wolf Performance Enthusiast

    Joined:
    May 7, 2007
    Messages:
    5,547 (1.82/day)
    Thanks Received:
    847
    thats why when you are system building you must choose parts that will go well together, the last thing you want is a brand spanking new shiny boxen with a bottleneck.

    likewise for people with older comps on AGP that buy the 7900GS or x1950 or 3850 for AGP, sure the card is awesome and new and up to date, but an old single core CPU with a gig of DDR wont do you any favours.
     

Currently Active Users Viewing This Thread: 1 (0 members and 1 guest)

Share This Page