What do you care about in a Benchmark?

[Aquinus]

I think the title pretty much speaks for itself but, you know me and alcohol, so this has to turn into something a little more than just a basic question. So buckle your seat belts, it's time to get deep. (...and before anyone says it, I'm sure that's what she said.)

This is a pretty vague question and there really is no right or wrong answer but, generally speaking, a benchmark is a means to gauge performance... the problem is that unless you're benchmarking the exact kind of situation you intend to use your machine for, you're not always going to really get a complete picture of how it will perform for your particular use case. This often leaves us with synthetic benchmarks to give us a glimpse into how hardware will perform in the most broadest or strictest of senses depending on the benchmark. The thing is that how applications are implemented, like using additional passes of AA in graphics or the conversion of floating point math to long integer fixed point math, can make a huge impact on how different kinds of hardware perform (without even getting started with multi-threading effectiveness,) even if the end result is programmatically and mathematically equivalent.

So with that said, we know what benchmarks give us in the most basic of senses. They usually reduce your machine into a number but, is that enough? We do have applications like 3DMark which gives us a basic breakdown into 3 categories but, that doesn't really say what our hardware is good at and what needs improvement. When I'm debugging code and trying to find bottlenecks, I'm not usually using a basic score to figure it out, I'm profiling the stuff I'm working with and finding out what's taking a long time to run. I might just be a mere software engineer but, it seems to me that benchmarks should be a lot more like profiling code, showing you what your bottleneck is and what takes the most compute resources to execute, not just simply how capable your machine is. Focusing on latency over a score that really doesn't mean anything outside how it compares to other machines.

So tell me, what do you think? Are you satisfied with the benchmarks you have now or is there something you've been craving that just hasn't really become a thing? This is something I occasionally think about and I doubt I'm the only person. So, spill the beans. What do you think?

 

[DeathtoGnomes]

I used to run a lot of benchmarks in the past (98,xp,vista) and could never understand how there could so much variance between the same tests over and over and over only to have to get out the pencil and paper to add up and extract an average by hand (omg the strain of holding a pencil!). I quit trusting benchmarks and take them with a grain of salt now. I've maybe one benchmark in the last, well, since win8.0 came out...

 

[TheHunter]

I focused only on gpu benchmarks

And for me its mostly visual fidelity and then performance, sometimes both if it looks just about average and why does it run so bad..

I also monitor extra with MSI AB, gpu %, cpu load % per core, to see the bigger picture if its single threaded or very multicore and heats cpu more..


Personally I would like to see proper next gen benchmarks, so far I saw only U4 Elemental, Infiltrator, UnityE Adam tech demos,. Sometimes I really wonder if Cyberpunk2077 will look so CGI, apparently they target such gfx as in this video.

 

[Aquinus]

I used to run a lot of benchmarks in the past (98,xp,vista) and could never understand how there could so much variance between the same tests over and over and over only to have to get out the pencil and paper to add up and extract an average by hand (omg the strain of holding a pencil!). I quit trusting benchmarks and take them with a grain of salt now. I've maybe one benchmark in the last, well, since win8.0 came out...

Being reproducible is a big one in my book and that's not just times but, the result because if the result isn't what you're expecting it to be, your machine is probably unstable if you're overclocking. There are a lot of things that impact this though but, a good benchmark shouldn't have too much variance in its results (time wise,) in my opinion.

I focused only on gpu benchmarks

And for me its mostly visual fidelity and then performance, sometimes both if it looks just about average and why does it run so bad..

I also monitor extra with MSI AB, gpu %, cpu load % per core, to see the bigger picture if its single threaded or very multicore and heats cpu more..


Personally I would like to see proper next gen benchmarks, so far I saw only U4 Elemental, Infiltrator, UnityE Adam tech demos,. Sometimes I really wonder if Cyberpunk2077 will look so CGI, apparently they target such gfx as in this video.

Would you say that sensor values are a worthy thing to care about during a benchmark, such as temperatures when talking about results? Do you think that should be reported by a benchmark in some way similar to performance? I hadn't really thought of that and I think that's an awesome idea.

 

[Melvis]

Hardware utilisation, as in a benchmark to use all of the hardware that is available and at the same time.

eg: Unigine is great for GPU's but useless when it comes to CPU usage.

 

[FR@NK]

That reminds me, I still have a 2009 vintage in the wine cellar I need to crack open sooner or later. It should bench pretty high.

 

[CAPSLOCKSTUCK]

I am happy with Heaven for GPU. Mainly because folks here use it.

I use R15 for CPU because hardly anyone uses it and it means i can hold a few records with my Xeons.

 

[Aquinus]

That reminds me, I still have a 2009 vintage in the wine cellar I need to crack open sooner or later. It should bench pretty high.

This is clearly the standard at which all benchmarks should be held to. I feel like I've been outdone. I clearly need to get craftier.

I am happy with Heaven for GPU. Mainly because folks here use it.

I use R15 for CPU because hardly anyone uses it and it means i can hold a few records with my Xeons.

Is it just about records or is there a goal other than just getting a high score?

Hardware utilisation, as in a benchmark to use all of the hardware that is available and at the same time.

eg: Unigine is great for GPU's but useless when it comes to CPU usage.

I like the idea of showing how balanced a system is for a given workload, like games.

 

[alucasa]

I mostly don't care about general benchmark.

When I read a review of something, I focus on its relative strength compared to its competitions around it. Outside of that, I don't bench or care. I don't record fps unless I clearly feel shruggish.

For me, CPU is far more important than GPU since my main activitiy is rendering. I find Cinebench very accurate on CPU's rendering power, so I keep that on HDD.

 

[qubit]

I'd say I'm mostly satisfied with benchmarks overall.

The only improvement I'd make is to have every major game have a benchmark mode built in. For example, I've just bought Deus Ex: Mankind Divided and was pleasantly surprised that it has a benchmark mode available through the main menu* which runs through an identical scene each time. Using this, I was able to measure the difference between DX11 and DX12 performance. DX11 gave an average framerate of 70 while DX12 an average of 82 which is quite a big difference and visibly smoother. My current hardware is in my specs.

*Why some games make the benchmark difficult to get to by running a secret exe or fiddling with configuration files using secret settings is a mystery to me.

 

[SniperHF]

I'm pretty happy with the synthetic benchmarks available and contrary to one of the usual refrains of checking game performance closer than benches, I find them just as accurate so long as you have a frame of reference and comparable test beds. Certainly I wouldn't mind seeing more games with bench tools but there are enough to get by right now.

One thing I dislike about 3dmark though is when searching for other user results in the list you always end up with the highest OC cards at the top and since GPUs generally scale pretty well you have to really dig to find more comparable results for mild OC's or stock settings. It would be nice if you could filter by clock rate.

That mainly goes to what I use benches for, which is gauging how hardware works as an upgrade. If I read a review with a bench in it that's great for a 1:1 comparison, but I want to know how the latest video card is gonna run on my 2500k or my brothers G3258, not on the HEDT test bed review sites are using. So I find the 3dmark search a pretty neat way to get a gauge of how much of a performance hit I'm going to take compared to the same card on that test bed.

Or a similar example is what I did just on the black friday sales. For $169 I grabbed one of those Power Color RX 480's to replace my R9 280+7950 crossfire. Benchmarks were invaluable in determining whether it was worth it or not and so far the benches done by reviewers have been right on the money with my own testing. The only notable exception is the continual march of driver updates making the comparisons harder.

 

[Solaris17]

No I am not happy.

I am more into system tests now and stress tests. I like you think about these things but my scale might be a bit different.

I honestly wish futuremark or OCCT or another benchmark manufacturer would make the same suite available to linux, OSX and Windows. and while I understand obviously that this cant necessarily work on a 1:1 ratio I wish there was a "full battery" of comparable benchmarks between platform types (per platform type, not talking osx vs ms vs linux) that worked on all of these OSs.

It bothers me on a profound level that this isnt available.

 

[AsRock]

Personally nothing really, benchmarks from steam for example 3D Mark is useful for getting the DX updates and such other wise they mean pretty much nothing to me.

I like OCCT for testing.

 

[slozomby]

mostly i use userbenchmark to make sure I'm within expected performance of the components when i finish a build.. sometimes it'll catch something out of whack. I'm not benching to try and squeeze another 2 fps out of <insert game here>.

 

[TheHunter]

Being reproducible is a big one in my book and that's not just times but, the result because if the result isn't what you're expecting it to be, your machine is probably unstable if you're overclocking. There are a lot of things that impact this though but, a good benchmark shouldn't have too much variance in its results (time wise,) in my opinion.

Would you say that sensor values are a worthy thing to care about during a benchmark, such as temperatures when talking about results? Do you think that should be reported by a benchmark in some way similar to performance? I hadn't really thought of that and I think that's an awesome idea.

I think it would be good by cpu focused gfx benchmarks., so you know what to get by similar demanding games if not monitoring it.

For me top game cpu benchmarks , also cpu OC stability by testing; LostPlanet2 - testB, ResidentEvil5 -fixed dx9 (jobthread=8 by both mt engine games), Dying Light, X-rebirth, GTA5, Rift, Quantum Break - steam, Forza Horizon3.


Or new Ashes of Singularity, dx12 mode Crazy present 1080p
I was shocked when I tried cpu benchmark, all 8 threads to 100% and heats up to 70C.., never seen a game benchmark stress this cpu in entire 3.5yrs owning it , ok LP2 dx11 can by end of benchmark when that monster jumps in water, but this thing was like that entire time..




And for example, Im capped at 52fps by gpu benchmark, no aa or 2xmsaa or 4xmsaa, all same 52fps and it looks like gpu limited, but then why get same 52fps without msaa, so its cpu then.. very strange benchmark lol

 

[Ithanul]

Being reproducible is a big one in my book and that's not just times but, the result because if the result isn't what you're expecting it to be, your machine is probably unstable if you're overclocking. There are a lot of things that impact this though but, a good benchmark shouldn't have too much variance in its results (time wise,) in my opinion.

Would you say that sensor values are a worthy thing to care about during a benchmark, such as temperatures when talking about results? Do you think that should be reported by a benchmark in some way similar to performance? I hadn't really thought of that and I think that's an awesome idea.

I agree on temperatures, they need to state that early in a benchmark, not at the end or not at all (heck, even fan speed and noise level).
Also there is difference in temps of open bench, open side, high air flow case, and low air flow case. These latest cards respond greatly to temps, especially when pushing them hard.

I usually only see benchmarks as ball park figures, majority of the time I don't even bother looking at them. Every system is going to respond differently considering the variety of hardware configurations and applications used from machine to machine. There is also difference in running these applications under Windows or Linux.

 

[Mussels]

the key is to keep in mind that benchmarks are often not comparing apples to apples.

I use them to compare before and after results, not to compare between two different products (EG, before and after re-pasting a CPU or replacing a cooler, to see if thermal throttling went away, or to see if an OC is faster)

 

[Grings]

I generally just use them to make sure everything is running at comparable speeds to similar gear other people are using, and testing the stability at new overclocks

Too many benches in the past have run better on 1 piece of hardware than another, when the situation is reversed in real world situations

The current one that annoys me is seeing crap tlc hard drives look ultra fast using part of their memory in slc AND using what is basically a ramdisk to show bullshit fast speeds in benches, despite being way slower than old 2010 era drives in real use

 

[Ebo]

I dont use them at all, simply because I dont care

 

[hat]

Most of the benchmarks I look at are in our own video card reviews. I use them to get an idea of how card x stacks up to card y in relative performance and power consumption.

I also take a look at the Intel CPU benchmarks that come around every now and again... the ones that pit two processors from different generations against eachother at the same clock speed (and then consider what the chips are capable of clocking up to). This way I get a rough idea of what happened with raw performance across generations.

 

[Loosenut]

I couldn't give a rat's a** about benchmarks. As long as my system allows me to play my games, I'm good.

I'm here to learn and when possible offer advice, not compare e-peens. For those who put value in those things

 

[PLAfiller]

I want them to be interesting. Catzilla had an actual story behind and awesome music. More often than not, what @Mussels said. I compare before and after changes on the same PC for example.

 

[Mussels]

i just used a benchmark to demonstrate to somehow that their PC was upgraded, and worth it.

40 Gflops in an 8 core, vs 103 Gflops on a 4 core i5.

Pointed out that meant each core was 5x faster, and his games are single threaded. He got the hint.

 

[Toothless]

i just used a benchmark to demonstrate to somehow that their PC was upgraded, and worth it.

40 Gflops in an 8 core, vs 103 Gflops on a 4 core i5.

Pointed out that meant each core was 5x faster, and his games are single threaded. He got the hint.

What benchie program?

 

[FYFI13]

I'd love to see some benchmark that includes all:

- CPU single core
- CPU multi core
- Memory throughput (read, write)
- Storage performance (system drive: read, write, latency)
- Graphics performance
- Combined score so we can compare our systems