Now that my first CPU review is posted, I'd like to ask the community for thoughts on methodology, benchmark selection etc.
Well.. we've already exchanged a few posts, but I'm glad this thread was created.
Whatever benchmarks you'll use, I do have 2 general suggestions.
I. Let's keep stuff fairly
mainstream, accessible and - ideally - make at least some of the benchmarks
downloadable. This would make it possible for us to check how our machines compare with the newest stuff.
As such, I think you should consider free or paid mainstream software only, so - for example - not things like Matlab, AutoCAD, Photoshop etc.
Moreover, you could at least think about starting a "3rd party" community results' database. I'm not saying it's essential, but could work as an extra resource next to results in controlled environment.
II. I'd love to see some
more statistics in the results, but keeping it as simple as possible: FPS quantiles for gaming, number of samples and dispersion in time-measuring benchmarks etc.
As for the benchmarks, here are some possibilities:
1) IMO a serious
Excel test is a must. Access could be incorporated (as today it's more an Excel back-end for data storage than a sensible database).
Germany's population is 80 mio, so it's very likely that just in your country there are more people looking at Excel for >4h a day than those doing serious 3D rendering all over the planet.
2)
Database (ideally both SQL and NoSQL). This should include some advanced analytic coding and scripting (as basic data manipulation tests the storage speed more than anything else). You'll need a DB with decent scripting for that (PostgreSQL?). You could consider SQLite as well, because it's hugely popular as embedded db in a lot of software.
3)
Numerical computation / data analysis - huge amount of software to choose from: R, Octave, Python etc. This could include generating 2D images as a result (a complicated plot/graph can take a few seconds, so it shouldn't be underestimated
).
4)
Virtualization - ability to work with multiple VMs (each with some CPU-heavy tasks).
5)
Everyday tasks / productivity - running a lot of things at the same time rather than separate. Such benchmark tests a lot more than just multi-thread performance. I'd go for a movie + compression + firefox + file copying etc. I'd exclude gaming, because it doesn't offer stable CPU load and would ruin the results.
6)
Antivirus scan - totally ignored in many reviews for a reason I don't understand. A weekly full scan is one of the most irritating things I do (even worse than off-site backup).
I'm not suggesting video encoding and rendering for a simple reason - some very good benchmarks are already available.
+
=jboydgolfer
