Comparative Linux Gaming Benchmarks

[silentbogo]

Recently there has been lots of fuss about SteamOS and the state of Linux gaming performance in general. So, I thought it was a perfect time to do some obsessive benchmarking.

I prefer to play video games on Windows and despise SteamOS for a mess that it is, but I still have high hopes for Linux gaming, especially if you look at all the progress made in the last 2-3 years. Last time I attempted something like this over 7 years ago, but at that time there weren't that many multi-platform games except Id3-based arena shooters and a few RPGs.

Initially I wanted to test everything under SteamOS, but my ASUS PCE-N53 WiFi adapter and lack of SteamOS pre-installed tools/functionality to compile WiFi drivers forced me to use Ubuntu 15.10.

I'll start with a simple set of GPU benchmarks and then move on to more practical gaming benchmarks (tomorrow or later this week). This process is time consuming, so if you are interested in this topic, you'll have to be patient and periodically check this thread for updates.

I am using my primary x58 rig, which served me well for over 6 years and after a few upgrades it is still relevant for gaming and other stuff

TEST SETUP:
CPU: Intel Xeon X5650 @3.3GHz
MoBo: ASUS Rampage II GENE
RAM: G.SKILL Ripjaws DDR3-2133 @ 750MHz CL7
VGA: MSI GTX 750Ti Gaming [2G] @ stock
SSD: SanDisk Extreme II 240GB
HDD: Seagate 7200.14 1TB

OS: Windows 10.1511 , Ubuntu Gnome 15.10 (kernel 4.2)

Ubuntu 15.10 is a fresh install with NVidia 352.6 drivers. Windows 10 Pro is also a fresh install with NVidia 359.00 drivers.



NOTE ON GLXOSD
Due to some unknown bug I am getting some crazy stutters while running GLXOSD overlay. This, in fact, significantly impacts the game performance, even though most of the time FPS numbers were fairly accurate.

Here's an example from Borderlands 2 at low settings. Periodic stutters cause these ripples of very slow frame draws: 19-20 times longer than average Δt.

In order to compensate for these stutters I made some adjustments to GLXOSD framelog processing. Instead of using raw numbers and GLXOSD online analyzer, I wrote a script which filters out all abnormal frame drops and calculates average FPS based on remaining numbers.

I know this is a crude way to fix things, but unless I find better software to log my FPS, it is our only option.
If you don't like it - eat a sandwich. Life always seems better after a nice sandwich.

SYNTHETIC BENCHMARKS

Spoiler

GPUTest v0.7.0

Full chart with numbers here

As you can see in synthetic OpenGL benchmarks the difference is not just minimal, but is pretty much negligible.

Unigine Heaven Benchmark v4.0

To make this bench more practical I've used a custom preset with 1080p resolution(Fullscreen, Medium detail, Moderate tessellation) in addition to standard settings in Basic and Extreme presets. To spice things up a bit more I've also tested OpenGL performance in Windows 10.

Full chart

Valve Titles

Left4Dead 2

Spoiler

As any other Source-powered title, this game is a bit outdated, so I've only decided to test this game in two presets: maximum all around+MSAAx8 and the same thing + no AA.
It was said way too many times that L4D2 runs faster on Linux, but apparently my rig decided to think otherwise. Here are the benchmarking results:

At MSAAx8 Linux port was approximately 5% faster. With AntiAliasing disabled it was 10% slower...

Portal

Spoiler

Pretty much same results as with L4D2. Some variance can be attributed to my somewhat flawed benchmarking technique (* see the GLXOSD notes), but it is consistent.

Half-Life 2: Lost Coast

Spoiler

Tried this one just for fun, and because it has a built-in stress test. All settings maxed out at 1080p. Textures are set to Very High, reflections set to Reflect All. I do not see a reason to run it at lower settings, because this is a very-very....very old demo.

219 vs 164 average FPS... Seems like Valve forgot to optimize this one...

Dota 2

Spoiler

Just noticed that Valve implemented OpenGL support on Windows for Dota 2, which means we can add it to our comparison.

Overall the performance is very-very similar, but if you dared to install OpenGL support through DLC, you will suffer from a nasty 15-20% frame drop. I've also noticed that in OpenGL rendering mode on Windows my frame rates are jumping from 50 to 110, in contrast to smooth 85-110 in DX mode. Linux port also kept my FPS smooth.

Team Fortress 2
...Coming soon...

 

[silentbogo]

GAME BENCHMARKS

In most cases I will try to use built-in benchmarks.
If a game does not include a built-in benchmark I will use FRAPS on Windows 10 and GLXOSD on Ubuntu 15.10.

Metro: Last Light Redux

Spoiler

This is one of my favorite games and represents everything that S.T.A.L.K.E.R. should've been many years ago. Since its early port to Linux, Metro was pronounced by many as the savior of Linux gaming due to its immaculate execution and excellent performance.

I've only tested Meduim and Very High detail. I do, however, have results for Low @ 1080p on Windows, but forgot to run a bench in Ubuntu for some reason, so I'll just add it later.

Here are the presets I used:

Med:
- 1920x1080
- Medium Quality
- SSAA off
- AF x4
- Tesselation: off
- Motion Blur: off
- Physics: off

Ultra:
- 1920x1080
- Very High Quality
- SSAA x2
- AF x16
- Tesselation: Normal
- Motion Blur: Normal
- Physics: off

Somehow Ubuntu sprinted ahead of Windows 10. It might be the AA issue @Ikaruga mentioned, but it does not attribute to a split in Medium preset, which has AA off on both platforms. Might need some re-testing or configuration checks, but several runs on these settings gave the same result...

The Talos Priciple

Spoiler

This game gave me the most trouble. Initially my results were unpredictable, but I got it sorted-out (sorta...)

For this game I am using a 1080p resolution and forcing 1080p rendering surface with AntiAliasing off.
Except the abovementioned options, all performance menu items are set to Lowest, Medium and Ultra for each corresponding benchmark. Once again, Windows 10 is tested in both OpenGL and DirectX11 performance.

As you've probably noticed, OpenGL framerates in both platforms are very close. DX11, however, is 40-60 percent ahead on Low and Medium settings. Yesterday I had the opposite picture: forgot to disable AA when switching to default settings and my framerates went all over the place. Linux scaled well for x4 (~4-5FPS drop), but Windows went haywire (half the performance of Linux).

Alien Isolation

Spoiler

This game does not have a built-in benchmark, so I had to suffer through all the pain that GLXOSD brings to new users. While the utility works and does what it's supposed to, it has a serious problem on my PC: I do not see any huge deviations in framerates and GLXOSD framelog, but while I'm playing a game even without running a benchmark, I get some annoying stuttering/freezing every 2 seconds (even on some older and less power-hungry titles like Nexuiz and OpenArena). As to whether this problem affects recorded framerates we will find out soon.

Another thing that bothers me is that in Windows, Alien Isolation relies on Windows Experience Index to disable certain game settings regardless of whether user wants it enabled. Dunno if the same feature is implemented in Linux port, but that's just another thing to consider. We are testing games as end-users, so I will not do any tests with options not available in the game menu. If performance suffers - we'll just blame it on devs!

All presets use 1080p resolution with AA disabled. The rest is as follows:

Low:
- All detail Low
- AF x4
- All advanced features: Off

Medium:
- All detail Medium
- AF x16
- All advanced features off

Ultra:
- All detail ultra
- AFx16
- All advanced features On

On the lowest graphics settings Ubuntu has problems with framerates. It is barely distinguishable from Medium detail FPS and is behind Windows 10 by almost 40%!

Penumbra: Overture

Spoiler

The first creation of Frictional Games and the first horror-adventure since Silent Hill 2 that I actually liked.
This game is still buggy as hell and mostly abandoned, so I was really surprised to see it working on Win10.
Had it in my library for a while and figured it's worth testing just to satisfy my curiosity.

Testing at max available settings with 1080p resolution and VSync off.

Woops..... With VSync off we are still locked at 60 FPS. Both OSes suffer from identical screen tear but otherwise look the same.

Borderlands 2

Spoiler

All tests were performed at 1080p with physics set to Low. Framerate is set to Unlimited.
Presets are as follows:
Low:
- FXAA off
- AF off
- DoF off
- Ambient Occlusion off
- All detail settings Low
Medium:
- FXAA off
- AF x8
- DoF off
- Ambient Occlusion off
- All detail/fading settings Medium
High:
- FXAA off
- AF x16
- DoF on
- Ambient Occlusion on
- All detail settings High

As with some other titles we can see a common trend of under-performing on lower settings. The game is still very playable even at the highest available graphics settings and Ubuntu even beats Windows 10 by a few FPS.
Also, I'd like to point out that in both Ubuntu and Windows we have a very small difference between medium and low settings framerates, which might have been caused by an underused CPU or an insignificant differences between Medium and Low textures... Either way, even on the lowest settings Borderlands 2 look excellent.

The Witcher 2: Assassins of Kings

Spoiler

Very demanding game that makes my GTX 750Ti work very hard for each frame. Tested with standard graphics presets @1080p with VSync off. All tests were performed at the first location ("By the King's Will") : approximately the same route around the camp with 120 seconds of framerate logging.

Middle-earth: The Shadow of Mordor

Spoiler

That's one though cookie to chew on. Out of all games in this lineup Shadow of Mordor showed the worst performance ratio between Windows and Ubuntu, in addition to some questionable issues with settings themselves. On my tiny and quiet GTX 750 Ti it is barely playable at 1080p even at lowest settings...
SoM was ported by Feral Interactive, the same company that did a Linux version of Alien: Isolation, dozens of OSX ports and several other Linux ports.

Once again, I am using only standard presets for graphics settings with the exception of Autoconfig. I only tested automatic configuration only to see if OSes are treated equally and whether or not Devs and a Publisher were aware of performance issues before release.

What I noticed off the start is that Windows version of SoM had initially set my rendering resolution to 100%(1920x1080), while Linux version started with a 50% rendering surface(960x540). After changing the video settings to 100% I tried an autoconfig on both platforms.

Here's what we got (Windows->Right, Ubuntu->Left):

It is not a secret that I really hate WBGames division, but for this review I wanted to hold on to my arsenal of rotten tomatoes... Unfortunately, when I see this - I just can't. Apparently, instead of giving time to Feral to finish the Linux port of Middle-earth, WB rushed it to Steam as is. In Ubuntu automatic settings are significantly lower on the same system, because this is the only game configuration that lets you play somewhere near-adequate 60FPS (...sounds familiar?).

In addition to all of the above I tried to find the optimal quality setting at which the game may act and look normal in Ubuntu, but discovered only more strange things:
- DoF only works on Ultra settings. Otherwise it is ignored and does not affect FPS at all
- Same thing goes for Camera Blur
- At any settings with lower rendering resolutions I get horrible FPS jumps from 0 to 60+ FPS

Anyway, back to our testing:

At Ultra settings we get a huge -60% FPS drop on linux, because we've definitely exceeded a 2GB frame buffer of my GTX 750Ti. All other settings show a constant 30% drop in framerates for Team Orange

P.S. It actually kinda surprised me that SoM even worked on Windows 10. The last few times since the Bright Lord DLC came out with its patches, I could not run this game on Win7 for more than 2 minutes. This time even Linux gods were nice to me and allowed to test this port with only 1 crash.

Bioshock: Infinite

...coming soon...


RESERVED FOR SUMMARY/CONCLUSION?

 

[Ikaruga]

Linux has a bug with AA at this moment, so you might want to disable it.

 

[silentbogo]

Thx. I ran most of the stuff so far with AA disabled, but it seems to work fine in Metro: Last Light.

Having some weird results with Talos Principle on Windows. On medium settings I only get half of the performance of Linux port with the same settings (in both DX11 and OpenGL rendering mode). Only Ultra preset is somewhat equal in DX11, but slower in OpenGL.

 

[CAPSLOCKSTUCK]

sub'd, i will enjoy this one.

 

[Liquid Cool]

Sub'd +1

 

[silentbogo]

Added Metro: Last light and the list of games I want to test in the near future.

If anyone, by the chance, has a copy of Borderlands 2 and is willing to donate it in the name of science - please shoot me a PM.

 

[silentbogo]

Added Talos Principle and Alien: Isolation. More game benchmarks coming soon.

 

[Kursah]

Thanks for taking the time to do this.

 

[RejZoR]

The main problem with Linux is that it's just too fidly and too unnecessarily overcomplicated and lacks any kinds of standards. That's why it just never catches on. The other day I installed Kubuntu on my laptop with AMD E-40 and I was getting weird artefacting and graphical glitches. So I wanted to install drivers. Oh boy was I optimistic. Downloaded sseveral versions that were offered to me and neither even wanted to install. So I just gave up and installed Windows 10 back. Despite dumb forced driver updating in 10, at least it gets installed and isn't artefacting like mad.

 

[silentbogo]

The main problem with Linux is that it's just too fidly and too unnecessarily overcomplicated and lacks any kinds of standards. That's why it just never catches on. The other day I installed Kubuntu on my laptop with AMD E-40 and I was getting weird artefacting and graphical glitches. So I wanted to install drivers. Oh boy was I optimistic. Downloaded sseveral versions that were offered to me and neither even wanted to install. So I just gave up and installed Windows 10 back. Despite dumb forced driver updating in 10, at least it gets installed and isn't artefacting like mad.

Not today. Actually I had approximately equal amount of problems on Ubuntu 15.10 than I had on Windows 10. All of them were caused by my ASUS PCE-N53 wireless adapter.

Ubuntu does not have a working driver for this Ralink-based adapter, which means every time I do a fresh install I have to compile and install driver manually. Same goes for every kernel update (just happened last night BTW).
Windows 10 on the other hand works flawlessly, but occasionally drops the connection or drastically loses speed(~20-30KB/s), which requires adapter reset.

To get a gaming system up and running all you need to do is:
1) Have some free space on your HDD/SSD
2) Install the OS alongside windows with few clicks. If you want to get rid of swap partition, it will require less effort to remove it than disable swap file in Windows.
3) Install proprietary drivers through Additional Drivers menu. Works automatically, does not require any supernatural abilities (I still do it with apt, though).
4) Install Steam by downloading a .deb package off the Steam website and double-clicking on it.
5) Play

In my opinion Linux is not catching up only because of these stereotypes from 90's. A person with no OS preference will not feel more scared of Ubuntu or Linux Mint any more than Win7/8 or 10.
The only factor in his/her choice is a "friend who knows a bit more about computers", who will most likely say "Hell no! Linux is hard and not for neanderthals like you", because this friend tried Slackware or Mandrake a decade ago and did not like it.

In regards to your laptop experience: Kubuntu 15.xx is quite heavy. KDE 5 has some documented glitches, but I personally have never encountered any of them (used Kubuntu 14.xx and 15.xx on this rig and my ASUS R500V laptop since early 2014 and replaced with Ubuntu Gnome just few days ago).

For a low-end CPU, like your AMD E-series you should try Lubuntu or Xubuntu. Much smoother experience and needs a lot less resources. I've recently fixed a Lenovo laptop based on E1-2500 APU and it was so slow and sluggish, that the old Core2Duo T2500 with Radeon X1300 onboard felt a lot faster under both Win7 Basic and Ubuntu with Unity... Definitely not a good platform to have first good Linux experience.

 

[RejZoR]

It's not a 90's stereotype, it's a reality in 2015. How often do you have to use CMD in Windows to change something? Every freaking guide online for Linux is a noodle of commands. In Windows, only time that exists it's as a shortcut because you need to write less. On Linux, it's the only option.

And the fact that Linux doesn't have unified installers is just crap. On Windows, run that EXE and it'll do what it's suppose to do. On Linux it's always some weird package that may or may not do anything whatsoever. And then you have like 6 installers for what is essentially the same thing with a different GUI. No, just no. I've always loved Slax because I had it on those mini CD's, I loved Ubuntu because of the same portability, but as an every day OS, just too damn clumsy. I still give them a go here and there just to see it hasn't really changed much and then I reinstall WIndows again... Which is a shame because it would be nice to have a free Windows alternative that actualyl works.

 

[RCoon]

@silentbogo appreciate what you're doing, think it's great information that definitely needs to be out there.

My only qualm is with the charts. Perhaps make the font a little bigger? It also might help if you add finer lines regarding FPS figures. It's probably worth adding the actual figures at the base of the column so people know what they are - this might mean you have to make the bars a bit thicker.

(Also, when this is all done, it might be worth taking 10 minutes to make the second post with all the results a bit neater and tighter overall. Make it easier to read for people who may come here from google results.)

Keep up the work!

 

[silentbogo]

Thx @RCoon! I am using an online charts builder and unfortunately TPU does not allow embedded iframes and the builder does not have an option to display values next to bars. I am providing a link to each chart for now, but once I find a suitable tool, I will re-do all charts with FPS numbers in more accurate and presentable way. Worst case scenario: I'll just do it in Libre Office. Acquired data is not going anywhere, so redoing the data representation won't be that hard.

Still having difficulties with benching on Linux: GLXOSD really does cause the stutter and significantly drops FPS.
You can see it in the framelog for Alien Isolation on the chart below. A lot worse in Source-based games: framerate drops by 40-50% in Portal and Left4Dead and screen freezes make almost any game unplayable with OSD running...

I've tried Alien Isolation with just a Steam overlay on Ultra settings + SMAA T1X and got a solid 45FPS average, 31 min and 67 max. No freezes, no stutters.
Makes it very much playable on max settings with AA off, even though my PC is "old" and "outdated" comparing to all the GTX980 monsters.
There is no other viable alternative for FPS monitoring that I could find [yet], so I guess I'll have to look for more games with built-in benchmarks instead.

 

[silentbogo]

Thanks to @jboydgolfer and his awesome Thanksgiving Giveaway we now have a copy of Borderlands 2!

I've re-edited my charts to be more user-friendly and decided to cover only Average FPS from now on, because this stuff only distracts from the main point of this review/research.

Added several Valve titles and few other games. Saving Borderlands 2, Bioshock: Infinite and Witcher 2 for last ))

 

[Liquid Cool]

Re-edited my charts to be more user-friendly.

I've decided to cover only Average FPS

Thanks SilentBogo....

I appreciate the hard work you're doing here....

Looking forward to the Witcher 2 and Shadow of Mordor comparisons, these are on my short list. Borderlands, Metro, and Bioshock I've already loaded without problems.

I've gotten a few of my Origin/Windows/GOG games up and running in SteamOS with little to no problems. Someone might want to start a how-to thread on this down the road.

Best,

Liquid Cool

 

[silentbogo]

Witcher 2 and Shadow of Mordor are coming soon. I have to download all these games twice, so it might take some time even with my 100Mbit/s connection.

I think this month will be the very first time I got past 1TB of Internet traffic

 

[silentbogo]

Moved things around again, because I am exceeding a 10 images per post limit.

Added Dota 2 and Witcher 2!!!

 

[broken pixel]

This thread makes me want to install a distro on my R4BE on a spare SSD, maybe fake RAID0? Thanks for taking the time to test & make graphs.

 

[silentbogo]

Added Shadow of Mordor benchmark results.
Only one more game to go!

 

[CAPSLOCKSTUCK]

Big up @silentbogo working hard enjoying himself.




I entered " other" in the poll.

I use Ubuntu in a 2 P cruncher system ( see specs 2ThugXeon) I am almost a total novice, however this afternoon i am taking a massive leap and will attempt to read a HDD on that system that W 7 wont recognise.

I literally have Psensor and BOINC on the Ubuntu desktop and they were already on the HDD
i am looking forward to it, i enjoy a challenge and i know it wont be difficult (gulp)

 

[Blue-Knight]

I voted "Exclusively".

If I am going to play any games... I'm currently with no time or interest for the foreseeable future.

Spoiler: Off topic

Good, now we just have to slap a "US$ 119.99" label on it, implement an activation system, and call it "Ubuntu Ultimate".

Thank you for doing the time consuming work, silentbogo!

 

[silentbogo]

I think I might also add some Unreal Engine 4 tests. I've downloaded some demos from Unreal Engine website and they look awesome in both Linux and Windows.
Also I already have some data for linux-native games like Xonotic and OpenArena.
Not sure if I can make it all in one day, but definitely should be ready for weekend.

 

[m0nt3]

It's not a 90's stereotype, it's a reality in 2015. How often do you have to use CMD in Windows to change something? Every freaking guide online for Linux is a noodle of commands. In Windows, only time that exists it's as a shortcut because you need to write less. On Linux, it's the only option.

And the fact that Linux doesn't have unified installers is just crap. On Windows, run that EXE and it'll do what it's suppose to do. On Linux it's always some weird package that may or may not do anything whatsoever. And then you have like 6 installers for what is essentially the same thing with a different GUI. No, just no. I've always loved Slax because I had it on those mini CD's, I loved Ubuntu because of the same portability, but as an every day OS, just too damn clumsy. I still give them a go here and there just to see it hasn't really changed much and then I reinstall WIndows again... Which is a shame because it would be nice to have a free Windows alternative that actualyl works.

I have to use CMD prompt all then time in windows while doing repairs. Winsock and IP reset, flushdns, sfc /scannow, dism.exe for when sfc cannot fix corrupt files and of course chkdsk and it easier to find the MAC address in command prompt as well

I voted other, because I would prefer to play my games in linux, however not all the games I want to play are available in linux. If Battlefield would get ported over and run good. Done, I would leave windows for good.

 

[R-T-B]

The real "problem" with linux (if you can call it a problem) is people try to used closed source drivers with a moving target kernel, AKA it's OSS kernel that can change on any given day of the week. Windows doesn't see kernel changes often at all so closed source drivers play well with it. Linux? If the driver was not compiled against the kernel, it usually won't run. End of story. With all the different configs out there, this makes the usual windows users "lemme have a driver" search end very badly if they don't know how to compile against an existing kernel.

This would not be a problem if drivers were fully OSS. But they aren't, at least not for things that matter like 3d acceleration that works well. And so the command line is required to sort of mold the open source parts of the closed source driver around the kernel. It's vexing to someone coming from windows, and leaves linux better suited to techie types if not a well known-distro with a driver package manager of some quality.