Honestly, yes.
Since we have the continuation of shitty car analogies (car analogies are ok, these just suck), I am looking at picking up Bel Air. Yay, I found one! But shoot, it has the straight six. Doesn't really make much sense to leave it "numbers matching" when it is going to be a big, slow, pile of turd - does it? Or does it make sense to actually break this silly notion of "numbers matching" and implement a change that has tangible performance difference?
Now, your case is a little different in that your "matching numbers" shenanigans is actually detrimental to your own cause and may cause you to miss your goal of 5ghz. On top of that, it could even start to rain on your parade of maximum single core goodness. And to top all that shittyness, it even cost you more money to get less cooling performance. And one more thing, the whole reason you kept this case was because you liked the business look (and the sound deadening) but now you aren't even going to have the side on. I am not exactly sure what the business look is but it surely isn't a bunch of unsleeved wires in a rats nest. There are plenty of cases that have clean lines, no bling, and a reserved, windowed side panel to show off those aesthetics.
Now, if it were me (which it has been made abundantly clear it is not), I would have bought a better performing cooler for less money and then colorized it to fit my "theme". Now you can have "numbers matching", better cooling performance, less noise, and have a better shot of hitting your other goals that you said are a priority over everything else.
Edited for spelling mistakes.
