So very true. But its also a standards / luxury thing. My cue for an upgrade (for the CPU) is when I see cores maxed out, resulting in in-game frame drops or stutter. So it may be fine for 'most games' but when a CPU cannot do ALL tasks I throw at it perfectly, I start looking at what's on offer. In my book, a CPU is where it all starts in terms of smoothness. That is also why I didn't jump on first or second gen Ryzen just yet - its good, but its not perfect. And that is why Ryzen needs those higher clocks, too.
Games have become progressively more demanding on the CPU the past few years, there is additional load (and benefit from high thread count) due to the inclusion of DRM like Denuvo, and consoles have pushed the boundary to 6-8 cores for scaling. You mentioned unified shaders, the useful thread count shift I consider to be a somewhat more gradual, but equally important move - we've been whining for years that devs should utilize multiple threads more. Not too long ago, most gaming was exclusively single threaded - and it sucked monkey balls. Today, every half-serious PC gamer considers a high refresh screen, effectively doubling the potential benefit of more CPU power as well.
On top of that development we've seen IPC and frequency progress come to a near-complete halt, as well. To get more performance we simply NEED to move further up the product stack, because GPUs did get 30% faster every gen for the past decade. My previous rig was a 3570K 4C/4T that slaughtered everything at a mere 4.2 Ghz - the first game that killed the CPU was Crysis 3, showing great benefits from HT. Now I see most games still benefit from 4.8 Ghz and 6/8 cores. Quite a jump.
That's very true about games finally moving to 6C/ or 6C/12+ these days, performance has come on in leaps and bounds but even tough most of us are PCMR it's partly thanks to the consoles finally having decent'ish specs and a proper amount of RAM to work with with regards to textures etc. Shouldn't be too long until 6C/12T will ned 8C/16T haha.
Consoles moving to 8-core CPUs have definitely had a huge impact - both in positive and negative terms, but mostly positive IMO. (The negative mainly comes down to development blunders like the horrible work done on things like Batman: Arkham Knight and similar examples of developers not being able to understand the differences between the platforms.) I mean, CPU power had effectively been stagnant for ~6 years in the PC space up until Ryzen came in a kicked things up a bit, leading PC-focused developers to solely look into adding GPU-powered features. This has to a large degree meant "prettier" games (by increasing non-CPU dependent GPU loads) with near-zero gains in terms of more complex worlds, AI, simulations, and so on. If developers finally kick the "we need to bring the 2012-era 2c4t i3s with us!" mantra to the curb (which some have been doing for a while, thankfully) we could get some really, really cool games that allow for far deeper interactivity and interesting interplay between the player and the simulation (which, arguably, could likely also be done with GPGPU coding, but that would cripple the graphics).
The interesting question IMO is how scaling might be implemented given the increased span of CPU power out there. Scaling graphical quality (within reason) is one thing - you're still playing the same game, after all. If the simulation needs to be simplified, or the AI gets dumbed down? Suddenly people with slow CPUs would be playing a markedly different game than those with fast CPUs. I suppose quite a few hardcore gamers could appreciate a faster CPU making enemies smarter or something, but that's ... a tricky proposition. Still, I'm really looking forward to what developers can do with the new CPU capabilities over the coming decade.
On the other hand, in real-world terms it'll take some quite radical shifts to reach a level where a good 4c8t still can't do quite decently. Sure, they no longer cut it when it comes to high-FPS AAA gaming, but that's a rather niche use case. Even juggling 5-6-7 game threads a fast 4c4t PC CPU could probably keep up with the 8 Jaguar cores in current-gen consoles (AFAIK both reserve one core for the OS/system), though scheduling issues might affect smoothness. The upcoming (at least 8c) Zen2 consoles - now that's something else. I can't imagine games designed from the ground up for those platforms being really playable on anything less than a
really fast 4c8t CPU, and with very significant improvements moving to 6c or even 8t - but as with current-gen consoles, it'll take a few years before this becomes a reality. Heck, the current consoles arrived in 2013, and it's only the past two years or so that we've seen a lot of games leave the 1-2-thread focus behind.
To bring this back on topic, though: I can't imagine the OP will really feel any effect of the 9700K compared to the 8600K for the lifetime of their current GPU, and even likely the next. As such, the money would be much better spent on a future GPU upgrade.