Definitely NH-D15 or Cryorig R1. Can't go wrong. Scythe Mugen 5 rev.B and Fuma too. The Scythe coolers are on par with those two, but tend to cost significantly less. For the money you get a really quiet, high quality, high-performing cooler. Any of those will more than adequately cool an OC'd 1700... enough to keep things reasonably quiet under anything but sustained max loads, even. Yeah, 4ghz on a 1700 @ 1.4v is up there. That's probably always gonna run on the warmer side. But, when configured at their most optimal, Ryzens aren't too hard to cool. Should still be tameable, particularly with your case setup - that's almost like a hybrid open air. Honestly I haven't seen too many cases like that. Assuming a good fan setup I think we can rule that out as a cause.
H7 is pretty good for the lower core-count Ryzens, like the 12/1300 or 14/1500. That is where those mid-compact towers shine. You have decent headroom there. Not a whole lot more, though. Even more so with the lower-end Zen+ chips. You could probably even use a 212 for those. But all of that current driving the extra cores in top-end Ryzens really adds up quickly. It's like they hit a steep threshold towards the top range where temps start to go up exponentially. High manual overclocks are a completely different ballgame from PBO and XFR. Those are pretty optimally balanced to squeeze out the most performance possible as efficiently as possible.
Your temps seem particularly high, but I can't say I'm surprised with it running like that. It's not THAT far off from what can be expected from an under-cooled, overclocked, high-end Ryzen. Pretty much lines up with my experiences.
I haven't gotten to do much overclocking of the vanilla Ryzens, honestly. Only vanilla chip I've really pushed was my own 2600, whereas for commissions I've generally stuck to X-models running stock or maybe slight tweaking when asked, only playing a little with all-core OC's just to see how they do. They generally run pretty cool with XFR and PBO configurations. And especially compared to my manually-OC'd 2600, which rapidly gains heat as I approach the max stable OC of 4.2ghz. Definitely takes some real cooling, even running it at a much lower 1.32v. Part of why I tend to avoid having to do it.
I did have a run once. When I pushed my 2600 to 4.3ghz, I was using a Scythe Mugen MAX, which is similar to the Mugen 5 and comparable to any of the coolers I mentioned before, if not just slightly trailing. At that point I believe I was running at least 1.4v, if not a touch more. And that was the point when temps started to run away like yours seems to be doing now. Couldn't comfortably do more than short stress tests (linpacks would take it almost right up to the throttle point, though they still passed.) Didn't do much more than bench it. I wish I could remember the wattage and current. I only remember that they were much higher than I've ever seen any Ryzen push under normal circumstances. I wanna say something like 190w. Something ridiculous. Efficiency just goes out the window with those high clock speeds. Like temperatures, it's not very linear at the top. I'm betting you're sitting at least a little below that sort of power/heat output under max loads.
4.2, on the other hand was warm, but doable. The power difference is major there. It pushes ~78C under heavy stress testing, 45-50C for everyday, and 65C max for gaming and some of the heavier stuff. Always idled ~32C. My case has nowhere near the airflow of yours, either.
Obviously the 2600 doesn't eat as much power as a 1700 under max workloads, but if I can push one like that under a decently high-performing air cooler, I'm sure they can also accommodate a 1700 running a more normal OC. Even at that voltage.
You might also consider dropping the clocks down a little. I don't think you'd need to with a better cooler, but it's still worth trying. The difference between 4.0 and 3.9 likely won't be significant performance-wise, but ime that little tick down can make a world of difference when it comes to voltage and temperature. Not always, but only ever when I've been more or less maxing a chip. Those last little nudges just add so much more power demand.
