The GCD on Navi31 and 32 are onm TSMC N5. It is the APUs that are on N4.
As for the performance, I think it is more likely that AMD over estimated their software teams' ability to utlitise the dual-issue FP32 on RDNA3.
Also the reduction in Infinity Cache size does not help. Navi21 has 128MB...