I've tried to remove the graphic card to be sure it's what is so loud and it indeed is the graphic card. Without it the second most loud sound is the hard drive disk ^^
gpu-z and other tools does report ~4100 rpm at all time
I've already tried to stress my card, in games (up to 80% load with modded skyrim) the fan does not speed up, and the card remains at 40°C even under prolonged full load which indicates some powerful fan cooling i think, coherent with a "full speed fan" scenario.
I'm going to open my case and have a look at these pins