Well it could be a card issue, could be CPU or memory related. I would say exchange the card first...if the second one has the same issues, then get a refund..your system can't handle a card that new for some reason.
Maybe double check that it's fully seated, take it out, take a look over it really good, feel the heatsink, does it feel solid and firmly mounted? Re-seat it, maybe clear your CMOS whilst inspecting the card just to try something a tad different.
Sure you'll have to re-do your bios settings, but don't right away, just go into windows with defaults loaded from a cleared CMOS and see what happens. Maybe you've already done this....I don't remember reading so, but maybe there's something there.
What does GPU-z tell you? Does CPU-z show a full 16x PCI-e under the MB tab?
80C load on a GPU isn't too hot anymore, it's getting hot, but 90C seems to be around the HOT mark anymore. I read somewhere that, my x1950xtx was guarenteed by ATI to run 10 years at 100C, I don't believe it would, but it does run pretty warm under load.
A 3870 will treat you well, but hard saying if you'll have compatability or similar issues as you are here, if so, then you have something wrong beyond what you're either telling us, or something you've done. You might not agree, but there's only so many options, and in the end it can only be so many things...you could have a board that doesn't like that card, the memory could be in erroring (no BSOD's, random restarts though? RUN A MEMTEST TO MAKE SURE), run ORTHOS w/Priority 9 and Small FFT's to stress and verify your CPU is stable. Try some of these basic things and see what comes of it. Run Memtest for at least one 100% pass, and Orthos for at least overnight if you can, if not a couple hours will suffice here.
There's either something we're overlooking, something you haven't told us, something you've done but don't necessarily remember you did, or something is just plain wrong. But the options are getting slim.
Again, you'd be suprised how good C2D processors are compared to AMD, and there's no contest between your single core and a C2D. I'd still say get 2 gigs of mem and a dual core to help your gaming anyways, and it will. Call BS all you want...facts are facts.
I do hope we can find a solution for you without too much more hassle, but at this point, try those stability tests, let us know, submit a GPU-z for us. And if all else fails, exchange the card...maybe have BestBuy test it to verify if the card works in a different rig.
