Wait just a second. There seem to be two related question here, then a huge leap in logic. First, a decent quality 650 watt PSU should power up that system with no problem. I've got a 650 watt Corsair supply that powered 4 mechanical drives, a core2 quad, and 6870 for three years; that PSU is now powering a 2500k, 6870, three mechanical drives, and an SSD without a problem. So by my reckoning, there are two questions:
1) Is my instability caused by my 650 watt PSU?
2) How do I make my system stable?
Right?
1) You're using an Antec supply. There are two ways this could be the problem, but barring more substantial information (a screenshot of the voltages in something like Speedfan, freeware, would be needed at minimum) all we can do is conjecture.
Are you, perhaps, using a power connector that converts Molex to the required 6/8 pins connectors? If so, the rail in that PSU is overloaded and that's where the instability comes from. I've seen some Antec supplies that have very poor peaks on the Molex/Peripheral power supply rails.
If one isn't true, then how old is the PSU? Antec supplies, again in my experience, aren't the most solid as they age. A supply that is a couple of years old might well be on its last legs, though I've had new ones that barely managed to scrape by. Again, we'd need more information to do anything more than speculate.
2) This is going to take a bit of doing, and I'm going to assume you know nothing. As such, please don't take offense if you already know/have done what I am saying.
a) Check the RAM stability in Windows. Utilize the Intel Burn Test to make sure the CPU and RAM are stable, thereby removing them from the instability question.
b) Confirm that the PSU is outputting the correct voltages, and that everything is securely plugged in. Speedfan takes software readings of voltages (it isn't 100% accurate, but I assume that no tools are available).
c) Make sure that the airflow to the card is unrestricted. Cable wiring is often a messy task, but just allowing wires to hang impedes air flow. In order to keep cool air flowing into the card you need to make sure that there are as few obstructions as possible.
d) Run GPU-Z and make sure the graphics card shows up properly. GPU-Z is excellent at determining if a card is being artificially throttled, or if there is something wrong with the voltage supplied to the card.
e) Barring everything else, get an RMA. The cheapest component to replace may be the PSU, but if everything else (on this list) tests out then you're down to a GPU problem. I've seen more expensive CPUs arrive DOA, and this is why RMAs and warranties exist. A 400 USD card should work, but they don't always. Just make sure you're getting your moneys worth.