Check what type of RAM your 2x 256's are then make sure your 512 is the exact same kind.
I use Everest home edition to get all the system info about my pc & that will tell you what kind of ram the 256's are.
never used memtest though so dont know about that one.