thats because in Vista the /3gb switch doesn't do anything, MS disabled that feature, its still in Longhorn Server but not in Vista
Then why would MS show you how to do it on Vista then. They tell people that you should just use BCDEDIT to do it. Like I did and most ever body I know too.
Tested the same on the XP version, they work fine also.
Plus, you should mess the boot.ini file, use BCDEDIT like mention before. Or us these steps to enable it:
To enable the 3GB switch on Windows Vista
1. Right-click Command Prompt in the Accessories program group of the Start menu. Click Run as Administrator.
2. At the command prompt, enter “bcdedit /set IncreaseUserVa 3072″
3. Restart the computer.
To disable the 3GB switch
1. Right-click on Command Prompt in the Accessories program group of the Start menu. Click Run as Administrator.
2. At the command prompt, enter “bcdedit /deletevalue IncreaseUserVa”
3. Restart the computer.
http://msdn.microsoft.com/en-us/library/ms791558.aspx