HDMI really is the best way to go, provided your graphics card supports everything it should (since that's going to be your audio device now if you go with HDMI), and your HDMI enabled receiver is a quality one. I'm not sure for nVidia, but I know ATi's cards can do DTS and Dolby decoding to LPCM if your receiver can't, given the neccessary software. That means you can still enjoy DTS-HD Master audio lossless tracks on Blu-rays even if the receiver can't decode the format itself. As far as sound/picture quality when running PC>Receiver>TV via HDMI, it really depends on the receiver (and attached speakers), for both picture and sound quality. The receiver's DACs and decoding (for DTS-HD, DTS, DDL, Dolby Master Audio, etc etc) are what matters in terms of audio quality. Look for reviews, get to know what matters to you.
Any receiver supporting HDMI will also have some sort of video processing functionality. The basic ones would just have a repeater. SO for digital video its video in = video out (but perhaps more latency, and some other bad things). The better ones, like my overkill Onkyo TX-SR876 have a built in video processing chip such as the excellent
HQV Reon, for example, which will do pulldown conversion, color calibration, noise reduction, upscaling, cadence detection, deinterlacing and a bunch of other complicated math to make "Poor quality video in = somewhat better quality video out" and "good quality video in = same or slightly better video out" with very little latency added and no reduction in quality.
There's really a LOT more to it than just buying a receiver and hooking it up with digital, whether you go optical or HDMI, and getting a better sound. Like has been said, if the DACs in your receiver aren't better than the DACs in your Sound card, it won't sound much better if at all - this is true for Optical and HDMI. (it should be better in some ways since the receiver/amp should drive speakers better, and you'd likely hook better speakers to your amp)
If you are already using optical your sound card isn't doing much at all. Whatever the optical is plugged into is doing the Digital>Analog conversion and much if not all of the signal proccessing
Really the ONLY difference between using Optical or HDMI is that HDMI:
- HDMI has HDCP to protect the audio/video stream (DRM!)
-HDMI supports many formats that Optical does not
-HDMI supports 7.1 (8 Channels ) uncompressed LPCM along with support for a bunch of other compressed formats (so the receiver doesn't NEED to be able to decode things, the computer can do it and send LPCM which is an effectively lossless process. Depending on the capabilities of the receiver, with HDMI some software can also send the data directly to the receiver and allow the receiver to decode the DTS, DOLBY, or whatever stream you happen to be using. That's called bitstreaming, which depending on your what your receiver can handle as a bitstream, can be better or worse than LPCM. the Optical is limited to 5.1 compressed or 2.0 stereo uncompressed, so less channels and lower quality.)
-HDMI requires the reciever to also handle the video, whether by processing it, or just passing it on.
All of this said, I use my Radeon 5870's HDMI out into my Onkyo receiver, and the sound is far better than what I got out of my previous $250 XFi with some small powered speakers. Why? The speakers I have now are better, the receiver has as good if not better DACs onboard (not to mention no noise picked up because there's no analog signal until it's already out of the EMI rich PC case and into the receiver), and the video proc is top notch. My advice would be to take an HDMI equipped laptop, if you have one, to a local audio store and ask to test out receivers. Find the one that sounds and looks the best to you for what you're willing to pay, and then buy it from wherever is cheapest. Pair it with some decent speakers and you'll be grinning ear to ear. I was when I finally saved up for my setup. Truly the best I've ever heard a computer sound. Remember though, better quality sound hardware won't make crappy mp3 files sound much better. Games, blu-ray discs, lossless audio files will sound FANTASTIC. The hardware only makes the reproduction better. If you start with crap, don't expect anything but a polished turd (i.e. don't waste your money)
Check out this thread, and ask specific questions to get quality answers