I have used Fraps extensively along with having tried Game Cam, PlayClaw, Camtasia and DxTory, and I can tell you without a doubt DxTory is by far the best capture program available and costs only $5 more than Fraps. Many whom have tried DxTory feel it's the capture program that has finally dethroned Fraps. So, before I continue, I certainly hope you haven't purchased Fraps.
Here's the differences between DxTory and Fraps. I had an entire list made up that I've posted before, but can't find it. I may be forgetting some things, but as soon as I can get back on to GameFront (down for maintenance or something) I can double check it.
1. DxTory can capture without lag and record at any FPS you choose (30 is optimal for YouTube) while playing at your normal gaming FPS without any capture lag. Fraps is limited to capturing at the FPS you play at, and is known for it's heavy lag
2. DxTory can use more than one HDD to write captures simultaneously with it's "Distribution HDD Writing" feature. This is how it captures lag free. Other capture prog devs have claimed their's record lag free (Game Cam, PlayClaw), but in reality they just dumb down the quality of video to do so. This is the FIRST capture program to use this type of feature to record lag free without image quality compromises.
3. DxTory can be set to use any size output file. With Fraps you only get half and full frame options.
4. DxTory can capture audio and video uncompressed, with one of 3 settings using it's own codec, or via any bitrate you choose with a number of other installed codecs
5. DxTory has a built-in FPS limiter. You can set it to any FPS or not use it at all
6. DxTory has a built-in HDD write speed bench tool
7. DxTory has a built-in cropping feature
8. DxTory has a built-in AVIMux tool
9. DxTory has a built-in AVIFix tool
10. DxTory has a support forum
Now for some general capturing/encoding tips.
1080p looks nice, but few people view YouTube vids in 1080p because it buffers so slowly. You can try it and maybe ask people what size they're viewing in. If most say 720p, than you will have wasted lots of file size making those 1080p output files when you could have made 720p ones. 1080p files also take MUCH longer to encode and upload.
DxTory will allow you to play at 1080p with the full FPS your rig is capable of and capture at 720p (or anything you choose), while outputting the file in a perfect for YouTube 30 FPS.
x264 offers superior image quality and can be plugged into VirtualDub easily
If you want good image quality, even using x264 bitrates for 720p can range from 3000-6000Kbps. For 1080p it can range from 7000-10,000Kbps.
There are guides on how to compress with x264, or you can DL it and ask about anything you don't understand here.