Traditional, Hardware Specific and Post Processing AA
Hmm lets see...
There are several different types of AA that can be used/seen in games today.
Quincunx AA - Sometimes used on PS3 instead of MSAA. Uses 2 geometry sample points just like 2xMSAA, but it also uses 3 samples belonging to neighboring pixels. This results in blurring.
Examples:
Quincunx
vs
MSAA
-SSAA - Super Sampled AA
-MSAA - I think we all know what this is
-Transparent AA - NVIDIA for Alpha textures, etc
-Adaptive AA - AMD for Alpha textures, etc
Nvidia's CSAA - Coverage sampled AA. Used to increase frame rate by using a reduce number of coverage samples normally found in MSAA. This is typically mistaken by some as MSAA when infact it is not. IE: "I can use 8xAA in games" can be misleading when in fact that person is using 8x CSAA. If they were using true 8x MSAA it should read 8xQ (or whatever they are using now). CSAA is found in some games along with MSAA.
CSAA in Color, Z and Coverage
- 2x....= MSAA (1, 2, 2)
- 4x....= MSAA (1, 4, 4)
.....
- 8x...= CSAA (1, 4, 8)
- 8xQ = MSAA (1,8, 8)
.....
- 16x..= CSAA (1, 4, 16)
- 16xQ = CSAA (1, 8, 16)
ATI's CFAA - Custom-Filter Anti-Aliasing that takes samples from "outside the pixel boundary". It provides more samples per pixel than MSAA without any real impact on the memory footprint. They can work alongside floating-point blending based High Dynamic Range rendering and anti-aliasing stencil shadows. The draw back to Narrow and Wide Tent is blurring. Edge Detect performs a pass to discover any edges on the rendered scene, resolving those edges only (if I recall correctly). This is done by using AA while the rest of the image is AA'd using a standard box filter and a lower number of samples. If my memory recalls correctly you want to use either 8xAA (or higher) or Edge Detect you want to overclock the GPU in order to increase the Shader Clock to reduce the additional performance penalty using them.
Mode Filter
- 2x, 4x, 8x Box = Normal AA
- 4x CFAA 2x + Narrow tent
- 6x CFAA 2x + Wide tent / 4x + Narrow tent
- 8x CFAA 4x + Wide tent
- 12x CFAA 8x + Narrow tent / 4x + Edge detect
- 16x CFAA 8x + Wide tent
- 24x CFAA 8x + Edge detect
There is one more called Analytical AA which I believe is something similar to adaptive AA algorithm. But haven't found a whole lot about it.
There is also deferred rendering and forward rendering: MSAA, FSAA, AAA, etc algorithms
Practical Morphological Anti-Aliasing
Recently, I've come across another AA method called
Practical Morphological Anti-Aliasing (MLAA). This type of AA appears to be used most with Sony (GOW III, KillZone 3, etc) and it appears to have originated from Intel according to
this article. The quality of MLAA is between 4xAA and 8xAA MSAA at a fraction of the memory and time consumption. Because this process works as a post-processing it should work with any game. How it works is by detecting specific patterns in color or depth information of boarders and anti-aliasing them by blending pixels of those borders according to the type of pattern and position they belong to within the pattern.

MLAA

8xAA

1xAA
Take notice of the railing in the upper right corner. And here is another article
another article about MLAA.
Directionally Localized Anti-Aliasing This will be found in
DLAA in Force Unleashed II (link demonstrates DLAA vs no AA). DLAA is similar to MLAA but little is explained as to how it works at this time. Current execution times for DLAA are:
"Xbox 360: 2.2 +/- 0.2ms @ 1280x720
"PlayStation3 (5 SPUs): 1.6 +/- 0.3ms @ 1280x720"
[DLAA (
Directionally Localized Anti-Aliasing) - our custom anti-aliasing algorithm implemented in Star Wars : The Force Unleashed 2. In terms of picture quality it is comparable with MLAA, but due to perception based probabilistic nature it features high temporal stability and implemented on both the GPU (X360, PC) and PS3 SPUs ]
Below are some examples of DLAA vs no AA.
There is also
Directionally Adaptive Edge AA. But I've not seen it in games yet.
Nvidia's Version
SRAA
FXAA
Fear 3 uses FXAA below are examples of what FXAA looks like vs 4x MSAA.
FXAA example
vs
4xMSAA
Take notice of the grating on the crate.