AMD H.265 destroys AV1 in OBS ???

[Krit]

I have tested video quality at the same bitrate 10000kbps, 2560x1080 in (OBS 20.2.3) AMD HW AV1 vs AMD HW H.265 (HEVC)

For some reason H.265 has better image quality, better detalization and more clear image overall even at higher bitrates like 50000kbps.
I tested it in many games H.265 always wins slightly or even significantly in some areas like s.t.a.l.k.e.r. shadow of chernobyl where difference is clearly visible with naked eye.

I'm really confused because a year ago when I had a RX 7900 XT I tested AV1 and it was better than H.265 at the same bitrate so what's going on there ? Or i'm wrong and H.265 always has been better than AV1 when recording gameplay with Radeon RDNA3 gpu ? Please share your thoughts and experiences on AV1 vs H.265 in terms of image quality at the same bitrate.

Left AV1 and Right H.265 video quality comparision.

 

[Dr. Dro]

AMD's hardware encoders are very poor. It only really started getting better now with Navi 31. AV1 is the superior codec, but AMD's encoder might not be up to par.

 

[Timbaloo]

The AV1 result looks a lot worse than I'd expect (even from subpar HW encoding implementation). Like this is really bad. Sure nothing in the settings is fubar?

It might be that h265 does slightly better with in-game graphics (as in dealing with aliasing, sharp edges/gradients), but the difference here is just too much.

 

[Krit]

The AV1 result looks a lot worse than I'd expect (even from subpar HW encoding implementation). Like this is really bad. Sure nothing in the settings is fubar?

That's way i startet this thread becouse a year ago with RX 7900 XT AV1 was better than H.265!!!! It's big WTF to me? How can it be.

 

[InVasMani]

AMD's hardware encoders are very poor. It only really started getting better now with Navi 31. AV1 is the superior codec, but AMD's encoder might not be up to par.

Ironically HW encoder quality is a area Intel has done well with. I'd actually use my iGPU on 14700K for video playback if desktop color depth was 10-bit. It can do video playback at 10-bit actual desktop bit depth is 8-bit. It's not worth the aggravation even though the video quality is better. I care more about standard usage across image details across all programs than I do just video playback. I don't feel like switch display outputs or going multi-display either it's a unneeded nuisance. The iGPU itself is actually moderately reasonable though. It's not great, but for integrate I'm still pleasantly surprised at how capable modern iGPU's have gotten. Intel's encoder quality is clean and crisp.

 

[OneMoar]

AMDS AV1 hardware encoder is dogshit if you want av1 that works get a Arc or Nvidia card for transcode duty

 

[chrcoluk]

I am going to test Nvidia new NVENC in OBSS in the coming days.

On my 3080 software x264 was the daddy, quality combined with reasonable file sizes. The downside it took a lot of the CPU resources and consumed a lot of power.
Selecting NVENC in OBSS and even after lots of fine tuning the quality was either very poor or ok combined with massive file sizes. The upside is it barely used any resources and much lower power consumption.
Using HEVC unofficial addon in OBSS brought the quality and filesize closer to software x264, but I do see random frame drops in the recordings, meaning the NVENC chip was saturated at points.

So will test the new official NVENC AV1 encoding and hope it finally is as good as software x264.

 

[OneMoar]

nVIDIA supports AV1 just for screen capture? Like Shadowplay.... Not for streaming!!!

non-sense shh

 

[Dr. Dro]

Ironically HW encoder quality is a area Intel has done well with. I'd actually use my iGPU on 14700K for video playback if desktop color depth was 10-bit. It can do video playback at 10-bit actual desktop bit depth is 8-bit. It's not worth the aggravation even though the video quality is better. I care more about standard usage across image details across all programs than I do just video playback. I don't feel like switch display outputs or going multi-display either it's a unneeded nuisance. The iGPU itself is actually moderately reasonable though. It's not great, but for integrate I'm still pleasantly surprised at how capable modern iGPU's have gotten. Intel's encoder quality is clean and crisp.

Yeah. It's a bit of a shame the Apex Encore doesn't have any circuitry for the iGPU. I mean, I get it, it's dead weight for a XOC board, even though Tachyon X has it.

It effectively turns K/KS CPUs into KF, the graphics related pins aren't wired and there are no iGPU power stages. I used to encode a few things with QuickSync Video on my CPU's UHD 770 on the old motherboard.

 

[Krit]

RX 7900 XT, XTX, RX 7800 XT, RX 7700 XT users do you have the same poor AV1 video quality results ? On paper AV1 should be better than H.265 but i remember with RX 7900 XT it was like that one year or so ago.

 

[Rus4kova]

Two screen shots

This is from a software encode using SVT-AV1 and using variance boost and tune 0 and preset 4

This is from a GPU encode using VCEEncC_8.22_x64 with the slow preset.

The SVT-AV1 file is 16.514.604 bytes
The VCEEncC_8.22_x64 file is 16.851.052 bytes

The AV1 result looks a lot worse than I'd expect (even from subpar HW encoding implementation). Like this is really bad. Sure nothing in the settings is fubar?

It might be that h265 does slightly better with in-game graphics (as in dealing with aliasing, sharp edges/gradients), but the difference here is just too much.

I agree something looks off. H265 is notoriously bad when it comes to dark scenes. AV1 is significantly better in that area. Overall AV1 has come a long way and I prefer it over H265 any day. Even though I have done thousands of H265 encodes using AVI-synth over the years. With a 9950x using avx-512 you can sometimes encode AV1 at > 100 fps using preset 5 and variance boost in 1080p.

 

[DaemonForce]

It looks like a standard tripod shot but I like the subtle difference in quality.
How well is the encoder holding up while these are streamed?
This stuff is like 90% why I'm pushed towards the 7900XT for anything but I just can't seem to get one that works.
My RX 580 can run 720p60 (preferred) and 1080p30 semi-stable but I need two encoders running on one package.
Wat do?

 

[Krit]

It looks like a standard tripod shot but I like the subtle difference in quality.

Static videos will not show the difference! No significant difference there. It's moving games in motion that makes the biggest difference in terms of video quality between AV1 and H.265

 

[Dr. Dro]

Static videos will not show the difference! No significant difference there. It's moving games in motion that makes the biggest difference in terms of video quality between AV1 and H.265

Encoders also tend to have severe issues with grass at low bit rates. Makes for a good test

 

[DaemonForce]

That's the sort of thing that's always on my mind because I don't have a solid way of paying attention to encoder hiccups.
I also don't want to delegate the encode job to my FX-8370 sooooo the goal is basic Twitch non-parter bitrate (~6000) but a quality where I would want to watch me.
That's a pretty tall order no matter how many different ways you slice it and that's on top of the VR+desktop experience. Extremely complicated.

 

[chrcoluk]

I am going to test Nvidia new NVENC in OBSS in the coming days.

On my 3080 software x264 was the daddy, quality combined with reasonable file sizes. The downside it took a lot of the CPU resources and consumed a lot of power.
Selecting NVENC in OBSS and even after lots of fine tuning the quality was either very poor or ok combined with massive file sizes. The upside is it barely used any resources and much lower power consumption.
Using HEVC unofficial addon in OBSS brought the quality and filesize closer to software x264, but I do see random frame drops in the recordings, meaning the NVENC chip was saturated at points.

So will test the new official NVENC AV1 encoding and hope it finally is as good as software x264.

Done this now, short answer I think AV1 is trash compared to HEVC.

So the topic title fits for me.

To get HEVC level of quality, I need about 3x the size of the file.

Now previously using a fine tuned VBR in OBS 27.x StreamFX plugin, this plugin exposes loads of extra options. OBS stock configurability is really limited. In the latest OBS I cannot use Xstream as its now behind a paywall, but both HEVC and AV1 are built into OBS for NVENC, I spent must be what 3-4 hours on AV1, and yeah trash, I went into the built in HEVC mode and within 5 mins got it almost as good as the Xstream setup although on CQP, the VBR is trash in stock OBS compared to Xstream.

I am seriously considering restoring old OBS from snapshot, but is a chance I stick with the built in NVENC HEVC on the OBS 30.2.

CQP 26 on HEVC is as good as CQP 10 on AV1, the difference is that big. Shame I cant get best of both worlds and have AV1 support exposed whilst having all the tuning knobs for HEVC.

(seems its just the NVENC version, tested SVT-AV1 and thats fine)

 

[Dr. Dro]

Done this now, short answer I think AV1 is trash compared to HEVC.

So the topic title fits for me.

To get HEVC level of quality, I need about 3x the size of the file.

Now previously using a fine tuned VBR in OBS 27.x Xstream plugin, this plugin exposes loads of extra options. OBS stock configurability is really limited. In the latest OBS I cannot use Xstream as its now behind a paywall, but both HEVC and AV1 are built into OBS for NVENC, I spent must be what 3-4 hours on AV1, and yeah trash, I went into the built in HEVC mode and within 5 mins got it almost as good as the Xstream setup although on CQP, the VBR is trash in stock OBS compared to Xstream.

I am seriously considering restoring old OBS from snapshot, but is a chance I stick with the built in NVENC HEVC on the OBS 30.2.

CQP 26 on HEVC is as good as CQP 10 on AV1, the difference is that big. Shame I cant get best of both worlds and have AV1 support exposed whilst having all the tuning knobs for HEVC.

Your encoding settings must be wrong. There's no way AV1 looks worse than HEVC, it's comparable to VVC in quality.

 

[chrcoluk]

Your encoding settings must be wrong. There's no way AV1 looks worse than HEVC, it's comparable to VVC in quality.

There is barely any settings to change, its just poor, I mean its ok if you dont pay attention to finer details, it did admittedly take me a while to notice, but I was testing file sizes on the load game screen in Vesperia, and then noticed that some very fine lines were almost missing from the AV1 recordings, I could get them back by dropping CQP to 10, but then I am dealing with file sizes that are very inefficient.

The settings are basically NVENC preset, CQP, b-frames, and look ahead, psycho visuals. So for me, with my OCD it is CPU x264 > NVENC HEVC > NVENC AV1.

I have noticed is a fair few people who seem ok with massive file sizes, like 15+ gigs per hour. If I was ok with that level of wastage, then yeah AV1 is fine.

NVIDIA NVENC Obs Guide

Configure OBS to get the most quality out of your stream.

www.nvidia.com

Chatting to someone about it, seems Nvidia's AV1 is pretty immature right now, I notice SVT-AV1 is way better than NVENC AV1.

 

[Krit]

Your encoding settings must be wrong. There's no way AV1 looks worse than HEVC, it's comparable to VVC in quality.

No it's true! I tested it many times. At the same file size AV1 gets beaten badly. AV1 is just a hype job tech on paper it's good but in reality no it isn't. I'm using OBS, CBR or Radeon Software and it's almost the same in terms of quality. Dr. Dro test for yourself and you will see.

 

[Dr. Dro]

No it's true! I tested it many times. At the same file size AV1 gets beaten badly. AV1 is just a hype job tech on paper it's good but in reality no it isn't.

That's not right. AV1 will beat HEVC at low bit rates and match it at higher bit rates. Your encoder or encoding settings are whack

Codecs for the 4K Era: HEVC, AV1, VVC and Beyond

This article was written in March 2023.

chipsandcheese.com

 

[chrcoluk]

There isnt any mention of NVENC in that article, I think Nvidia have played games to get speed, NVENC x264 sucks against CPU x264, and it seems its the same with AV1, SVT-AV1 is pretty good though.

 

[tfdsaf]

Nvenc is good for speed, but its garbage for quality. I've used it many times with my old 1060 6GB and its bad. It is though consuming very few resources and it makes games playable, you only lose maybe 5-10fps.

 

[Rus4kova]

No it's true! I tested it many times. At the same file size AV1 gets beaten badly. AV1 is just a hype job tech on paper it's good but in reality no it isn't.

When it comes to low light scenes AV1 runs circles around HEVC. AV1 is also better at not introducing banding.

Nvenc is good for speed, but its garbage for quality. I've used it many times with my old 1060 6GB and its bad. It is though consuming very few resources and it makes games playable, you only lose maybe 5-10fps.

NVEnc is better on later hardware, since it supports more features i.e B-frames on 30xx GPU's. NVEnc on a 10xx GPU is ok but not great. Personally I find NVEnc to be good at re-encoding 2160p HDR. It looks really good being encoded on a 30xx GPU.
AV1/HEVC encoding on an AMD GPU using VCEEnc looks good, but can't beat SVT-AV1.

 

[chrcoluk]

I have managed to improve it by adding custom flags, a circa 20 minute recording from my gameplay last night was about 800meg in size which is decent for 1080p 60fps encode, and better quality than the basic tuning in OBS allows, with no frame drops in the recording.

But to me, if a vendor is to claim something is AV1 it needs to match the original software implementation. So I think its a combination of the hardware based encoder's not being fully compliant to specs, and lazy implementation by OBS.

Software encodes are so much better, but sadly just too resource heavy. I did also do a few clips last night in SVT-AV1 and wow.

 

[Dr. Dro]

I have managed to improve it by adding custom flags, a circa 20 minute recording from my gameplay last night was about 800meg in size which is decent for 1080p 60fps encode, and better quality than the basic tuning in OBS allows, with no frame drops in the recording.

But to me, if a vendor is to claim something is AV1 it needs to match the original software implementation. So I think its a combination of the hardware based encoder's not being fully compliant to specs, and lazy implementation by OBS.

Software encodes are so much better, but sadly just too resource heavy. I did also do a few clips last night in SVT-AV1 and wow.

20 minutes on 1080p/60 at around 800 MB looks like you're pushing the codec to the max when it comes to compression. I'd estimate no more than 4 to 5 Mbps - you have a 4080 Super, right? Try this. Not the prettiest but for the file size, can't complain either.