Need 6700XT BIOS, but is missing from the database?

[akarypid]

Hello everyone,

I don't think I've ever posted here before but I was looking for a BIOS and was surprised to find that the Gigabyte 6700XT BIOS list does not contain my card: the Gigabyte AORUS 6700 XT ELITE (GV-R67XTAORUS E-12GD).

I'm trying to use it with VFIO via passthrough but have failed miserably so far. Some people suggested I try another BIOS and pointed me here. See here for the full story and context of why I'm looking for a BIOS, but basically how come there are none for my card?

Thanks!

 

[Regeneration]

Samsung

Gigabyte RX 6700 XT VBIOS

12 GB GDDR6, 500 MHz GPU, 2000 MHz Memory

www.techpowerup.com

Micron

Gigabyte RX 6700 XT VBIOS

12 GB GDDR6, 500 MHz GPU, 2000 MHz Memory

www.techpowerup.com

 

[eidairaman1]

Hello everyone,

I don't think I've ever posted here before but I was looking for a BIOS and was surprised to find that the Gigabyte 6700XT BIOS list does not contain my card: the Gigabyte AORUS 6700 XT ELITE (GV-R67XTAORUS E-12GD).

I'm trying to use it with VFIO via passthrough but have failed miserably so far. Some people suggested I try another BIOS and pointed me here. See here for the full story and context of why I'm looking for a BIOS, but basically how come there are none for my card?

Thanks!

Why do you need it!? There is nothing to gain from bios flashing any XT card.

TechPowerUp

Extensive repository of graphics card BIOS image files. Our database covers submissions categorized by GPU vendor, type, and board partner variant.

www.techpowerup.com

TechPowerUp

Extensive repository of graphics card BIOS image files. Our database covers submissions categorized by GPU vendor, type, and board partner variant.

www.techpowerup.com

I suggest you contact Gigabyte for your specific card

 

[akarypid]

Why do you need it!? There is nothing to gain from bios flashing any XT card.

So, I linked the thread in L1 forums: I'm trying to use the card in a VFIO setup, where the GPU is assigned for direct access to a Virtual Machine. I am using it on a Gigabyte Aero G X570S board but it appears that the combination fails to do a proper reset when restarting the VM.

I've been struggling with this for quite a while and amongst other things it was suggested I use a different vBIOS (using the romfile=xxx.rom option for QEMU).

Sadly, this has not helped either...

This link contains the full story of my Gigabyte saga...

I've spent days trying to get this to work to no avail (other than the sleep/power-on hack).

 

[jhu543369]

What kernel are you using? I saw plenty of people getting this working with the 6800xt, but I could never get my Hellhound 6700 xt to resume properly (z490 and z590 MB). Your solution if it works reliably might be the best as I doubt it is anything to do with the MB bios. I recall there were some AMD driver updates in 5.15 to improve 6700 xt performance (mine now peaks at 256 watts!!!!! WTF), I am running 5.17.rc3 but don't have my passthrough setup on this (my rx 580 died so no 2nd card atm) to see if the pass back of the card works. Check this thread: rx 6800 xt passthrough

 

[akarypid]

What kernel are you using? I saw plenty of people getting this working with the 6800xt, but I could never get my Hellhound 6700 xt to resume properly (z490 and z590 MB). Your solution if it works reliably might be the best as I doubt it is anything to do with the MB bios. I recall there were some AMD driver updates in 5.15 to improve 6700 xt performance (mine now peaks at 256 watts!!!!! WTF), I am running 5.17.rc3 but don't have my passthrough setup on this (my rx 580 died so no 2nd card atm) to see if the pass back of the card works. Check this thread: rx 6800 xt passthrough

I am using Proxmox VE 7.1 as the host and have the vendor-reset extension due to using the RX480 alongside the 6700XT (ironically the RX480 woks very reliably). Here are the versions:

root@pve:~# pveversion
pve-manager/7.1-7/df5740ad (running kernel: 5.13.19-2-pve)
root@pve:~# dkms status
vendor-reset, 0.1.1, 5.13.19-2-pve, x86_64: installed

Currently I suspect it has something to do with power management. I noticed this kernel message on boot:

Feb 13 11:51:57 pve kernel: pci 0000:0b:00.1: D0 power state depends on 0000:0b:00.0

Note that these are the GPU and its audio, here is the list of related devices:

root@pve:~# lspci -nnk | grep 1002
09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [1002:73df] (rev c1)
0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]

root@pve:~# lspci -t | grep 0b -A1
           +-03.1-[09-0b]----00.0-[0a-0b]----00.0-[0b]--+-00.0
           |                                            \-00.1

I also noticed that the audio device is split into its own IOMMU group (I don't need the acs override patch, the motherboard/cpu do this fine). Everything related to the GPU (upstream/downstrem port, GPU and audio) are in their own group each:

root@pve:~# dmesg | grep group | egrep "09:00|0a:00|0b:00"
[    0.534401] pci 0000:09:00.0: Adding to iommu group 23
[    0.534416] pci 0000:0a:00.0: Adding to iommu group 24
[    0.534442] pci 0000:0b:00.0: Adding to iommu group 25
[    0.534467] pci 0000:0b:00.1: Adding to iommu group 26
root@pve:~# dmesg | grep "iommu group 2[3-6]"
[    0.534401] pci 0000:09:00.0: Adding to iommu group 23
[    0.534416] pci 0000:0a:00.0: Adding to iommu group 24
[    0.534442] pci 0000:0b:00.0: Adding to iommu group 25
[    0.534467] pci 0000:0b:00.1: Adding to iommu group 26

I am going to see if I can get the GPU and audio to go into the same IOMMU group as the RX480 works like that (currently I removed it from the system to reduce the surface of debugging).

My current theory is that the reason the GPU won't power down fully is because the audio device needs to power down at the same time and since both are in their own IOMMU group this causes QEMU to power down the GPU first (which is prevented by the audio being still active). Hopefully if I can group them the power-off will be done in one step for both.

 

[jhu543369]

Does amd back port fixes into the 5.13 kernel gpu drivers? I run mint, and wasn't until I got to 5.15 that I could say I had a stable gpu experience with the 6700 xt. I was lucky, both audio and gpu were in the same group for me. Also what kernel options did you use? Mine were:

intel_iommu=on iommu=pt iommu=1 kvm.ignore_msrs=1 vfio-pci.ids=1002:73df,1002:ab28

I had to prevent the kernel from loading the amdgpu drivers for the card (audio and gpu).
Also, I had to make more changes to the qemu files beyond the rom file. Unfortunately I don't have them here anymore. I know some of it was from the link I gave you, even though that is a single gpu approach. Still think you need to examine the kernel issue though...

 

[eidairaman1]

I seriously doubt a bios will fix your problem when it's the software at fault.

 

[akarypid]

I seriously doubt a bios will fix your problem when it's the software at fault.

@eidairaman1 What makes you sure this is a software issue?

Yes, the BIOS did not help at all, but am just wondering why you are certain there's no hardware at fault here. I have a window to return the GPU until the 23rd of February so would be nice to have confirmation that I can keep it and wait for future updates to Linux kernel and AMD drivers (for example).

Another possible source of the issue is the motherboard I guess. I tried 3 different BIOS versions for it to no avail (all result in same issue).

Ironically my RX480 and an RX550 that I have access to both work fine on my motherboard (Gigabyte Aero G X570s) with the same software (Proxmox 7.1). I've even tried a newer kernel with Fedora 35 (version 5.16) with the same results...

 

[Athlonite]

does it work ok on its own (RX6700XT) without the RX480 or the RX550 in the mix

 

[akarypid]

does it work ok on its own (RX6700XT) without the RX480 or the RX550 in the mix

So the RX6700XT never works more than once: I tried both top slots on my PC with 3 different BIOS versions of my motherboard. It does not work by itself, and does not work alongside the RX480 or alongside the RX550. In all cases it boots ONCE and I am able to use it in a VM. When I shut down that VM (or simply restart it) it will:

1) on first attempt to reuse in new boot of VM logs:

Feb 12 15:22:05 pve kernel: vfio-pci 0000:0b:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 12 15:22:05 pve kernel: vfio-pci 0000:0b:00.0: vfio_bar_restore: reset recovery - restoring BARs
# ... restoring BARs repeats several more times

2) on second attempt to reuse in new boot of VM logs:

Feb 12 15:24:03 pve kernel: vfio-pci 0000:0b:00.0: timed out waiting for pending transaction; performing function level reset anyway
Feb 12 15:24:05 pve kernel: vfio-pci 0000:0b:00.0: not ready 1023ms after FLR; waiting
...and so on doubling the interval until it gives up

I have a very detailed description on this thread at L1 forums.

I also tried passing vBIOS but had no success.

Ironically, both the RX480 and RX550 *work fine* in all combinations: together in slots 1/2, together in slots 2/1, alone in slot 1, alone in slot 2, alongside the 6700xt (in either slot)...

That's why I suspect it's the 6700XT and not something else...

 

[eidairaman1]

So the RX6700XT never works more than once: I tried both top slots on my PC with 3 different BIOS versions of my motherboard. It does not work by itself, and does not work alongside the RX480 or alongside the RX550. In all cases it boots ONCE and I am able to use it in a VM. When I shut down that VM (or simply restart it) it will:

1) on first attempt to reuse in new boot of VM logs:

Feb 12 15:22:05 pve kernel: vfio-pci 0000:0b:00.1: vfio_bar_restore: reset recovery - restoring BARs
Feb 12 15:22:05 pve kernel: vfio-pci 0000:0b:00.0: vfio_bar_restore: reset recovery - restoring BARs
# ... restoring BARs repeats several more times

2) on second attempt to reuse in new boot of VM logs:

Feb 12 15:24:03 pve kernel: vfio-pci 0000:0b:00.0: timed out waiting for pending transaction; performing function level reset anyway
Feb 12 15:24:05 pve kernel: vfio-pci 0000:0b:00.0: not ready 1023ms after FLR; waiting
...and so on doubling the interval until it gives up

I have a very detailed description on this thread at L1 forums.

I also tried passing vBIOS but had no success.

Ironically, both the RX480 and RX550 *work fine* in all combinations: together in slots 1/2, together in slots 2/1, alone in slot 1, alone in slot 2, alongside the 6700xt (in either slot)...

That's why I suspect it's the 6700XT and not something else...

Something in the software is at fault, Polaris and RDNA are different.

Report problem to software developer.

The 6700 is fine.

 

[Athlonite]

Something in the software is at fault, Polaris and RDNA are different.

Report problem to software developer.

The 6700 is fine.

Yup that's exactly what I was thinking is that there is a problem with RDNA vs GCN and the software not being updated to take advantage of the RDNA architecture properly