10 Gigabit Won't Work

[Eric_Cartman]

I bought a Netgear 10 Port switch that has 2 10G ports.

I bought this switch.

https://www.amazon.com/dp/B076642YPN/

I needed a new switch so I figured I'd get one with 10G ports.

I also bought 2 10G network cards for my computers.

These are the network cards I bought.

https://www.amazon.com/gp/product/B01N5AOWW6/

I've connected my main computer and my media server to the 10G ports with the 10G cards.

The problem is that it seems like my connection is limited to about 3 Gigabits.

It says 10G in windows but when I try to run iperf between the two computers I only get about 3G.

And transfering files is also a lot slower than 10G.

I'm scratching my head now trying to figure out the problem.

Any suggestions?

 

[Selaya]

Are you running NVMe SSDs on your mediaserver?
Because SATA will cap out well below 10,000BASE speeds and bottleneck your network.

 

[kayjay010101]

Are you running NVMe SSDs on your mediaserver?
Because SATA will cap out well below 10,000BASE speeds and bottleneck your network.

iperf doesn't use storage, it sends only network packets. If iperf is reporting sub ~10gbit, it's a connection issue.

 

[Selaya]

Yeah I just realised that.

 

[W1zzard]

Did you enable jumbo frames on all equipment?

 

[TheLostSwede]

I have pretty much the same setup. Got the EMX version of the switch though and two cards from Aquantia.
Not having any of the issues you're seeing. Not using Jumbo Frames, as there's zero speed difference enabling at 10Gbps.
I obviously don't get exactly 10Gbps either, but it's a lot faster than what you're seeing.
Check the settings so one card isn't set to 5Gbps or something like that. The link lights on the switch should both be green on the 10Gbps ports.
I also presume both machines are at least PCIe 2.0 (ideally PCIE 3.0) and that you're using a x4 slot for the cards? Anything else and you won't get 10Gbps.

 

[Bill_Bright]

What about your cables?

CAT5e has a maximum bandwidth of 1Gbps. CAT6 can go up to 10Gbps up to 55meters (~180 feet). But note that is under ideal conditions. Cheap connectors, bad crimps, even the number and radius of the bends (as in kinks) in the cable can affect the effective bandwidth.

Ethernet cables are cheap, flimsy, easily damaged, but critically important network devices. I always recommend making your own cables. For one, if I need a 13 foot, or a 15 inch cable, I can make them instead of buying a 25 foot and a 3 foot cable.

But I learned long ago it is important to "invest" in a quality crimper. Cheap (as in low budget) crimpers provide cheap (as in poor quality) crimps. Same with the connectors. And investing in a decent cable tester reduces (or at least does not raise) blood pressures and hair loss.

But there is a skill involved so practice makes perfect. Be ready to sacrifice some cable and a few connectors.

 

[Eric_Cartman]

Did you enable jumbo frames on all equipment?

I tried enabling jumbo frames, it made no difference.

I have pretty much the same setup. Got the EMX version of the switch though and two cards from Aquantia.
Not having any of the issues you're seeing. Not using Jumbo Frames, as there's zero speed difference enabling at 10Gbps.
I obviously don't get exactly 10Gbps either, but it's a lot faster than what you're seeing.
Check the settings so one card isn't set to 5Gbps or something like that. The link lights on the switch should both be green on the 10Gbps ports.
I also presume both machines are at least PCIe 2.0 (ideally PCIE 3.0) and that you're using a x4 slot for the cards? Anything else and you won't get 10Gbps.

View attachment 197516
View attachment 197517

This is what I get with iperf. I'm not even getting 3G now.

Both cards say they are connected at 10G in Windows.

Both cards are in 3.0 x4 slots.

Are you running NVMe SSDs on your mediaserver?
Because SATA will cap out well below 10,000BASE speeds and bottleneck your network.

I have a SATA SSD in my computer and a pair of RAID0 hard drives in the server.

That I guess explains why I'm not getting full 10G during file transfers but I think it should be faster.

And it doesn't explain the crap iperf results.

What about your cables?

CAT5e has a maximum bandwidth of 1Gbps. CAT6 can go up to 10Gbps up to 55meters (~180 feet). But note that is under ideal conditions. Cheap connectors, bad crimps, even the number and radius of the bends (as in kinks) in the cable can affect the effective bandwidth.

Ethernet cables are cheap, flimsy, easily damaged, but critically important network devices. I always recommend making your own cables. For one, if I need a 13 foot, or a 15 inch cable, I can make them instead of buying a 25 foot and a 3 foot cable.

But I learned long ago it is important to "invest" in a quality crimper. Cheap (as in low budget) crimpers provide cheap (as in poor quality) crimps. Same with the connectors. And investing in a decent cable tester reduces (or at least does not raise) blood pressures and hair loss.

But there is a skill involved so practice makes perfect. Be ready to sacrifice some cable and a few connectors.

I have store bought 6ft Cat5e cables.

I read Cat5e can handle 10G up to 40m or something like that.

So the cables shouldn't be a problem.

 

[dragontamer5788]

Try a direct connection without the switch.

I find it unlikely that the switch is the problem, but the fewer things you're testing, the better. I've heard that 10Gbit can overheat some switches (causing slowdowns). Again, I don't think that's happening in your case, but might as well cross that issue off the list by just doing direct connections (for test purposes).

 

[Bill_Bright]

I read Cat5e can handle 10G up to 40m or something like that.

So the cables shouldn't be a problem.

No. That is wrong. As I noted, CAT5e only supports up to 1Gbps. This is easily verified with our friend Bing Google.

Cat5e vs. Cat6 (cablematters.com)

How to Choose an Ethernet Cable | Digital Trends

 

[TheLostSwede]

Are you sure the cards are getting the correct bandwidth? That is 2.5Gbps speed, which suggests some setting is wrong.
Could very well be the Cat 5e cables, not all cables are the same, so there's no guarantee it'll work, as Cat 5e was not designed for 10Gbps speeds at all. Yes, it can work on a short length of cable, but no way 40m. At least try getting a couple of Cat 6 cables, as they can do 10Gbps up to 55m.

Try a direct connection without the switch.

I find it unlikely that the switch is the problem, but the fewer things you're testing, the better. I've heard that 10Gbit can overheat some switches (causing slowdowns). Again, I don't think that's happening in your case, but might as well cross that issue off the list by just doing direct connections (for test purposes).

It's not the switch, I've had mine for a couple of years and I have never had any issues with it. It's on 24/7/365.

 

[dragontamer5788]

No. That is wrong. As I noted, CAT5e only supports up to 1Gbps. This is easily verified with our friend Bing Google.

"Supports" is not a concept that works very well in the physical world.

CAT5e is designed for 1Gbps. But in practice, you'll see speeds in excess of 1Gbps if you put equipment on them (or less, depending on local conditions). As a physical device, you can affect the speed of a connection by twisting or stepping on cables sometimes. CAT6 is designed for 10Gbps. That means its noise characteristics are low enough that 10Gbps can be "heard" through to the other side.

Ethernet (including 10Gbit Ethernet) is designed to auto-negotiate. At the beginning of any connection, there's a period where the computer tests what speed its able to get to. If its able to get to 2.5 Gbps, then it will use that (even if its a 10Gbit card). The physical wire may be too noisy. Even a CAT6 cable could be installed wrong and perform poorly. Twist the cable the wrong way, and the stress/strain on the copper inside the cable can degrade the signal (so now its "too noisy" for 10Gbps, but maybe 5Gbps or 1Gbps will work). Or twists / kinks in the wire, or maybe broken shielding, or poorly installed heads.

In contrast, if you really baby the cable: keep it short, have high quality connections, do a good job installing the head... then even CAT5e can do 10Gbps (even though its not designed for it). In any case, working with higher quality cables (ex: CAT6) would help remove one possible issue.

 

[elghinnarisa]

"Supports" is not a concept that works very well in the physical world.

CAT5e is designed for 1Gbps. But in practice, you'll see speeds in excess of 1Gbps if you put equipment on them (or less, depending on local conditions). As a physical device, you can affect the speed of a connection by twisting or stepping on cables sometimes. CAT6 is designed for 10Gbps. That means its noise characteristics are low enough that 10Gbps can be "heard" through to the other side.

Ethernet (including 10Gbit Ethernet) is designed to auto-negotiate. At the beginning of any connection, there's a period where the computer tests what speed its able to get to. If its able to get to 2.5 Gbps, then it will use that (even if its a 10Gbit card). The physical wire may be too noisy. Even a CAT6 cable could be installed wrong and perform poorly. Twist the cable the wrong way, and the stress/strain on the copper inside the cable can degrade the signal (so now its "too noisy" for 10Gbps, but maybe 5Gbps or 1Gbps will work). Or twists / kinks in the wire, or maybe broken shielding, or poorly installed heads.

In contrast, if you really baby the cable: keep it short, have high quality connections, do a good job installing the head... then even CAT5e can do 10Gbps (even though its not designed for it). In any case, working with higher quality cables (ex: CAT6) would help remove one possible issue.

Worth adding is that the IEEE states that cat5e is a "reliable medium for support of 2.5Gbps and 5Gbps" for up too 100m
Crosstalk and interference in general however could still cause problems.

10.1109/IEEESTD.2018.8457469
126.7.2

I have also seen a lot of people over the years successfully running 10Gbps over cat5e up to 50 meters, although results vary depending on environment and quality of cable.

Point being, cat5e can without a doubt handle more than just 1Gbps.

 

[Bill_Bright]

CAT5e is designed for 1Gbps. But in practice, you'll see speeds in excess of 1Gbps if you put equipment on them

The specs for CAT5e state it supports up to 1Gbps. The OP has already shown where he is getting 3Gbps - so we know it can happen. But he has a 10Gbps switch and NICs and is looking to get, and rightfully so, at least close to 10Gbps.

While faster speeds with CAT5e are possible, that is with ideal conditions - good cable, good connectors, good crimps, short distances, no kinks and minimum to no EMI/RFI.

It should also be pointed out that other factors can limit over all speeds too - such as the speed of the drives you are copying to and from.

Regardless, since we are not talking about pulling new cables threw walls, I would replace those CAT5e with CAT6 and then you know your cables are not your bottleneck.

 

[TheLostSwede]

"Supports" is not a concept that works very well in the physical world.

CAT5e is designed for 1Gbps. But in practice, you'll see speeds in excess of 1Gbps if you put equipment on them (or less, depending on local conditions). As a physical device, you can affect the speed of a connection by twisting or stepping on cables sometimes. CAT6 is designed for 10Gbps. That means its noise characteristics are low enough that 10Gbps can be "heard" through to the other side.

Ethernet (including 10Gbit Ethernet) is designed to auto-negotiate. At the beginning of any connection, there's a period where the computer tests what speed its able to get to. If its able to get to 2.5 Gbps, then it will use that (even if its a 10Gbit card). The physical wire may be too noisy. Even a CAT6 cable could be installed wrong and perform poorly. Twist the cable the wrong way, and the stress/strain on the copper inside the cable can degrade the signal (so now its "too noisy" for 10Gbps, but maybe 5Gbps or 1Gbps will work). Or twists / kinks in the wire, or maybe broken shielding, or poorly installed heads.

In contrast, if you really baby the cable: keep it short, have high quality connections, do a good job installing the head... then even CAT5e can do 10Gbps (even though its not designed for it). In any case, working with higher quality cables (ex: CAT6) would help remove one possible issue.

I mean, it was designed for 1Gbps, but as has been proven, it works perfectly fine for 2.5Gbps, as that is what the 2.5GBASE-T standard was designed to use.
If you have some good cables, it can also work for 10Gbps. I was using that for a while, by mistake, but it was a 2 metre run.

As for Cat 6, that's only up to 55m of 10Gbps and now 100m of 5Gbps, but you need Cat 6A for longer runs at 10Gbps.

Worth adding is that the IEEE states that cat5e is a "reliable medium for support of 2.5Gbps and 5Gbps" for up too 100m
Crosstalk and interference in general however could still cause problems.

10.1109/IEEESTD.2018.8457469
126.7.2

I have also seen a lot of people over the years successfully running 10Gbps over cat5e up to 50 meters, although results vary depending on environment and quality of cable.

Point being, cat5e can without a doubt handle more than just 1Gbps.

Actually, 5Gbps needs Cat 6 to reach 100m, but it will work with Cat 5e up to 55m.

 

[Bill_Bright]

Everybody is talking about "in theory" and while that may be right, it is not helping the OP who is working in the real world - a place where "in theory" frequently does not jive with reality.

He need to try CAT6 cables, designed for his switch and NICs, and see what happens.

 

[ERazer]

If you gonna spend money on 10g connection why cheap out on cable?

as already mentioned get cat6 cables

 

[evernessince]

As others have stated, swapping cables should be your first option.

 

[Eric_Cartman]

No. That is wrong. As I noted, CAT5e only supports up to 1Gbps. This is easily verified with our friend Bing Google.

Cat5e vs. Cat6 (cablematters.com)

How to Choose an Ethernet Cable | Digital Trends

Ok.

I will buy new cables and see if that helps.

If you gonna spend money on 10g connection why cheap out on cable?

as already mentioned get cat6 cables

It was just the cables I already had.

Since I read from several places that Cat5e works with 10G just fine I figured I wouldn't bother replacing them.

 

[Aht0s]

If you could plug those two computers together and run iperf, I think you will be able to tell for sure if your cables are the problem or it is something else.
Like someone mention earlier, try working your way up to the problem. Direct connection then add other hardware once you confirm it works the way it should be.
I think is ideal to figure the problem and then you can decide on the purchase.

 

[Bill_Bright]

Since I read from several places that Cat5e works with 10G just fine I figured I wouldn't bother replacing them.

Well, CAT5e does "work with" 10Gbps. But it is the same as USB 2.0 devices "work with" USB 3.0. Or slower RAM "works with" faster RAM.

But as is the case with many things, the "weakest" or in this case, the "slowest" link sets the performance limit for the entire system.

 

[efikkan]

I have store bought 6ft Cat5e cables.

Assuming these are in excellent physical condition, these shouldn't be the problem.
If you haven't already, try hooking up one cable directly between the machines (as suggested by others), if you still only get 2.5G with two or three different Cat5e cables, then the cables are not the problem.

While I have no experience with the TRENDnet 10G card, I do suspect the drivers. Have you tried various driver versions?
I would even try to boot up Ubuntu from a USB stick to see how it detects the card, just to see if it successfully detects the proper speed, but this may be outside your comfort zone.

BTW; Are you able to return these cards if you can't make them work? (And then buy some Intel x550-t1 cards instead…)

 

[Eric_Cartman]

Assuming these are in excellent physical condition, these shouldn't be the problem.
If you haven't already, try hooking up one cable directly between the machines (as suggested by others), if you still only get 2.5G with two or three different Cat5e cables, then the cables are not the problem.

While I have no experience with the TRENDnet 10G card, I do suspect the drivers. Have you tried various driver versions?
I would even try to boot up Ubuntu from a USB stick to see how it detects the card, just to see if it successfully detects the proper speed, but this may be outside your comfort zone.

BTW; Are you able to return these cards if you can't make them work? (And then buy some Intel x550-t1 cards instead…)

I tried connecting the computers directly with a cable and get the same results.

I bought everything off ebay so returns will be a hassle.

But I'm also not going to spend $300 per network card to buy Intel cards either.

I used the driver that Windows loaded as well as the latest driver from trendnet's website.

I guess since people are saying Cat5e isn't capable of 10G I'll just wait for the Cat7 cables I ordered to arrive.

They are supposed to be here tomorrow.

 

[Bill_Bright]

Keep us posted.

 

[efikkan]

I guess since people are saying Cat5e isn't capable of 10G I'll just wait for the Cat7 cables I ordered to arrive.

They are supposed to be here tomorrow.

The best of luck.

And as a tip for you and others; don't pay extra for Cat 7 over Cat 6A, as there is no Cat 7 standard for RJ45(8P8C). Cat 7 is a special standard for datacenters using TERA or GG45 plugs. Whenever you see "Cat 7" terminated with RJ45 plugs, these are just Cat 6/6A cables with a markup. Cat 6A is comparable to the specs of Cat 7 for datacenters. The successor of Cat 6A is Cat 8.1, but that only becomes relevant with 25Gbps Ethernet.