10Gb network/connection

[repman244]

Ok, I need somebody that is more into 10Gb networking than I am. Right now, all of my computers are on a gigabit network.
What I would need is to connect my main PC to my old storage server with something faster than gigabit connection.
The reason behind that is I moved my SAS tape drive to my storage server and would like to do tape backups directly via network (without using HDD in the storage server). For the old LTO-2 tape drive it's not a problem since it's slow, but the newer LTO-4 needs a transfer speed of 80 to 120 MB/s - and that old onboard gigabit NIC cannot keep up - not to mention if I want to run both tape drives at once.

So the best solution for me would be to install two 10Gb NIC (one in the server and one in my PC), and connect them directly. (I would prefer SFP+).
I was looking at older used server 10Gb NIC, like HP NC523 or NC530 which are cheap and easy to get from ebay. But before I buy anything, would my setup even work, due to direct connection?
Also which cable I would need to use since I see that there are different SFP cables (the required distance is around 3m)?

 

[Solaris17]

Dont go with the HP cards, they are Qlogic chips and have terrible driver support and quirks.

Go with mellonox connextX 2 or 3 cards. They are just as cheap.

Next you need to decide how to connect them. Or rather what you are willing too do. 10G RJ45 transceivers via Cat6? fiber? DAC cable?

Finally, if it works all depends on your software. Its totally possible on any OS to assign two different addresses to a NIC. I do that on my servers. Thats how my 10GB network speaks. Using 10. and 172.

 

[repman244]

Ok, will check those out, if they are cheap in EU as well - hopefully yes.

If I use DAC cables, I don't need transceivers or do I? I'm trying to keep it simple and withing a reasonable budget.

My PC is running Win 10 and on the server it's Win Server 2008 R2, if the backup software will work like it should I cannot know - but I think there should not be problems.

 

[Solaris17]

DirectAttachCables have the transceiver built
In.

 

[repman244]

Checked the Mellanox cards and they are around 75€ vs 22€ for the HP - but I also saw reports of the HP running really hot - not a problem in the server due to plenty of airflow but it is a problem in my PC.
But I checked what seems to be a newer model the 530SFP and that one is at 9W which is ok, but it's still using qlogic chip. Is there any other alternative which is proven to have good drivers and support?

 

[Solaris17]

Intel or chelsio

see if there are any Broadcom chips available to you, that’s what mellonox runs on.

 

[repman244]

Found HP 560SFP for 50€ each, and that one is running a intel controller so I guess it should be better than the qlogic chips. Don't know about the thermals/power consumption.

Anyway after a good sleep over my issue I remembered I don't have any pci-e x8 slots free on my PC motherboard. All I have left is the two x1 slots (which would have enough bandwidth for my use case but I won't be able to use the card to it's full potential. I don't even know if the card will even work in an x1 slot.

 

[repman244]

Just to report back, bought 2x Intel x520-DA2 cards. Got them for the same price as HP 560sfp (it's the same NIC just HP branded).
Also got a DAC cable incoming, hopefully everything will be here by the end of the week.

I just hope that the CPU in my old server will be fast enough to sustain speeds for my tape drive, otherwise I'll have to design an enclosure for the tape drive to attach it to a more modern DL380G6 that I have.

 

[Octopuss]

I am checking Ebay for good deal on either Chelsio T520(-SO)-CR, Mellanox ConnectX-4 or Intel T550 (keep dreaming, these are expensive as hell) semi-regularly myself.
Of course this would be a pricey network upgrade as I need a switch as well and another 10Gb card for my PC (screw the rest of the house, I'm impatient and need speed bump the most )

 

[repman244]

Yes some options even used are still quite expensive, also I didn't even look at RJ45 NICs due to higher price and much higher power consumption/heat output.
Much cheaper to go with sfp+ (used sfp NICs are all server offloads so there is ton of them for cheap).
Bought the cable new from fs.com with Intel coding.

 

[Octopuss]

Yup, I think all those cards I am looking for have SFP+ ports, and I'm waiting for a price drop on TP-Link T1700G-28TQ switch.

 

[repman244]

Got everything delivered today, the only issue I have/had is that the cards are HP branded 560SFP+ (which technically are Intel X520-DA2) but the description and the picture on ebay said otherwise.
I'll contact the seller and warn him about false description.
Despite that, I installed the cards, connected the cable, installed the drivers, configured the network and everything is ok for now.
Still doing some adjustments of the settings to optimize them for tape backup and to get the required performance on this ancient server I have. The main issue is CPU performance (running 2x Xeon Nocona at 3GHz and 4GB of DDR ECC RAM).

File copy was around 300MB/s to RAM due to slow SCSI disks on a quick dirty run I tried.
I will try to get some iperf or similar benchmark going tomorrow or during the weekend to see the maximum performance I can achieve.
And of course some pictures

 

[Toothless]

Jesus those chips are old. Might be worth considering a cheap upgrade to the 26xx series of Xeons for the storage speed bump.

 

[Solaris17]

Not really, no real Gaines by upgrading the CPU at all. Iperf will show for sure but don’t forget you might need to cool the NICs
Oof mistook the arch those are brutally old. Do you maintain them for the sake of the tape drives? I’d be interested in iperf anyway

 

[repman244]

Yes the CPU is very old, it's around P4 era but they are 64-bit.

The reason I didn't replace it yet because it just works - no extra funding is needed to keep it running. If I decide to get something used, but more modern I will definitely have to invest into it (the servvers I have I got for free and fully functional).
I also have a DL380G5 and a DL380G6 which I have a plan to use them as a replacement of this ancient ML. The only thing I need to design is an extra case with a fan to put the tape drives in to provide cooling, since I cannot fit them in a 2U chassis.

Ok so I did a quick and dirty run of iperf (didn't really do any optimization for it, just left the settings as I had for tape backup).

Yes the cards are not used to their full potential but when eventually I replace this server I will be able to use these same cards. For now they are doing their job just fine.

 

[Solaris17]

Is this machine to machine though?

not machine to storage?

well I guess you aren’t doing it to an appliance so answered my own question

 

[repman244]

Machine to machine yes, for tape the maximum I need is around 120MB/s when it's at 2:1 compression, otherwise it's at 80MB/s.
Of course if I change to a newer LTO tape drive I will have to change the server also due to much higher speed demand. For LTO-8 you need around 900MB/s sustained when at 2:1 compression if you don't want any slow downs.