Go Back   OC3D Forums > [OC3D] Hardware & Software > Networks & Security
Reply
 
Thread Tools Display Modes
 
  #1  
Old 16-05-19, 09:01 PM
Diablo Diablo is offline
OC3D Elite
 
Join Date: Aug 2008
Location: London
Posts: 1,531
10Gbps networking on the (relative) cheap

Many enthusiasts find that even with their NVME SSDs, their steam drive stretching into the terabytes and unlimited Netflix there is still a need for some network attached storage. Maybe it is to run their music in the living room, hold their photo collection, or stream their video collection they “acquired”.
Then they build their NAS, or maybe even a mini server, find a RAID card on ebay (or Amazon warehouse deals – pro tip) and fill it with disks. It is at this stage they connect to it and find it is really, really slow. A full back up takes hours, if not days, transfer rates are at best around the 100MB/s mark.
No matter, they connect via network cable, and to no-one’s surprise – it is still slow. So now they start to look for faster ethernet – and then run at the price.

When I started to need my own server, I had got a RAID card, 5 1TB disks, and had run out of drive bays in my PC. At the time a 10Gbps card was £250+ and I would have needed two. So I shelved the idea.
Then two years ago Aquantia brought out their 10Gbps network controllers. They can be found in all manner of cards, including the Asus 10Gbps NIC, and some of their high end motherboards. Before I continue; let me state this is “on the relative cheap” – this is about new hardware, multiple computers and network attached storage capable of 300MB/s speeds. If you have everything in one system, rely on WiFi for most of your needs, or think £300 is too much for a motherboard, you would find better value elsewhere.
For the people left with me, the components you will need are:
• A pair of 10Gbps network cards such as ASUS XG-C100C ~£99
• Cat 6A cable ~£1
If you have a thunderbolt enabled laptop (this is not USB-C + Displayport, and can be identified by the lightning logo), try a thunderbolt to RJ45 10Gbps adapter (expect to pay £180 ish)
Connect them up, set the IP address manually in IPV4 settings, enable jumbo packets (e.g. 9046 bytes) and job’s a good’un. You should have transfers limited by the speed of you disks / SSD (or network for NVME to NVME transfers).
Alternatively use an ASUS XG-U2008 Unmanaged 2-Port 10G/8-Port switch to connect your computers to the main house network.

But I have more than two computers which need 10G connections…here we start to get expensive. Maybe you want to connect more than a pair of computers, and then use link aggregation to connect computers at 2Gbps?
I went down the route of an 8 port 10Gbps switch; the price difference between 5 and 8 port is very small. Furthermore, I prefer to have a manged switch, so I can pick the jumbo frame size, and have the option to create V-Lans etc. This one is a matter of preference, but I had good luck with the XS708T from Netgear (about £500).
Finally I decided to upgrade my router to allow the switch to connect to the router, and then connect the router to my existing 24 port smart switch via link aggregation. The only router for a reasonable price I could find with an SFP+ connection or 10Gbps connection was the R9000 from netgear. The downside is that this has 802.11ad wireless, which is not really current anymore versus 802.11ax.
Finally to connect these, use a direct attach copper SFP+ cable (£25) – you can use fibre, but this is properly expensive (often >£200 for the cable)

This ends up being quite expensive, but for less £1000 you can have a multi-computer 10G network at home. You will see transfer rates in excess of 125MB/s (typically I see 220-250MB/s on the mechanical drives, and for SSD to SSD transfers, I can see >600MB/s.
Finally to hold this all in place, I bought a Lack (£9) from Ikea (a coffee table). This is 19” between the legs, so is perfect for 19” rackmount kit. As a bonus my stereo amp sits perfectly under the switches.
Downsides:
• Price (obviously)
• Plex support – if you enable jumbo frames on your plex server, it will not work correctly; the trick is then to run a second network card, bind the port for the plex server onto the second card (try using Bind IP) (also same problem for BlueIris CCTV)
• Multiple connections may require you to play around with the network metrics, particularly if you are direct connecting two computers
I would not recommend doing this unless you have:
• A fast SSD
• Loads of storage and a need to transfer data quickly
• A decent monitor, GPU, keyboard, mouse etc. – this is because upgrading these will make more of a difference to your quality of life
My kit:
• Asus Rampage VI Extreme
• Asus XG-100C
• Server with 5960X + adaptec 5805 and adaptec 8805 RAID cards (5 x 1TB RAID 5, 3 x 4TB RAID 5, 8 x 10TB RAID 5, 3 x 1TB SSD RAID 0)
• Netgear R9000
• Netgear XS708T 8 port 10G switch
• Netgear JGS524E 24 Port Gbps switch
• TP LINK 24 Port Gbps POE+ switch + 3 POE cameras
• Lack from Ikea for a lack rack
A quick update – it is now possible to do this for slightly cheaper with a cheaper 5 port switch. My caution to this is that a five port switch gives very little upgrade and expansion room. When you have a laptop with thunderbolt, a pair of fast computers and a router, you will find yourself with exactly one port left…
Is this fun – yes; does it make life easier to run big backups, run a steam library on a NAS for the house, and hold 2TB of RAW files across the house – yes; does it enable all users to hit the server as fast as they need; yes. Finally; does it make your geek cred higher – oh yes!

__________________

i9 7980XE 4.6GHz - EK Supreme EVO - Asus - X299-RVIE - 32GB 3677MHz RAM - 2 x 2080Ti EK Cu Block - Creative Titanium Pro - Tower - Samsung 970 Pro 512GB - Plextor 240GB PCIe SSD - 6TB HD - Dual Loop
Reply With Quote
  #2  
Old 17-05-19, 10:51 AM
Kei's Avatar
Kei Kei is offline
Advanced Member
 
Join Date: Apr 2013
Location: South Wales
Posts: 490
Just done this myself. I stayed away from copper and stuck with fibre. It can be done dirt cheap using OM3 multimode and the good news is that unlike copper, Fibre is perfectly capable of 40 and 100Gbps upgrades in the future so the infrastructure doesn't need to change. It is also nowhere near as distance limited. Fibrestore (FS.com) are really cheap and excellent quality. Direct attech copper (DAC or Twinax) is even cheaper but obviously limited by distance.

Mikrotik make some very cheap managed switches with multiple SFP+ ports for 10Gbe.
CRS305-1G-4S+IN = 4x 10Gbe + 1x 1Gbe = £120
CRS309-1G-8S+IN = 8x 10Gbe + 1x 1Gbe = £210
CRS317-1G-16S+RM = 16x 10Gbe + 1x 1Gbe = £310

There is also a TP-link that's pretty well regarded.
TP-LINK T1700G-28TQ = 24x 1Gbe + 4x 10Gbe = £250

SFP+ network cards can be had dirt cheap, particularly those that are OEM branded by Dell, IBM, HP, oracle and so forth. They are also mostly dual port so could be LAGed to certain devices to nearly double the bandwidth to 20Gbe.
Mellanox Connect-X 2 = £30-40
Chelsio N320 = £30-40
Solarflare SFN5122F - £30-50
Solarflare SFN6122F - £40-60
Solarflare SFN7122F - £50-80
Mellanox Connect-X 3 = £50-80
Intel X520 = £70-100
Chelsio T520 = £100-150
Intel X710 = £120-150

The way I went:
Juniper EX3300-24P - £360 - new open box
Intel X710 x2 - £250 - I could have gone much cheaper here but I wanted a current model PCIe 3.0 card
SFP's x6 - £90
Fibres x3 - £17 (2m @ £3, 10m @ £6 & 15m @ £8)

If I ever need to expand the 10G section, I can get an 8 port SFP+ swtich and LAG two ports from the juniper.
__________________
Gigabyte Aorus X399 Gaming 7 ⁞⁞ Ryzen TR 1920x ⁞⁞ EK Blocks + αcool XT45 rads ⁞⁞ AMD Radeon Vega 56 ⁞⁞ 32GB 8 Pack 3200C14 ⁞⁞ Enthoo Primo
Seasonic Prime 750W ⁞⁞ Polaris 961 256GB ⁞⁞ 850 Pro 512GB ⁞⁞ M500 240GB ⁞⁞ WD Se 4TB & 1TB x2 ⁞⁞ Samsung C27HG70 + Dell 2408WFP
Ducky Shine 5 ⁞⁞ AJA Kona LSe ⁞⁞ RME HDSPe AIO ⁞⁞ Marantz AV7701 ⁞⁞ Tannoy SRM 10B ⁞⁞ AKG K601 ⁞⁞ Beyer DT770

Reply With Quote
  #3  
Old 17-05-19, 06:36 PM
FTLN FTLN is offline
OC3D Elite
 
Join Date: Feb 2012
Location: France
Posts: 1,492
Quote:
Originally Posted by Kei View Post
Just done this myself. I stayed away from copper and stuck with fibre. It can be done dirt cheap using OM3 multimode and the good news is that unlike copper, Fibre is perfectly capable of 40 and 100Gbps upgrades in the future so the infrastructure doesn't need to change. It is also nowhere near as distance limited. Fibrestore (FS.com) are really cheap and excellent quality. Direct attech copper (DAC or Twinax) is even cheaper but obviously limited by distance.

Mikrotik make some very cheap managed switches with multiple SFP+ ports for 10Gbe.
CRS305-1G-4S+IN = 4x 10Gbe + 1x 1Gbe = £120
CRS309-1G-8S+IN = 8x 10Gbe + 1x 1Gbe = £210
CRS317-1G-16S+RM = 16x 10Gbe + 1x 1Gbe = £310

There is also a TP-link that's pretty well regarded.
TP-LINK T1700G-28TQ = 24x 1Gbe + 4x 10Gbe = £250

SFP+ network cards can be had dirt cheap, particularly those that are OEM branded by Dell, IBM, HP, oracle and so forth. They are also mostly dual port so could be LAGed to certain devices to nearly double the bandwidth to 20Gbe.
Mellanox Connect-X 2 = £30-40
Chelsio N320 = £30-40
Solarflare SFN5122F - £30-50
Solarflare SFN6122F - £40-60
Solarflare SFN7122F - £50-80
Mellanox Connect-X 3 = £50-80
Intel X520 = £70-100
Chelsio T520 = £100-150
Intel X710 = £120-150

The way I went:
Juniper EX3300-24P - £360 - new open box
Intel X710 x2 - £250 - I could have gone much cheaper here but I wanted a current model PCIe 3.0 card
SFP's x6 - £90
Fibres x3 - £17 (2m @ £3, 10m @ £6 & 15m @ £8)

If I ever need to expand the 10G section, I can get an 8 port SFP+ swtich and LAG two ports from the juniper.
Running the CRS305-1G-4S+IN on my home lan with a few Chelsio T520-SO-CR dual 10gb NICS (one in freenas, one in my workstation and another in my pfsense box). The Chelsio cards can be had on ebay for around 50-60 euros and drivers are always getting updated for windows and pretty much built into all Linux kernels that you can name.

The Mikrotik is great for pure switching but is a but flaky when it comes to some advanced configurations.
__________________
Intel Core i9 9900kK @ 5.00GHz @1.275v - Asus ROG Maximus Hero - Corsair 3000mhz CL16 LED 4x16GB SKHYNIX - Thermal Take View Glass - PM961 1TB x 2 RAID0 - Nvidia GTX 1080 Ti - Corsair H110i - Corsair Commander Pro - Asus ROG Swift PG279Q 2560x1440P G-Sync 165Hz
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump










All times are GMT. The time now is 07:14 AM.
Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.