Marky's Network Attached Storage (NAS) 2024 Build Series
- 01 - Why am I building a new NAS?
- 02 - What NAS am I building?
- 03 - Case Selection
- 04 - Hardware Selection & Pricing
This post I going to cover my new networking infrastructure to support 20 Gbit Ethernet.
Previous to this upgrade, I had a 1gbe 24 port POE switch for the house. I have ethernet run into my office, living room, and playroom all terminated to a patch panel in the basement. In the office, I have been running a 4 2.5gbe 2 10gbe switch for my NAS/Workstation, and Virtualization Servers.
This setup allowed me to have 10gbit connection to my NAS, while allowing my Internet and Proxmox servers to run at 2.5gbe. All other devices were limited to 1gbit. This is something I've wanted to change but there isn't any hardware on the market that handles what I want without getting multiple switches.
Requirements
- 4+ SFP+ 10gbit
- 2+ RJ45 10gbit
- 12+ 2.5gbe
- POE+
If this doesn't make sense, I basically want to have at least 4 10gbit fiber ports for my servers. These offer slightly lower latency 10gbit connection but most importantly it runs at lower power. As my office is multiple floors from the basement where I am housing the servers, I either need to run fiber or use my existing rj45 wiring. Finally, I want all other ports to be 2.5gbe with Power over Ethernet support.
Without spending thousands of dollars, this configuration is impossible to do in one switch. I found one switch that was very close, 2 RJ45, 2 SFP+, 16 2.5gbe with Poe. I almost got this switch, it is made by QNAP and is almost what I want. With this switch, I would be limited to 2 SFP+ for my primary NAS, but would need to use 2.5gbe for my 2nd NAS if I want to use two RJ45 10gbit for my workstation.
In this setup, I want to use two 10gbit ports for both my workstation and my servers to be able to use 20gbit speeds. Now typically this doesn't allow one device to use more than one network card, so I wouldn't get any speed improvements. That is until Microsoft released SMB multichannel support. This allows you to use Samba/SMB shares with multiple TCP connections. This allows you to take advantage of multiple network cards on the same network. This allows me to break the theoretical max speed of 10gbit of 1100MB/s to around 2200 MB/s.
pv ~/Desktop/tempfile > tempfile
5.00GiB 0:00:02 [2.43GiB/s] [=====>] 100%
As I couldn't find a single switch that handled what I wanted, I went with three switches.
- Sodola 12 Port 10Gbe SFP+ Managed Switch
- Sodola 8 Port 2.5gbe 1 Port SFP+ Unmanaged POE Switch
- Sodola 8 Port 2.5gbe 1 Port SFP+ Unmanaged POE Switch
Basically a single core backbone 10gbe switch with 12 10gbe SFP+ ports. Then two 8 Port 2.5Gbe with 10Gbe SFP+ uplinks to the backbone switch. Servers are then plugged in directly to the backbone switch using 10Gbe SFP+.
For my workstation to reach the backbone switch through the existing Cat 6e wiring, I installed two RJ45 transceivers into the backbone switch. This gives me 20Gbe connectivity between my workstation and the primary and secondary NAS. My virtualization servers will be connected to the two auxiliary 2.5gbe switches as they only have dual 2.5gbe nics anyway.
This configuration gives me 10 usable 10Gbe ports (2 used to uplink the 2.5Gbe switches) and 16 2.5Gbe POE enabled ports.
Cost
- 10Gbe SPF+ Switch - $220
- 2x 2.5Gbe POE Switch - $100
- 2x RJ45 Transceivers - $35.99
- 2x 0.25m SFP+ DAC cables - $8.79
- 2x 1.5m SFP+ DAC cables - $12.88
Total cost: $535.22
The 10Gbe RJ45 cards cost me $23 and the 10Gbe SFP+ cards cost me $15 each.
I was looking at the Ubiquiti line for something that would meet my criteria and this was the closest I could find.
I would only have two usable 10Gbe ports, and 8 2.5Gbe ports for almost twice they money. I could go to the 48 port version and squeeze in a few more critical ports.
Finally, another option is their RJ45 10Gbe switch which is probably the most attractive option, but still doesn't have POE and would require all RJ45 and the 25Gbe would be unusable without running fiber two floors.
If I wasn't trying to reduce power consumption, I'd probably opt for the 24 port 10Gbe RJ45 and just use a dedicated 2.5Gbe POE switch for the WIFI/Cameras. If it was only one POE device, I could get away with an injector.
We are in a funny place right now with switches. 2.5Gbe and 10Gbe are becoming more mainstream outside of large business, but the switch options are still only 4 - 8 ports typically. There are some really cheap 2 port 10Gbe options for $100-200. For about $400 you can get an 8 Port 10Gbe. If you want POE and 2.5Gbe ports, you are kind of out of luck without multiple switches.
Power usage
With four SFP+ ports and two RJ45 ports on the 10Gbe SFP+ switch, I am pulling 18W. The two 2.5Gbe switches pull 4.25W each.
If I yoink a single RJ45 10Gbe from my workstation, the main switch drops from 18W to 15.5W. So the transceivers are using 2.5W each, and a similar 2-3W drop on my workstation. That's about 5-6W or $10-12/yr in electricity per RJ45 connection. Compared to less than half a watt for the SFP+ connections, that's a big difference.
Posted Using InLeo Alpha
