Budget storage server

[Kei]

Installed plex and hoping to configure it so that I can use it to stream old recordings lifted off my humax pvr. It works perfectly to my panasonic plasma but not got it to talk with vlc via upnp yet. (a non issue really as i can play the files direct via the shared storage) Still need to get the FTP server side of things running though.

6 disks finally migrated and fully initialized. Performance is pretty good. Intel expander should be here soon, though I have no need for it yet as I don't have enough disks to require it.

 

[Kei]

Tested out FTP today and that works perfectly too, uploading to it at ~8-10mb/s and downloading from at ~2-3mb/s. Need to tinker with the settings a bit to make sure it is secure.

Intel SAS expander arrived today. May well look into fitting it over the weekend.

 

[Kei]

The intel sas expander is now fitted and working perfectly. I was expecting to have to rebuild the array from scratch but the controller picked up the virtual drive as if nothing had changed and is working exactly as before. A minor headache came when I tried to fit the card as it requires a pcie 4x sized slot and my gigabyte board only has two 16x slots and three 1x slots. I've had to use the vertical mount on the case for it until I can come up with something suitable to help neaten the cabling back out. (considering a 1x to 16x riser card for low profile cards) I also crimped the connectors for the 38mm NMB-MAT fan to aid cooling as I had to loose the side panel fan to fit the expander.

 

[Kei]

Added the third breakout cable and adjusted the fan power connections so that they are a tad quieter. Also installed the proper nvidia drivers which has finally cured the fan on the quadro running at full tilt all the time. It's finally reasonably quiet now. Got 7 disks connected giving me 12TB, still got room for another 5. (I've put some spares in for now to simulate a full case so i can monitor temperatures)

 

[Kei]

A full network rebuild has been performed as my old tp link 1043nd router has been requiring a prod every couple of days as it would just freeze up and stop the network. I decided that i was up for a challenge and went with a cisco 1921 ISR, an hp 1910-8G managed switch and a D-link DAP-2695 wireless access point. It took me about 4 hours to configure the router sufficiently that i could access the internet. I've spent a further hour this evening configuring the dhcp pools so that some of the main addresses are bound to the hardware ids. I still need to work out how to configure NAT/PAT for port forwarding so that most software will work as cisco IOS based devices lack upnp and NAT-PMP so the ports cannot be opened via the software itself.

 

[YouWhat]

A full network rebuild has been performed as my old tp link 1043nd router has been requiring a prod every couple of days as it would just freeze up and stop the network. I decided that i was up for a challenge and went with a cisco 1921 ISR, an hp 1910-8G managed switch and a D-link DAP-2695 wireless access point. It took me about 4 hours to configure the router sufficiently that i could access the internet. I've spent a further hour this evening configuring the dhcp pools so that some of the main addresses are bound to the hardware ids. I still need to work out how to configure NAT/PAT for port forwarding so that most software will work as cisco IOS based devices lack upnp and NAT-PMP so the ports cannot be opened via the software itself.

Think the Budget storage server went out the window a short while ago :lol:

Time to rename the thread The Not So Budget Storage Server (Now) :lol: but I suppose it still could be more expensive if you tried to

Seriously though, that all looks great what you have done, and I remember the headaches I used to have configuring the Cisco IOS stuff a few years back (manage to fix thing A thing and it creates problem on thing B, the damn things are too clever for their own good sometimes!! so you have my sympathies on that.

 

[Kei]

Well to be fair, I've got most of this equipment fairly cheaply or free, which follows the idea of it being budget. My idea of cheap may differ from others though.

The cisco router is behaving well for the most part. I certainly get better performance from it vs the tp link. Very pleased there. (used to get low-mid 74 down and high 17- low 18 up, ping is about the same)

Config wise, it's taken a while to get the access list to behave itself but i seem to have gotten port forwarding to behave itself. I got FTP and SSH to my linux server to work after some headaches. I just need to add some more ports for teamspeak and steam games so multiplayer works and i think it'll be sorted. I'm not sure if I've covered all bases on the deny list but I'm hopeful that it's close.

!
! Last configuration change at 12:44:44 UTC Tue Nov 11 2014
version 15.2
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Cisco-1921
!
boot-start-marker
warm-reboot
boot-end-marker
!
!
enable secret 5.
enable password 7 
!
no aaa new-model
!
ip cef
!
!
!!!!!!DHCP server config
ip dhcp excluded-address 192.168.0.1
ip dhcp excluded-address 192.168.0.50 192.168.0.51
!
ip dhcp pool main_dhcp_pool
 network 192.168.0.0 255.255.255.0
 default-router 192.168.0.1
 dns-server 194.72.0.114 213.120.234.46 8.8.8.8
!
ip dhcp pool Kei-PC
 host 192.168.0.2 255.255.255.0
 client-identifier 01bc.ee7b.98e8.1e
 client-name Kei-PC
!
ip dhcp pool Humax
 host 192.168.0.4 255.255.255.0
 client-identifier 01dc.d321.8169.a2
 client-name Humax
!
ip dhcp pool Kei-NAS
 host 192.168.0.3 255.255.255.0
 hardware-address 0024.1d7d.1ef9
 client-name Kei-NAS
!
ip dhcp pool Server
 host 192.168.0.100 255.255.255.0
 hardware-address 0024.1d7d.1f09
!
!
!
ip domain name WesNet
ip name-server 8.8.8.8
ip name-server 194.72.0.114
no ipv6 cef
multilink bundle-name authenticated
!
!
!
license udi pid CISCO1921/K9 sn FCZ1810C0V2
!
!!!!!Archive every 7 days or on writing config
archive
 path ftp:/xxx/Cisco-1921
 write-memory
 time-period 10080
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
 no ip address
 shutdown
!
!
!!!!!! Note that Ge0/0 has no IP address
interface GigabitEthernet0/0
 no ip address
 ip tcp adjust-mss 1452
 duplex auto
 speed auto
 pppoe enable group global
 pppoe-client dial-pool-number 1
!
!
!!!!!! This is the LAN side
interface GigabitEthernet0/1
 ip address 192.168.0.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ip tcp adjust-mss 1452
 duplex auto
 speed auto
 no mop enabled
!
!!!!!! The ISP's given IP address will be configured via d1
interface Dialer1
 ip address negotiated
 ip access-group 101 in
 no ip unreachables
 ip mtu 1492
 ip nat outside
 ip virtual-reassembly in
 encapsulation ppp
 dialer pool 1
 ppp chap hostname bthomehub@btbroadband.com
 ppp chap password 7 1415060303092F23312A1337361115190205545856571A0D0C15
 ppp pap sent-username bthomehub@btbroadband.com password 7 0111120C54060307344E6E0B0D07051D0A08062B252066303A2F
!
ip forward-protocol nd
!
ip http server
no ip http secure-server
!
!!!!!! This is the dynamic PAT between Dialer1 (WAN) interface IP address 
!!!!!! and local IP addresses within ACL 1
ip nat inside source list 1 interface Dialer1 overload
ip nat inside source static tcp 192.168.0.3 20 interface Dialer1 20
ip nat inside source static tcp 192.168.0.3 21 interface Dialer1 21
ip nat inside source static tcp 192.168.0.3 22 interface Dialer1 22
ip route 0.0.0.0 0.0.0.0 Dialer1
ip route 192.168.0.0 255.255.255.0 GigabitEthernet0/1
!
access-list 1 remark INTERNET-ACCESS
access-list 1 permit 192.168.0.0 0.0.0.255
!
access-list 101 remark DENY FAKE IPs
access-list 101 deny   ip 192.168.0.0 0.0.255.255 any
access-list 101 deny   ip 172.16.0.0 0.15.255.255 any
access-list 101 deny   ip 10.0.0.0 0.255.255.255 any
access-list 101 deny   ip 127.0.0.0 0.255.255.255 any
access-list 101 deny   ip 255.0.0.0 0.255.255.255 any
access-list 101 deny   ip 248.0.0.0 7.255.255.255 any
access-list 101 deny   ip 224.0.0.0 7.255.255.255 any
!
access-list 101 remark DENY SPOOFING IPs
access-list 101 deny   ip host 0.0.0.0 any
access-list 101 deny   ip host 255.255.255.255 any
!
access-list 101 remark DENY VULNERABLE PORTS
access-list 101 deny tcp any any range 135 139 log-input
access-list 101 deny udp any any range 135 139 log-input
access-list 101 deny tcp any any eq 2000 log-input
access-list 101 deny tcp any any eq 2001 log-input
access-list 101 deny tcp any any eq 6000 log-input
access-list 101 deny tcp any any eq 6001 log-input
access-list 101 deny tcp any any range 5900 5910 log-input
access-list 101 deny tcp any any range 5800 5810 log-input
access-list 101 deny tcp any any eq finger log-input
!
access-list 101 remark DENY TRACEROUTE OUTSIDE
access-list 101 deny udp any any range 33400 34400 log-input
!
access-list 101 remark ALLOW ACCESS FOR SERVER FTP & SSH
access-list 101 permit tcp any any eq 20
access-list 101 permit tcp any any eq 21
access-list 101 permit tcp any any eq 22
!
access-list 101 permit udp any any eq ntp
!
access-list 101 permit icmp any any net-unreachable
access-list 101 permit icmp any any host-unreachable
access-list 101 permit icmp any any port-unreachable
access-list 101 permit icmp any any packet-too-big
access-list 101 permit icmp any any administratively-prohibited
access-list 101 permit icmp any any source-quench
access-list 101 permit icmp any any ttl-exceeded
access-list 101 permit icmp any any echo-reply
access-list 101 deny icmp any any
access-list 101 permit tcp any any established
access-list 101 permit udp any any
!
!
!
!
snmp-server community WesNet RO
snmp-server enable traps entity-sensor threshold
!
control-plane
!
!
!
line con 0
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
line vty 0 4
 password 7
 login
 transport input all
!
scheduler allocate 20000 1000
!
end

This is the output from show ip nat translation. (public ip has been changed from actual)

Cisco-1921>show ip nat translation
Pro Inside global         Inside local          Outside local         Outside global
tcp 78.53.134.234:49996  192.168.0.2:49996     173.194.78.189:443    173.194.78.189:443
tcp 78.53.134.234:50079  192.168.0.2:50079     64.233.167.188:443    64.233.167.188:443
tcp 78.53.134.234:50254  192.168.0.2:50254     74.125.230.229:443    74.125.230.229:443
tcp 78.53.134.234:50283  192.168.0.2:50283     173.194.78.101:443    173.194.78.101:443
tcp 78.53.134.234:50284  192.168.0.2:50284     74.125.230.149:443    74.125.230.149:443
tcp 78.53.134.234:50285  192.168.0.2:50285     74.125.230.143:443    74.125.230.143:443
tcp 78.53.134.234:50385  192.168.0.2:50385     74.125.230.154:443    74.125.230.154:443
tcp 78.53.134.234:20     192.168.0.3:20        ---                   ---
tcp 78.53.134.234:21     192.168.0.3:21        ---                   ---
tcp 78.53.134.234:22     192.168.0.3:22        ---                   ---
tcp 78.53.134.234:42240  192.168.0.3:42240     173.194.78.138:443    173.194.78.138:443
tcp 78.53.134.234:42241  192.168.0.3:42241     173.194.78.138:443    173.194.78.138:443
tcp 78.53.134.234:42910  192.168.0.3:42910     74.125.206.188:5228   74.125.206.188:5228
tcp 78.53.134.234:47758  192.168.0.3:47758     173.194.78.84:443     173.194.78.84:443
tcp 78.53.134.234:54760  192.168.0.15:54760    157.56.124.47:443     157.56.124.47:443
tcp 78.53.134.234:54770  192.168.0.15:54770    157.55.236.49:443     157.55.236.49:443
tcp 78.53.134.234:54772  192.168.0.15:54772    157.56.124.130:443    157.56.124.130:443
tcp 78.53.134.234:54822  192.168.0.15:54822    74.125.230.246:443    74.125.230.246:443
tcp 78.53.134.234:54823  192.168.0.15:54823    74.125.230.247:443    74.125.230.247:443
tcp 78.53.134.234:54825  192.168.0.15:54825    74.125.230.228:443    74.125.230.228:443
tcp 78.53.134.234:54835  192.168.0.15:54835    64.233.166.188:5228   64.233.166.188:5228
tcp 78.53.134.234:54995  192.168.0.15:54995    173.194.78.189:443    173.194.78.189:443
tcp 78.53.134.234:55004  192.168.0.15:55004    173.194.78.102:443    173.194.78.102:443
tcp 78.53.134.234:55039  192.168.0.15:55039    173.194.35.23:443     173.194.35.23:443
tcp 78.53.134.234:55040  192.168.0.15:55040    173.194.78.136:443    173.194.78.136:443
tcp 78.53.134.234:55041  192.168.0.15:55041    74.125.230.134:443    74.125.230.134:443

 

[Dawelio]

I also got remote desktop via vnc to work too.

What monior is that to the right?, looking for a monitor like that, that isn't a full sized 24" widescreen (if that makes sense? :huh?...

 

[Mysterae]

Looking good Kei, interesting read. I liked the idea of using old hardware and re-purposing it, that while overspec'd for what it is, will do the job admirably.

I'm doing the same thing but not to the extent you are! I'll be using Windows 8.1 (so not a server in the true sense) but the use of a SATA III 8 port RAID card is tempting. One difference to your system is that mine'll be connected to the main tv as a media pc too.

It's been a while since you updated here, hows the system running?

 

[camwallace]

I've been using 1&1 for my server outlet. It makes sense for me since I'm only accessing that stuff intermittently. They offer a very good service if you're interested.

 

[Kei]

Looking good Kei, interesting read. I liked the idea of using old hardware and re-purposing it, that while overspec'd for what it is, will do the job admirably.

I'm doing the same thing but not to the extent you are! I'll be using Windows 8.1 (so not a server in the true sense) but the use of a SATA III 8 port RAID card is tempting. One difference to your system is that mine'll be connected to the main tv as a media pc too.

It's been a while since you updated here, hows the system running?

System has been running faultlessly for a few months. The config remains unchanged, still using only 7 disks with 12TB capacity. It was moved into the loft as the noise from the fans was a bit excessive. I know it'll suffer some temperature extremes up there though, particularly in the summer.

I've been using it as a media server with plex, as an ftp server using sftp and as a teamspeak 3 server too. With the port fowarding set up, it all functions from outside too provided the correct user and passwords are known. I also set up fail2ban to deal with the persistent ssh attempts. (Though i may just nix the forwarding when I don't need it)

 

[remember300]

quite a nice little guide and build there, wish i saw it earlier, i whimped out and just got a nas drive but the other features like TS come in to their own when you have something like that.
Glad to see parts being used rather than lying around ... sadly when it comes to old pc cases i cut them up for mods lol ...
but a good way to make the most out of what ya had even if you had to throw a little more in to get it how you want.
and i look forward to seeing how it performs and lasts as you said in the extremes.

 

[Kei]

Well with most of summer gone it seems the server barely even broke a sweat in the loft. Not seen higher than 50 degrees out of it so far, but then I'm hardly taxing the cpu. RAID array is still in a nice healthy state with all 7 WD Se disks behaving as they should. Considering upgrading my LSI 8888elp to something with newer standards that would allow me to attach an LTO tape drive to it and run LTFS. Presently I can run a tape drive but I can't run LTFS as the HBA doesn't support it. Backup software like symantec backup exec would work but it's not as convenient and I'm not even sure you can get it or an equivalent for opensuse.

No idea on the power consumption as yet, need to consult the bills and run a comparison as we're nearly a year down the road.

 

[Kei]

Picked up a replacement GPU for the server. A nice tidy nvidia quadro NVS 310 with a measly 19.5W maximum power draw. The old FX3500 was a greedy card at 95W which was complete overkill for server purposes.
NVS310 by Kyle, on Flickr

I finally got VNC to work properly. It turned out that gnome 3 and even gnome classic DE's wouldn't play nicely with VNC due to hardware rendering being required. I've now switched to using XFCE which works perfectly. (and has the bonus of using less resources) I've also switched the login screen to lightdm which is easier on the hardware too. Space is getting chomped up quite quickly due to the expansion of my plex library. I think my handbrake encode quality might be a tad high as some of the bluray encodes are well above 20GB a piece.
Home Server by Kyle, on Flickr

 

[Kei]

Long overdue update. It ran faultlessly since completion, up until the end of march this year where something occurred with an update that broke samba. It seems that somehow a repository for a newer version of opensuse (42) added itself which then caused an incompatible version to install on my system running 13.1. I probably could have fixed it but it was far easier to just download the latest 42.2 iso and reinstall the os. Wiping the SSD and reinstalling took about half an hour. A few tweaks to the config, added repositories and programs and it was back to the way it was before the muck up. The array remounted to the same mount point and shared via samba and plex media server working as it should.

I went with straight XFCE instead of gnome or KDE this time. Something has made a fair difference as it uses 1GB less ram than before. Might be down to the fact that I've not yet reinstalled lsi megaraid storage manager.

Oddly it suffered from a complete freeze up the other day which it seems was caused by a kernel panic. Will be keeping an eye on it as it's the first crash I've had since I built it. The hardware is getting on a bit now, so I'm praying nothing is gonna go tits up on me.

 

[Kei]

All still running well since 2017. Nice and cool with the current winter temperatures. Starting to look at adding more disks (WD ultrastars as the Se is no longer made) and also considering an array rebuild as the stripe size of 256kb means that some folders stored on there are taking up vast amounts of space unnecessarily.

 

[Kei]

Doing networking upgrades to help consolidate the existing network down.

Cisco VA-DSL-A EHWIC fitted to my 1921 router to remove the ECI openreach unit. First one I received was faulty. (no modem PHY or firmware) The replacement was perfect.



It's nice having stats for the modem.

Controller VDSL 0/0/0 is UP

Daemon Status:		 Up 

			XTU-R (DS)		XTU-C (US)
Chip Vendor ID:		'BDCM'			 'IFTN'
Chip Vendor Specific:   0x0000			 0xB206
Chip Vendor Country:    0xB500			 0xB500
Modem Vendor ID:	'CSCO'			 '    '
Modem Vendor Specific:  0x4602			 0x0000
Modem Vendor Country:   0xB500			 0x0000
Serial Number Near:    FOC16390RTR 1921/K9 15.7(3)M2
Serial Number Far:     5501570926
Modem Version Near:    15.7(3)M2
Modem Version Far:     0xb206

Modem Status:		 TC Sync (Showtime!) 

DSL Config Mode:	 AUTO 
Trained Mode:	G.993.2 (VDSL2) Profile 17a
TC Mode:		 PTM 
Selftest Result:	 0x00 
DELT configuration:	 disabled 
DELT state:		 not running 

Full inits:		2
Failed full inits:	0
Short inits:		0
Failed short inits:	1

Firmware	Source		File Name 
--------	------		----------
VDSL		embedded   	VDSL_LINUX_DEV_01212008

Modem FW  Version:	4.14L.04A
Modem PHY Version:	A2pv6C039t.d24o_rc1
Trellis:		 ON			  ON
SRA: 			 disabled		 disabled
 SRA count: 		 0			 0
Bit swap: 		 enabled		 enabled
 Bit swap count:	 1441			 175
Line Attenuation:	 13.1 dB		  0.0 dB
Signal Attenuation:	  0.0 dB		  0.0 dB
Noise Margin:		  6.5 dB		  6.7 dB
Attainable Rate:	66474 kbits/s		 21532 kbits/s
Actual Power:		  0.7 dBm		  0.6 dBm
Per Band Status:       	D1 	D2 	D3 	U0 	U1 	U2 	U3
Line Attenuation(dB):   8.1	18.8	30.9	0.1	14.8	23.0	N/A	
Signal Attenuation(dB): 10.1	18.6	30.9	0.1	14.6	22.9	N/A	
Noise Margin(dB):       6.5	6.5	6.5	6.4	7.1	6.6	N/A	
Total FECC:		0			 6545
Total ES:		107			 96
Total SES:		15			 13
Total LOSS:		1			 0
Total UAS:		83			 68
Total LPRS:		0			 0
Total LOFS:		10			 0
Total LOLS:		0			 0


 		  DS Channel1	  DS Channel0	US Channel1	  US Channel0
Speed (kbps):	          0	       64159	         0	       18999
SRA Previous Speed:       0	           0	         0	           0
Previous Speed:	          0	       63650	         0	       18999
Reed-Solomon EC:          0	           0	         0	          36
CRC Errors:	          0	        8805	         0	         810
Header Errors:	          0	           3	         0	           0
Interleave (ms):       0.00	        0.00	      0.00	        0.00
Actual INP:	       0.00	        0.00	      0.00	        0.00

Training Log :	Stopped
Training Log Filename :	flash:vdsllog.bin

Looking to consolidate the rest by consolidating the GS108 switch into a unit with POE on board to power the access points and whilst I'm at it add in some 10Gbe ports. I started looking at the MikroTik offerings with the CRS328-24P-4S+RM standing out as the most ideal option. Considering the facilities on offer, it seemed a little too good to be true based on the cost. Reviews for it were scant and what I could find seemed a bit hit and miss. I decided to look for something higher end on the used market fitting the same specs. (8 or more 1Gbe POE+ports, 4 or more SFP+ ports) Narrowed it down to an Aruba S2500-24P, Dell X1052P and a Juniper EX3300. The Aruba is pretty hard to find in the UK and the Dell was likely to cost around £500 for a refurb. The Juniper can be had for £360 for an open box new item, easily covers all my needs and is enterprise proven unlike the mikrotik. It might be a little noisier and a tad greedier on power but it's a known quantity and if I ever have need of its layer 3 features, it'll thoroughly outgun the mikrotik.


It'll be slotted in my mini home made rack with the cisco router and 12 way MDU.


A basic network diagram with the 10Gbe tweak. A few devices are missing off this, but it's got the bulk of it.


The server itself will need the replacement 990FX motherboard that I have spare in order to support a 10Gbe ethernet card as my old 790FX motherboard lacks the necessary number of PCIe lanes. I'll need to decide whether to also use the FX8320 or stick with the old phenom II x4. I need to decide on the SFP+ network adaptors I want to use. The usual choice seems to be mellanox connect-X items. As usual, these seem to be more common in the US than the UK. I've seen HP ones about but not sure how they fair.

I've also been considering moving from opensuse to centos when I change the hardware as I've been unable to get the LSI stoage manager server to install properly on opensuse. I installed centos 7 on an old server in work to use as a floating license server and found it pretty similar in most aspects but with better support for some software and a more active community.

 

[looz]

That's really neat! Having networking gear with actual CLI is so nice compared to most consumer gear.

 

[Kei]

Juniper EX3300 24P, fitted and configured. This thing was an absolute steal considering it was brand new and has a september 2018 build of junos loaded. (15.1R7.9)


Now I need to find some SFP+ NICs. I had been looking at trying to find a mellanox connectx 2 or 3 card but they seem to be common in the US and china. The only cards that I've come across that suits my needs are intel x520 and a few solarflare models. Having done a lot of research, it seems the intel cards are probably my best bet, though I am likely limited to intel coded SFP's which isn't too much of an issue with the fibre transceivers. The DAC may be different if the juniper doesn't like an intel coded DAC and the intel NIC doesn't like a juniper coded DAC.

 

[Kei]

Upgrade to the spare 990FX board with an FX8320 in it.

New board ready to go in. Old 790FX board and phenom II to come out.


Things don't look too bad inside considering it's been in the loft for 4.5 years and I've never dusted it. I've checked it over every couple of months but it's never looked too bad. The dust filters were looking a bit sorry for themselves.


The 4x4GB memory sticks are came out to go into the x79 system I gave to my parents and their 2x8GB sticks went in this. I managed to squeeze the 2x2GB corsair sticks in but had to remove the fins from the one module to get it to fit under the cooling fan. The intel expander card is now in the board which makes things much tidier. I can also now fit the side panel fan in place to aid cooling the cards.


Back side looks reasonably tidy.


It initially wouldn't boot up as it didn't like the memory config. Tried upping the DRAM voltage and tried dropping to 1333 (from 1600) but no dice. Had to run 1066 in order to get it to boot. Linux seemed to cope ok with the changes, however I've run into network issues as it will not pickup an ip address. I'll need to look into it further tomorrow.

In other news, I've purchased one mellanox connectx-3 Pro EN and two intel X710-DA2 10Gbe SFP+ NIC's.