An Adventure in Homelab

For nearly 2 years now I’ve run some kind of homelab in my house. In that time it’s survived a house move and been greatly expanded upon. Right now I’m in the process of making some radical modifications and downsizing somewhat, so I thought I’d outline the history of the lab, it’s uses, and plans for the future.

It began with the purchase of a Dell R710, a favourite of the home server enthusiast everywhere, crammed on a desk in a spare bedroom. My specific model comprises two hexa core Intel X5660 CPUs running at 2.8GHz with 48GB of RAM and an H700 RAID Controller for the 6 3.5" drive bays. The use case here was to spin up unRAID and run a number of containers, as well as access the server as a NAS, but in order to do this I had to get the RAID Controller to play nice.

unRAID needs direct access to each drive, something the H700 can’t do as it doesn’t support IT (initiator target) mode - only IR (integrated raid) mode. So I had to replace it. During my search for a replacement I learned Dell servers can be a little picky over what RAID cards they’ll accept, but ultimately settled on an H330 which I then flashed into IT mode using this guide.

My goal was to use it as a NAS primarily, so I spun up unRAID and over the course of the next few months filled those 6 bays with hard drives and subsequently filled the drives themselves, prompting my next expansion: a Dell Xyratex Compellent HB-1235. This handy DAS gave me space for additional 12 3.5" drives, which I linked up to the R710 using a Dell H200E HBA and a couple of SFF-8088 SAS cables.

The final upgrades to my lab at this stage were an APC 3000 UPS to allow for tidy shutdowns in the event of a power failure, and (as I wanted to try out some hardware transcoding in Plex/Emby) an HP nVidia Quadro P2000 which with the help of a PCI riser cable and rear IO bracket removal fit snugly into the R710 case.

This setup was pretty spiffy. I was hovering around 60TB of capacity with 2 drives for parity and everything I wanted to run could, using the container functionality in unRAID. But upon moving house in late 2019 I saw the opportunity to take the lab to the next extreme.

The new house gave me a whole loft to play with, and so the first move was to get solid network infrastructure in place. I went for Ubiquiti hardware for this, snagging a Cloud Key Gen2, Security Gateway Pro 4, a 24 port PoE switch, a 16 port PoE switch, and a couple of AP Pros.

I then put the R710 into retirement and spun up a SuperMicro CSE-826 with 2 hexa core Intel Xeon E5-2630 V2s at 2.6GHz, 32GB of RAM, and 2 LSI SAS3801E-S controllers, along with a SuperMicro CSE-846 with a deca core Intel Xeon E5-2680 at 2.8GHz and 256GB of RAM.

Now, that sounds a like a hell of a lot of hardware - and you’d be right. My plan was to use the CSE-846 as my new file server, use the CSE-826 as my container server, and attach the 826’s backplane to the Dell H200E now installed in the CSE-846. That’s exactly what I did, and while not the most orthodox solution it works. The Quadro P2000 also found its way into the CSE-826 for direct access to the Emby Docker container. With 24 3.5" bays provided by the CSE-846, and 8 by the CSE-826, I now had room to fit a full FreeNAS array, comprising 3 RAIDz2 vdevs of seven 6TB drives, seven 8TB drives, and 7 10TB drives, toalling a whopping 98TB of usable space and allowing for 2 drive failures per vdev before complete data loss.

As you might imagine this setup is not without issue, which I’ll list below:

• I’m using out of date hardware. While it all works and probably will for some time, there’s just something nice about having up to date tech.
• FreeNAS on the CSE-846 does not need an E5-2680, demonstrated by CPU usage never exceeding 30%.
• FreeNAS does not need 256GB of RAM for a 100TB array. While nice to have, all of this is simply used for caching, which seems slightly overkill.
• My container server does not need 2 E5-2630s, again demonstrated by the incredibly low CPU utilisation.
• The joining together of one backplane into an expansion card of another server feels slightly wrong. Should I have made use of the Xyratex for additional bays? Probably. But that just adds to…
• The mammoth power usage.
• The noise generated by running two rack servers. Yes, I could swap out the case fans for some quieter ones, but that still leaves the incredibly loud PSU fans which I don’t really want to touch - digging around in a PSU is where I draw a line.
• FreeNAS spitting out SMART errors and kicking out perfectly good drives. This has happened three times now, and each time the drive has been absolutely fine and passed a short, conveyance, and extended test, and been happily reused. It’s not localised to any particular bay. My only theory is that the HBA is dying a slow death or FreeNAS is doing stupid things.
• I need more space. I have 4TB of space remaining meaning a purchase of another 7 drives which will have to go in the CSE-826 chassis and use the nasty backplane-into-HBA setup. And after that all my bays will be full and I’ll have to look into getting the Xyratex involved, adding to the noise, power usage, and…
• The multiple points of failure. Two backplanes, two HBAs, and potentially a DAS added to the mix. Not to mention I setup FreeNAS on a USB (as was recommended at the time) which I don’t trust all that much. If the FreeNAS box goes down that’s it, no file access.

It’s for all these reasons I’ve been looking for a new solution - and I think I’ve got one. A cluster of single board computers. Of course for this I’ll need to devise a way to safely migrate all the existing data, figure out a case solution, and if the whole thing is even viable, but hopeful. I’ll be updating this post with the continuation to the story once I’ve written it. Take it easy!