At the raspbian project I (as project cofounder) decided that the existing infastructure we had built for raspbian wheezy needed to be replaced before we started on raspbian jessie for a few reasons. Firstly because having all the build hardware in the founders basement on the end of a comcast cable line was considered far from ideal. Secondly because the software set-up was a horrible hack based on an ancient version of wanna-build*, and that building with only 1GB of ram was making some builds excruciatingly slow. Bytemark had previously offered to sponsor and host build hardware for us and so we opened discussions with them.
We build natively because cross-building support in Debian was (and still is) far too immature to be used to rebuild the whole distro as we do. So the hardware had to be arm based and powerful. My requirements included being reasonably affordable, having at least 2GB of ram, having native SATA (I've learnt the hard way that USB on arm systems is often unstable) and preferable having native Ethernet.
I evaluated the hardware options and decided the 2GB nitrogen6x was the best available. The wandboard models available at the time were rejected due to lack of native sata and only having 1GB of ram.
I also opened up discussions with bytemark at this point, they were cautiously optimistic but it took some time for them to deal with the formailities of making such a donation and I also said that I wanted to test the hardware before asking them to commit to it.
I ordered a 2GB nitrogen6x but it took ages for boundary devices to actually supply it, apparently they were having trouble getting the memory chips. I tested it by starting it working on building raspbian jessie (hosted at my place connected to the repo server over a VPN). It was also somewhat expensive (about $470 including the board itself, the 2GB ram option, sata cable, shipping, VAT and the fee for collecting the VAT)
When I first started testing I had some stability problems and it took a few kernel updates, addition of a heatsink and a lot of testing before I decided it was stable enough to use as an auto builder. It also became clear that one autobuilder was not enough to keep up with jessie and I calculated that two autobuilders was the absoloute minimum to keep up. I thought it was prudent to have two more to give us some breathing room for large updates and to give us some protection against hardware failure taking the total requirement to four.
While I was testing the nitrogen6x it came to my attention that the wandboard quad had been released offering much the same hardware as the 2GB nitrogen6x but on a physically smaller board and at about half the price. I ordered one to evaluate and after some initial bumps with the software setup set it building alongside the nitrogen6x.
The two boards side by side.
A few weeks of testing proved it to be sufficiently reliable and we moved to discussions about geting a cluster of four of them hosted at bytemark. Here are some pics from the build.
One of he power supplies, 5V 5A (2A for the wandboard, 1A for the HDD, 2A safety margin) with an IEC inlet (mouser part number 709-GS40A05-P1J) seperate power supplies were used for each board so we could reset crashed boards using bytemarks existing systems. The option of one big power supply and a custom relay board was considered (and is probablly what i'd have done if I was buliding it myself) but this was decided to be simpler to implement given the conditions.
Unboxing shot, well packaged.
500GB laptop hdd sitting on an adapter bracket, laptop hdds were used so we didn't have to worry about 12V power.
A 2U rackmount case, presumablly the remains of a gutted server.
The case from the front, presumablly those bays were originally hotswap but the backplane is no longer there.
Plywood mounting board to hold the wandboards.
Power supplies in case and connected to terminal blocks on mounting board, IEC extensions on the front two so all power connections can be made from outside the box.
Power for hdds and fans.
Power supplies, hard drives and associated wiring in place,
Closeup of the power wiring.
Wandboard quad with serial adapter, presumablly testing fit.
Pillars mounted on board
Marked out for remaining boards
All pillars mounted.
Boards in place.
IO panel, apparently the "result of an old Hellermann Tyton 1U 24 port back-to-back patch panel meeting a hacksaw"
Power and sata connected to boards.
Inside of the IO panel.
All wired up
Software wise we use Debian wheezy armhf as the "outer OS" and raspbian in the build chroots. We use Debian's buildd/sbuild software with BTRFS snapshots. The boards boot off microSD cards with the hard drives used for build chroots, various other temporary files from the buildd process and swap space. We use a version of u-boot built from robert nelsons sources (don't remember the exact version offhand). We currently use a multiplatform kernel built from debian's 3.13-1~exp1 source package. Prior to that we were using robert nelson's kernels.
The autobuilders have been building raspbian jessie for some time (I plan to move wheezy over as well but it's going to take some time to get around to it). The two test autobuilders at my place are currently also still active, we don't strictly need them but it gives us faster turnaround times.
* Mike at the time had failed to get current versions to work and decided hacking up an older and broken but slightly better docuemnted version was the way to go......
Big thanks to bytemark for donating, building and hosting this for us.