The Viridis arm server is $50K.
The goal is simple, “create a low cost arm cluster able to serve the maximum http requests”:
- low power consumption
- full redundancy (not a single SPOF)
- space efficiency
- minimal cost
- remote power reboot
The first cluster (i will make two for a start) have 16 nodes.
I have shortlist these items to proceed:
- The Allwinner A10: a great arm-clone running at 1Ghz.
- The mele a2000 platform provides the A10, 512Mo ram, 4Go NAND, network controller, the lowest cost … a case and a nice sata plug. All of this for $70… Crazy little platform. Thanks to Tom Cubie for his support. Take a look at his own board: Cubieboard
- Redundant PSU. It should provide enough power for the 16 mele a2000: Silverstone SST-ST42-GF or SST-ST50-GF: 80Plus, 5V 25A, 12V 33A
- Ethernet relay board. Denvoki ethernet 16 relay board. Great product.
I have split the work in two case, the PSU and a sort of PDU with the relay board. The PDU needs 5V (for the mele) and 12V for the ethernet relay board. The 5V 25A could be a bit light to power the 16 nodes with disk. I decided to use this 5V for 10 nodes and convert 12V to 5V for the 6 nodes left. It will share the load properly. I’m using a simple DC-DC 12V to 5V 10A converter to do so (simple step-down found on ebay for $15).
Assembly part 1 – the PSU:
This PSU have one 12V and one 5V output. To avoid opening the PSU, i have proceed like that:
- Short the PS_ON (p14) and COM (p15) from the ATX connector.
- Join 4 x 12V (yellow) from different connectors (pcie, …) ~ average power needed 12V 5A, max 12V 10A
- Join 4 x 5V (red) from different connectors (molex, …) ~ average power needed 5V 10A, max 5V 20A
- Join 4 x COM (black) from different connectors.
I use a simple 230V – 10A power cable with IEC connectors to output to my PDU:
- Neutral -> 5V
- Ground -> COM
- Phase -> 12V
Assembly part 2 – the PDU:
The PDU is nice and easy with the relay board:
- Get the incoming 12V from the IEC: phase of the ethernet card power, the step down converter and the relay card.
- split the output of the step down convertor to 6 relays.
- Get the incoming 5V from the IEC: split the output to the 10 relays.
- Get the incoming COM from the IEC: Neutral of the ethernet card, neutral of the step down convertor and neutral of all our outgoing Mele a2000 power cables. To do the assembly properly, i have plug the com of the step down convertor to the 6 Mele output cable but it’s not necessary.
- Plug each phase of Mele a2000 power cables to each relay output.
This is it 😉
I have now a remote power switch for the 16 Mele a2000 with a redundant PSU.
Software – Linux debian armhf:
The easiest way is to get the rootfs from rhombus.
Build our own debian distrib is fairly simple by following this guide.
Some tweaks needs to be done for armhf:
To cross compile from debian (unstable or testing):
echo "deb http://www.emdebian.org/debian/ unstable main" >> /etc/apt/sources.list
apt-get update; apt-get install gcc-4.7-arm-linux-gnueabihf gcc-4.7-base-armel-cross
The cross compile env becomes:
The debbootstrap becomes:
sudo debootstrap --verbose --arch armhf --variant=minbase --foreign testing mnt http://ftp.debian.org/debian
I had applied also the work from lundman and Vayo from the github issue about the wemac performance.
At the time i will publish this page, i think this issues will be patched in github directly.
Software – Boot from nand:
I didn’t wish to get a sdcard for each Mele…
The boot from nand is easy to achieve. Two methods:
- Reuse the u-boot.bin from the nanda /linux directory, you simply have to put your uImage in this directory to get a boot. You should also regenerate the /script.bin with sunxi-tools bin2fex/fex2bin and replace the mac address.
- Use the multi u-boot (http://forum.doozan.com/read.php?6,10012,10082#msg-10082)
I used the first one, it was before the multi u-boot became available. The multi u-boot is a far more elegant solution now.
sunxi-tools allows you to repartition the nand properly to get the whole space in one single partition:
/usr/local/sunxi-tools/nand-part /dev/nand 'LROOTFS 4069376'
reboot and you simply have to put your debian rootfs in the fresh nandb formated with ext4.
The rootfs is stable in the nand. I add a Sata SSD to each mele to get a fast and reliable storage synced with unison for the files i will serve.
Results part 1 – power consumption
Power consumption seems better than i expected:
|Cluster without SSD (1Ghz – governor: performance – idle)||280mA|
|Cluster with 16 SSD Crucial m4 (1Ghz – governor: performance)||385mA|
Update – Oct 25, 2012
I’m now in full production with the 32 nodes (2×16) balanced around two datacenters.
- the A10 is stable (uptime: 25 days for now and counting)
- the load average is around 0.1 – 0.3
- my biggest fear was the wemac ethernet. Perfectly stable too. No request failure reported from zabbix yet.
I’m pretty sure the ethernet relay board is useless now 😉
The total cost for the 32 nodes + 32 SSD + 2 PDU/PSU: $5100 or $160 per node with SSD or $99 per node without SSD
- Mele a2000: $70 (with shipping)
- SSD 64Go: $60
- PSU/PDU: $470