A10 cluster mele a2000

[Show as slideshow]

References:
http://armservers.com/2012/06/18/apache-benchmarks-for-calxedas-5-watt-web-server/
http://www.theregister.co.uk/2012/07/09/boston_viridis_arm_server/

The Viridis arm server is $50K.
The goal is simple, “create a low cost arm cluster able to serve the maximum http requests”:

• low power consumption
• full redundancy (not a single SPOF)
• space efficiency
• minimal cost
• remote power reboot

The first cluster (i will make two for a start) have 16 nodes.

I have shortlist these items to proceed:

• The Allwinner A10: a great arm-clone running at 1Ghz.
• The mele a2000 platform provides the A10, 512Mo ram, 4Go NAND, network controller, the lowest cost … a case and a nice sata plug. All of this for $70… Crazy little platform. Thanks to Tom Cubie for his support. Take a look at his own board: Cubieboard
• Redundant PSU. It should provide enough power for the 16 mele a2000: Silverstone SST-ST42-GF or SST-ST50-GF: 80Plus, 5V 25A, 12V 33A
• Ethernet relay board. Denvoki ethernet 16 relay board. Great product.

I have split the work in two case, the PSU and a sort of PDU with the relay board. The PDU needs 5V (for the mele) and 12V for the ethernet relay board. The 5V 25A could be a bit light to power the 16 nodes with disk. I decided to use this 5V for 10 nodes and convert 12V to 5V for the 6 nodes left. It will share the load properly. I’m using a simple DC-DC 12V to 5V 10A converter to do so (simple step-down found on ebay for $15).

Assembly part 1 – the PSU:

This PSU have one 12V and one 5V output. To avoid opening the PSU, i have proceed like that:

1. Short the PS_ON (p14) and COM (p15) from the ATX connector.
2. Join 4 x 12V (yellow) from different connectors (pcie, …) ~ average power needed 12V 5A, max 12V 10A
3. Join 4 x 5V (red) from different connectors (molex, …) ~ average power needed 5V 10A, max 5V 20A
4. Join 4 x COM (black) from different connectors.

I use a simple 230V – 10A power cable with IEC connectors to output to my PDU:

• Neutral -> 5V
• Ground -> COM
• Phase -> 12V

Assembly part 2 – the PDU:

The PDU is nice and easy with the relay board:

1. Get the incoming 12V from the IEC: phase of the ethernet card power, the step down converter and the relay card.
2. split the output of the step down convertor to 6 relays.
3. Get the incoming 5V from the IEC: split the output to the 10 relays.
4. Get the incoming COM from the IEC: Neutral of the ethernet card, neutral of the step down convertor and neutral of all our outgoing Mele a2000 power cables. To do the assembly properly, i have plug the com of the step down convertor to the 6 Mele output cable but it’s not necessary.
5. Plug each phase of Mele a2000 power cables to each relay output.

This is it 😉

I have now a remote power switch for the 16 Mele a2000 with a redundant PSU.

Software – Linux debian armhf:

The easiest way is to get the rootfs from rhombus.

Build our own debian distrib is fairly simple by following this guide.

Some tweaks needs to be done for armhf:

To cross compile from debian (unstable or testing):
echo "deb http://www.emdebian.org/debian/ unstable main" >> /etc/apt/sources.list
apt-get update; apt-get install gcc-4.7-arm-linux-gnueabihf gcc-4.7-base-armel-cross

The cross compile env becomes:
CROSS_COMPILE=arm-linux-gnueabihf-

The debbootstrap becomes:
sudo debootstrap --verbose --arch armhf --variant=minbase --foreign testing mnt http://ftp.debian.org/debian

I had to work around some issues on the u-boot (Forum) and Sata reset (Forum / Github).

I had applied also the work from lundman and Vayo from the github issue about the wemac performance.
At the time i will publish this page, i think this issues will be patched in github directly.

Software – Boot from nand:

I didn’t wish to get a sdcard for each Mele…

The boot from nand is easy to achieve. Two methods:

1. Reuse the u-boot.bin from the nanda /linux directory, you simply have to put your uImage in this directory to get a boot. You should also regenerate the /script.bin with sunxi-tools bin2fex/fex2bin and replace the mac address.
2. Use the multi u-boot (http://forum.doozan.com/read.php?6,10012,10082#msg-10082)

I used the first one, it was before the multi u-boot became available. The multi u-boot is a far more elegant solution now.

sunxi-tools allows you to repartition the nand properly to get the whole space in one single partition:
/usr/local/sunxi-tools/nand-part /dev/nand 'LROOTFS 4069376'
reboot and you simply have to put your debian rootfs in the fresh nandb formated with ext4.

The rootfs is stable in the nand. I add a Sata SSD to each mele to get a fast and reliable storage synced with unison for the files i will serve.

Results part 1 – power consumption

Power consumption seems better than i expected:

Update – Oct 25, 2012

I’m now in full production with the 32 nodes (2×16) balanced around two datacenters.

• the A10 is stable (uptime: 25 days for now and counting)
• the load average is around 0.1 – 0.3
• my biggest fear was the wemac ethernet. Perfectly stable too. No request failure reported from zabbix yet.

I’m pretty sure the ethernet relay board is useless now 😉

The total cost for the 32 nodes + 32 SSD + 2 PDU/PSU: $5100 or $160 per node with SSD or $99 per node without SSD

• Mele a2000: $70 (with shipping)
• SSD 64Go: $60
• PSU/PDU: $470

Mission complete.