A10 cluster mele a2000

[Show as slideshow]

References:
http://armservers.com/2012/06/18/apache-benchmarks-for-calxedas-5-watt-web-server/
http://www.theregister.co.uk/2012/07/09/boston_viridis_arm_server/

The Viridis arm server is $50K.
The goal is simple, “create a low cost arm cluster able to serve the maximum http requests”:

• low power consumption
• full redundancy (not a single SPOF)
• space efficiency
• minimal cost
• remote power reboot

The first cluster (i will make two for a start) have 16 nodes.

I have shortlist these items to proceed:

• The Allwinner A10: a great arm-clone running at 1Ghz.
• The mele a2000 platform provides the A10, 512Mo ram, 4Go NAND, network controller, the lowest cost … a case and a nice sata plug. All of this for $70… Crazy little platform. Thanks to Tom Cubie for his support. Take a look at his own board: Cubieboard
• Redundant PSU. It should provide enough power for the 16 mele a2000: Silverstone SST-ST42-GF or SST-ST50-GF: 80Plus, 5V 25A, 12V 33A
• Ethernet relay board. Denvoki ethernet 16 relay board. Great product.

I have split the work in two case, the PSU and a sort of PDU with the relay board. The PDU needs 5V (for the mele) and 12V for the ethernet relay board. The 5V 25A could be a bit light to power the 16 nodes with disk. I decided to use this 5V for 10 nodes and convert 12V to 5V for the 6 nodes left. It will share the load properly. I’m using a simple DC-DC 12V to 5V 10A converter to do so (simple step-down found on ebay for $15).

Assembly part 1 – the PSU:

This PSU have one 12V and one 5V output. To avoid opening the PSU, i have proceed like that:

1. Short the PS_ON (p14) and COM (p15) from the ATX connector.
2. Join 4 x 12V (yellow) from different connectors (pcie, …) ~ average power needed 12V 5A, max 12V 10A
3. Join 4 x 5V (red) from different connectors (molex, …) ~ average power needed 5V 10A, max 5V 20A
4. Join 4 x COM (black) from different connectors.

I use a simple 230V – 10A power cable with IEC connectors to output to my PDU:

• Neutral -> 5V
• Ground -> COM
• Phase -> 12V

Assembly part 2 – the PDU:

The PDU is nice and easy with the relay board:

1. Get the incoming 12V from the IEC: phase of the ethernet card power, the step down converter and the relay card.
2. split the output of the step down convertor to 6 relays.
3. Get the incoming 5V from the IEC: split the output to the 10 relays.
4. Get the incoming COM from the IEC: Neutral of the ethernet card, neutral of the step down convertor and neutral of all our outgoing Mele a2000 power cables. To do the assembly properly, i have plug the com of the step down convertor to the 6 Mele output cable but it’s not necessary.
5. Plug each phase of Mele a2000 power cables to each relay output.

This is it

I have now a remote power switch for the 16 Mele a2000 with a redundant PSU.

Software – Linux debian armhf:

The easiest way is to get the rootfs from rhombus.

Build our own debian distrib is fairly simple by following this guide.

Some tweaks needs to be done for armhf:

To cross compile from debian (unstable or testing):
echo "deb http://www.emdebian.org/debian/ unstable main" >> /etc/apt/sources.list
apt-get update; apt-get install gcc-4.7-arm-linux-gnueabihf gcc-4.7-base-armel-cross

The cross compile env becomes:
CROSS_COMPILE=arm-linux-gnueabihf-

The debbootstrap becomes:
sudo debootstrap --verbose --arch armhf --variant=minbase --foreign testing mnt http://ftp.debian.org/debian

I had to work around some issues on the u-boot (Forum) and Sata reset (Forum / Github).

I had applied also the work from lundman and Vayo from the github issue about the wemac performance.
At the time i will publish this page, i think this issues will be patched in github directly.

Software – Boot from nand:

I didn’t wish to get a sdcard for each Mele…

The boot from nand is easy to achieve. Two methods:

1. Reuse the u-boot.bin from the nanda /linux directory, you simply have to put your uImage in this directory to get a boot. You should also regenerate the /script.bin with sunxi-tools bin2fex/fex2bin and replace the mac address.
2. Use the multi u-boot (http://forum.doozan.com/read.php?6,10012,10082#msg-10082)

I used the first one, it was before the multi u-boot became available. The multi u-boot is a far more elegant solution now.

sunxi-tools allows you to repartition the nand properly to get the whole space in one single partition:
/usr/local/sunxi-tools/nand-part /dev/nand 'LROOTFS 4069376'
reboot and you simply have to put your debian rootfs in the fresh nandb formated with ext4.

The rootfs is stable in the nand. I add a Sata SSD to each mele to get a fast and reliable storage synced with unison for the files i will serve.

Results part 1 – power consumption

Power consumption seems better than i expected:

Update – Oct 25, 2012

I’m now in full production with the 32 nodes (2×16) balanced around two datacenters.

• the A10 is stable (uptime: 25 days for now and counting)
• the load average is around 0.1 – 0.3
• my biggest fear was the wemac ethernet. Perfectly stable too. No request failure reported from zabbix yet.

I’m pretty sure the ethernet relay board is useless now

The total cost for the 32 nodes + 32 SSD + 2 PDU/PSU: $5100 or $160 per node with SSD or $99 per node without SSD

• Mele a2000: $70 (with shipping)
• SSD 64Go: $60
• PSU/PDU: $470

Mission complete.