A10 cluster mele a2000

References:
http://armservers.com/2012/06/18/apache-benchmarks-for-calxedas-5-watt-web-server/
http://www.theregister.co.uk/2012/07/09/boston_viridis_arm_server/

The Viridis arm server is $50K.
The goal is simple, “create a low cost arm cluster able to serve the maximum http requests”:

  • low power consumption
  • full redundancy (not a single SPOF)
  • space efficiency
  • minimal cost
  • remote power reboot

The first cluster (i will make two for a start) have 16 nodes.

I have shortlist these items to proceed:

  • The Allwinner A10: a great arm-clone running at 1Ghz.
  • The mele a2000 platform provides the A10, 512Mo ram, 4Go NAND, network controller, the lowest cost … a case and a nice sata plug. All of this for $70… Crazy little platform. Thanks to Tom Cubie for his support. Take a look at his own board: Cubieboard
  • Redundant PSU. It should provide enough power for the 16 mele a2000: Silverstone SST-ST42-GF or SST-ST50-GF: 80Plus, 5V 25A, 12V 33A
  • Ethernet relay board. Denvoki ethernet 16 relay board. Great product.

I have split the work in two case, the PSU and a sort of PDU with the relay board. The PDU needs 5V (for the mele) and 12V for the ethernet relay board. The 5V 25A could be a bit light to power the 16 nodes with disk. I decided to use this 5V for 10 nodes and convert 12V to 5V for the 6 nodes left. It will share the load properly. I’m using a simple DC-DC 12V to 5V 10A converter to do so (simple step-down found on ebay for $15).

Assembly part 1 – the PSU:

This PSU have one 12V and one 5V output. To avoid opening the PSU, i have proceed like that:

  1. Short the PS_ON (p14) and COM (p15) from the ATX connector.
  2. Join 4 x 12V (yellow) from different connectors (pcie, …) ~ average power needed 12V 5A, max 12V 10A
  3. Join 4 x 5V (red) from different connectors (molex, …) ~ average power needed 5V 10A, max 5V 20A
  4. Join 4 x COM (black) from different connectors.

I use a simple 230V – 10A power cable with IEC connectors to output to my PDU:

  • Neutral -> 5V
  • Ground -> COM
  • Phase -> 12V

Assembly part 2 – the PDU:

The PDU is nice and easy with the relay board:

  1. Get the incoming 12V from the IEC: phase of the ethernet card power, the step down converter and the relay card.
  2. split the output of the step down convertor to 6 relays.
  3. Get the incoming 5V from the IEC: split the output to the 10 relays.
  4. Get the incoming COM from the IEC: Neutral of the ethernet card, neutral of the step down convertor and neutral of all our outgoing Mele a2000 power cables. To do the assembly properly, i have plug the com of the step down convertor to the 6 Mele output cable but it’s not necessary.
  5. Plug each phase of Mele a2000 power cables to each relay output.

This is it 😉

I have now a remote power switch for the 16 Mele a2000 with a redundant PSU.

Software – Linux debian armhf:

The easiest way is to get the rootfs from rhombus.

Build our own debian distrib is fairly simple by following this guide.

Some tweaks needs to be done for armhf:

To cross compile from debian (unstable or testing):
echo "deb http://www.emdebian.org/debian/ unstable main" >> /etc/apt/sources.list
apt-get update; apt-get install gcc-4.7-arm-linux-gnueabihf gcc-4.7-base-armel-cross

The cross compile env becomes:
CROSS_COMPILE=arm-linux-gnueabihf-

The debbootstrap becomes:
sudo debootstrap --verbose --arch armhf --variant=minbase --foreign testing mnt http://ftp.debian.org/debian

I had to work around some issues on the u-boot (Forum) and Sata reset (Forum / Github).

I had applied also the work from lundman and Vayo from the github issue about the wemac performance.
At the time i will publish this page, i think this issues will be patched in github directly.

Software – Boot from nand:

I didn’t wish to get a sdcard for each Mele…

The boot from nand is easy to achieve. Two methods:

  1. Reuse the u-boot.bin from the nanda /linux directory, you simply have to put your uImage in this directory to get a boot. You should also regenerate the /script.bin with sunxi-tools bin2fex/fex2bin and replace the mac address.
  2. Use the multi u-boot (http://forum.doozan.com/read.php?6,10012,10082#msg-10082)

I used the first one, it was before the multi u-boot became available. The multi u-boot is a far more elegant solution now.

sunxi-tools allows you to repartition the nand properly to get the whole space in one single partition:
/usr/local/sunxi-tools/nand-part /dev/nand 'LROOTFS 4069376'
reboot and you simply have to put your debian rootfs in the fresh nandb formated with ext4.

The rootfs is stable in the nand. I add a Sata SSD to each mele to get a fast and reliable storage synced with unison for the files i will serve.

Results part 1 – power consumption

Power consumption seems better than i expected:

Cluster without SSD (1Ghz – governor: performance – idle) 280mA
Cluster with 16 SSD Crucial m4 (1Ghz – governor: performance) 385mA

Update – Oct 25, 2012

I’m now in full production with the 32 nodes (2×16) balanced around two datacenters.

  • the A10 is stable (uptime: 25 days for now and counting)
  • the load average is around 0.1 – 0.3
  • my biggest fear was the wemac ethernet. Perfectly stable too. No request failure reported from zabbix yet.

I’m pretty sure the ethernet relay board is useless now 😉

The total cost for the 32 nodes + 32 SSD + 2 PDU/PSU: $5100 or $160 per node with SSD or $99 per node without SSD

  • Mele a2000: $70 (with shipping)
  • SSD 64Go: $60
  • PSU/PDU: $470

Mission complete.

22 thoughts on “A10 cluster mele a2000

    • No i didn’t try it.
      My only goal was to make the mele boot without SD.
      The rootfs in the nand is working great and does remove the possible SPOF of a network boot.
      Also, network boot means ram disk… and 512Mo is low. I need all the ram available for caching.
      The more advanced uboot is so far a10linux uboot-multi and network boot with the wemac driver is not yet managed.

  • Wow looks amazing, also thank you to giving me hope to make my ~single~ mele have Linux on it at some point. Like you I have a v1.7 board. However I’m not as skilled to get this working.

    When I have a spare day I’ll follow the guide and see how I go.

    • Oh yes, it will run. Mele, like mine, is just debian/ubuntu server after all.
      I’m not sure it will run fast because the mele has:
      – a bit of latency around its wemac ethernet. It’s just 100Mbit after all.
      – low memory (they seem to use apache, memcached, etc.. it’s heavy stuff).

      The viridis arm server is just too expensive 😉 $50K for 48 proc/192 cores… $260 per core.
      Soon you will have set-top boxes more advanced with 1Go of ram and quad-core 1.4Ghz… like with the freescale i.mx6
      Just have to wait, but the price will be a bit higher than a single mele.

  • Hi guillaume,

    Have you managed to move the rootfs to the SATA device ?

    Is there any wear problem with the NAND flash. Is it safe in the long term run ?

    Thanks !
    Alberto

    • Hi,

      Boot from sata will work.
      You have two/three methods (put your kernel/initrd on the nand):
      – use a proper uboot and kernel (sata inside or initrd)
      – force the root= in compiled kernel params and add sata inside.
      Or seek for a tuned uboot able to boot your kernel hosted on sata directly.

      For the nand stability, it seems quite ok. uptime: 77 days from now. Not a single failure on the 32 nodes.
      If your rootfs is quite static (99.99% read), no problem.
      I tuned my syslog.conf to write on the local ssd… that’s all.
      At least if the ssd fails, i can change it easily.

      Bye.

  • Hi Guillaume,

    I´m using your image at /mele-a2000-headless-debian-wheezy-armhf-with-nand-install-v1/

    From the /proc/config.gz I can see:
    CONFIG_SCSI=y
    CONFIG_SATA_AHCI_PLATFORM=y

    Does it means SATA is builtin ?

    Does the uBoot in that image supports uEnv.txt ?

    What repository/branch did you compile this kernel from ?

    I´m a newcomer in Linux and my last kernel build attempt (for a BeagleBoard) was a complete frustration. That´s because I´m asking “perhaps a little too much” Sorry !!

    Thanks !
    Alberto

  • AWSOME PROJECT!!!
    i wanna build 1 too,
    if only i can find a GREAT justification to build one \
    (saving money for something else)

    any downtime so far?
    or mele device related problems?

    also, you guys might be interested there’s apparently a Mele A2000G (same spec,other than 1gb ram with 8gb NAND flash)

    cheers

  • Guillaume, where did you buy this high amount of Meles? Haven’t you have any problems at customs? I bought one at DX and it was stopped at customs due to lack of CE conformity. Bad luck :(.

    • Hi,
      i bought them from aliexpress (2 packs of 20) without any kind of problem except the delivery time (nearly a month) and via fedex. I still have some spares here in Paris. I don’t ship 😉

Leave a Reply

Your email address will not be published. Required fields are marked *