References:
http://armservers.com/2012/06/18/apache-benchmarks-for-calxedas-5-watt-web-server/
http://www.theregister.co.uk/2012/07/09/boston_viridis_arm_server/
The Viridis arm server is $50K.
The goal is simple, “create a low cost arm cluster able to serve the maximum http requests”:
- low power consumption
- full redundancy (not a single SPOF)
- space efficiency
- minimal cost
- remote power reboot
The first cluster (i will make two for a start) have 16 nodes.
I have shortlist these items to proceed:
- The Allwinner A10: a great arm-clone running at 1Ghz.
- The mele a2000 platform provides the A10, 512Mo ram, 4Go NAND, network controller, the lowest cost … a case and a nice sata plug. All of this for $70… Crazy little platform. Thanks to Tom Cubie for his support. Take a look at his own board: Cubieboard
- Redundant PSU. It should provide enough power for the 16 mele a2000: Silverstone SST-ST42-GF or SST-ST50-GF: 80Plus, 5V 25A, 12V 33A
- Ethernet relay board. Denvoki ethernet 16 relay board. Great product.
I have split the work in two case, the PSU and a sort of PDU with the relay board. The PDU needs 5V (for the mele) and 12V for the ethernet relay board. The 5V 25A could be a bit light to power the 16 nodes with disk. I decided to use this 5V for 10 nodes and convert 12V to 5V for the 6 nodes left. It will share the load properly. I’m using a simple DC-DC 12V to 5V 10A converter to do so (simple step-down found on ebay for $15).
Assembly part 1 – the PSU:
This PSU have one 12V and one 5V output. To avoid opening the PSU, i have proceed like that:
- Short the PS_ON (p14) and COM (p15) from the ATX connector.
- Join 4 x 12V (yellow) from different connectors (pcie, …) ~ average power needed 12V 5A, max 12V 10A
- Join 4 x 5V (red) from different connectors (molex, …) ~ average power needed 5V 10A, max 5V 20A
- Join 4 x COM (black) from different connectors.
I use a simple 230V – 10A power cable with IEC connectors to output to my PDU:
- Neutral -> 5V
- Ground -> COM
- Phase -> 12V
Assembly part 2 – the PDU:
The PDU is nice and easy with the relay board:
- Get the incoming 12V from the IEC: phase of the ethernet card power, the step down converter and the relay card.
- split the output of the step down convertor to 6 relays.
- Get the incoming 5V from the IEC: split the output to the 10 relays.
- Get the incoming COM from the IEC: Neutral of the ethernet card, neutral of the step down convertor and neutral of all our outgoing Mele a2000 power cables. To do the assembly properly, i have plug the com of the step down convertor to the 6 Mele output cable but it’s not necessary.
- Plug each phase of Mele a2000 power cables to each relay output.
This is it 😉
I have now a remote power switch for the 16 Mele a2000 with a redundant PSU.
Software – Linux debian armhf:
The easiest way is to get the rootfs from rhombus.
Build our own debian distrib is fairly simple by following this guide.
Some tweaks needs to be done for armhf:
To cross compile from debian (unstable or testing):
echo "deb http://www.emdebian.org/debian/ unstable main" >> /etc/apt/sources.list
apt-get update; apt-get install gcc-4.7-arm-linux-gnueabihf gcc-4.7-base-armel-cross
The cross compile env becomes:
CROSS_COMPILE=arm-linux-gnueabihf-
The debbootstrap becomes:
sudo debootstrap --verbose --arch armhf --variant=minbase --foreign testing mnt http://ftp.debian.org/debian
I had to work around some issues on the u-boot (Forum) and Sata reset (Forum / Github).
I had applied also the work from lundman and Vayo from the github issue about the wemac performance.
At the time i will publish this page, i think this issues will be patched in github directly.
Software – Boot from nand:
I didn’t wish to get a sdcard for each Mele…
The boot from nand is easy to achieve. Two methods:
- Reuse the u-boot.bin from the nanda /linux directory, you simply have to put your uImage in this directory to get a boot. You should also regenerate the /script.bin with sunxi-tools bin2fex/fex2bin and replace the mac address.
- Use the multi u-boot (http://forum.doozan.com/read.php?6,10012,10082#msg-10082)
I used the first one, it was before the multi u-boot became available. The multi u-boot is a far more elegant solution now.
sunxi-tools allows you to repartition the nand properly to get the whole space in one single partition:
/usr/local/sunxi-tools/nand-part /dev/nand 'LROOTFS 4069376'
reboot and you simply have to put your debian rootfs in the fresh nandb formated with ext4.
The rootfs is stable in the nand. I add a Sata SSD to each mele to get a fast and reliable storage synced with unison for the files i will serve.
Results part 1 – power consumption
Power consumption seems better than i expected:
Cluster without SSD (1Ghz – governor: performance – idle) | 280mA |
Cluster with 16 SSD Crucial m4 (1Ghz – governor: performance) | 385mA |
Update – Oct 25, 2012
I’m now in full production with the 32 nodes (2×16) balanced around two datacenters.
- the A10 is stable (uptime: 25 days for now and counting)
- the load average is around 0.1 – 0.3
- my biggest fear was the wemac ethernet. Perfectly stable too. No request failure reported from zabbix yet.
I’m pretty sure the ethernet relay board is useless now 😉
The total cost for the 32 nodes + 32 SSD + 2 PDU/PSU: $5100 or $160 per node with SSD or $99 per node without SSD
- Mele a2000: $70 (with shipping)
- SSD 64Go: $60
- PSU/PDU: $470
Mission complete.
Have you tried to netboot with a more recent version of uboot?
No i didn’t try it.
My only goal was to make the mele boot without SD.
The rootfs in the nand is working great and does remove the possible SPOF of a network boot.
Also, network boot means ram disk… and 512Mo is low. I need all the ram available for caching.
The more advanced uboot is so far a10linux uboot-multi and network boot with the wemac driver is not yet managed.
Wow looks amazing, also thank you to giving me hope to make my ~single~ mele have Linux on it at some point. Like you I have a v1.7 board. However I’m not as skilled to get this working.
When I have a spare day I’ll follow the guide and see how I go.
Great work 😉 I’m still have problem with sata, just wondering which kernel version are you using?
Thanks.
I’m using the current version of allwinner-v3.0-android-v2.
https://github.com/amery/linux-allwinner/tree/allwinner-v3.0-android-v2
What kind of sata problem do you have?
I found you stat clock patch. Works for me 😉 It’s awesome.
Sorry, SATA not stat
That is just beautiful!!!! I’m drooling! You have your own ARM cloud!
wow nice . Viridis arm server is based on openstack, please can this setup run openstack cluster?
Oh yes, it will run. Mele, like mine, is just debian/ubuntu server after all.
I’m not sure it will run fast because the mele has:
– a bit of latency around its wemac ethernet. It’s just 100Mbit after all.
– low memory (they seem to use apache, memcached, etc.. it’s heavy stuff).
The viridis arm server is just too expensive 😉 $50K for 48 proc/192 cores… $260 per core.
Soon you will have set-top boxes more advanced with 1Go of ram and quad-core 1.4Ghz… like with the freescale i.mx6
Just have to wait, but the price will be a bit higher than a single mele.
please guillaume send me your email so we talk better please. thanks.
Hi guillaume,
Have you managed to move the rootfs to the SATA device ?
Is there any wear problem with the NAND flash. Is it safe in the long term run ?
Thanks !
Alberto
Hi,
Boot from sata will work.
You have two/three methods (put your kernel/initrd on the nand):
– use a proper uboot and kernel (sata inside or initrd)
– force the root= in compiled kernel params and add sata inside.
Or seek for a tuned uboot able to boot your kernel hosted on sata directly.
For the nand stability, it seems quite ok. uptime: 77 days from now. Not a single failure on the 32 nodes.
If your rootfs is quite static (99.99% read), no problem.
I tuned my syslog.conf to write on the local ssd… that’s all.
At least if the ssd fails, i can change it easily.
Bye.
Hi Guillaume,
I´m using your image at http://guillaumeplayground.net/mele-a2000-headless-debian-wheezy-armhf-with-nand-install-v1/
From the /proc/config.gz I can see:
CONFIG_SCSI=y
CONFIG_SATA_AHCI_PLATFORM=y
Does it means SATA is builtin ?
Does the uBoot in that image supports uEnv.txt ?
What repository/branch did you compile this kernel from ?
I´m a newcomer in Linux and my last kernel build attempt (for a BeagleBoard) was a complete frustration. That´s because I´m asking “perhaps a little too much” Sorry !!
Thanks !
Alberto
AWSOME PROJECT!!!
i wanna build 1 too,
if only i can find a GREAT justification to build one \
(saving money for something else)
any downtime so far?
or mele device related problems?
also, you guys might be interested there’s apparently a Mele A2000G (same spec,other than 1gb ram with 8gb NAND flash)
cheers
Can i buy these on dx.com, is it this one: http://dx.com/p/mele-a2000-1080p-android-2-3-network-multi-media-player-w-sata-usb-hdmi-lan-vga-wifi-4gb-131566
Yes.
Guillaume, where did you buy this high amount of Meles? Haven’t you have any problems at customs? I bought one at DX and it was stopped at customs due to lack of CE conformity. Bad luck :(.
Hi,
i bought them from aliexpress (2 packs of 20) without any kind of problem except the delivery time (nearly a month) and via fedex. I still have some spares here in Paris. I don’t ship 😉
Is this possible to run ALSA on this? I want to build a media center ;).
Kindly explain in details, how you have achieved redundancy in psu.