The Hacker's Guide to Building an AI Supercluster
New Hardware Unlocks Cheaper Clusters
Tenstorrent recently released their Blackhole product line, which has been described by them as "infinitely scalable". These cards network together directly through QSFP-DD cables with the ports on the device itself, circumventing the need for high-bandwidth PCIe lanes - one of the main costs for building a decent AI training rig with multiple cards. Since the PCIe lanes are only needed for data ingress/egress, you can use older generation crypto mining hardware (typically already equipped with ~3000W power supplies for the 300W cards) to host 4-8 cards at a time. Each card can hold 32GB of GDDR6, and perform 774 FLOPS of FP8 - a decent value for the $1,400 price tag on it's own. When you consider the networking capabilities 4x800G port networking capabilities, this thing can become a scaling monster.
This guide will show you how to build 4/8 card systems for 128/256 GB of GDDR6 out of Tenstorrent Blackhole cards and crypto mining rigs, including power and networking
Shopping list
- 1x GPU mining rig ~$65 USD
- 6x Molex QSFP-DD 400G Direct Attach Cable 1M ~$250 USD (optional, needed for training)
- 4x Nvidia 12VHPWR 12+4Pin to 2x8Pin Power Adapter Cable](https://ebay.us/m/sbC9YS) ~$50 USD
and of course the active ingredient:
- 4x Tenstorrent Blackhole p150a (p150a instead of p150b for better cooling in the rig) ~$5,640 USD with shipping.
All of this totals to ~$6,000 USD, almost exactly 1/2 of the $12,000 for the TT-QuietBox with theoretically the same AI compute abilities! This setup obviously has less CPU and PCIe capabilities, but if that is a major concern you can use the QuietBox as a host, and this rig as an extension through the networking. It is recommended to use a separate computer as the host machine (holding one card talking to the rest through the networking cables) if your mining rig does not have high enough PCIe bandwidth. A host machine is needed if your mining rig does not give enough BAR address space for the PCIe device, which is common for mining rigs.
BIOS
For the host bios, make sure to configure the following options:
- Enable Above 4G Decoding and (if present).
- Resizable BAR (General PCIe practice.)
- On Linux hosts, Tenstorrent requires AER “OS First” in BIOS for TT-SMI to work.
Power and Networking
For each card, you will need one 8 pin to 12VHPWR adapter, pretty straight forward.
The network topology for this setup will be a full mesh in the following configuration:
- A:QSFP1 ↔ B:QSFP1
- A:QSFP2 ↔ C:QSFP1
- B:QSFP2 ↔ D:QSFP1
- C:QSFP2 ↔ D:QSFP2
- A:QSFP3 ↔ D:QSFP3 (diagonal)
- B:QSFP3 ↔ C:QSFP3 (diagonal)
If you can, avoid sharp bends in these cables - they can increase the error rates in data transfer.
And that's it for hardware! Simply follow the TT install guide for the rest. This part is well documented by Tenstorrent so I will hand it off to this guide:
https://docs.tenstorrent.com/getting-started/README.html
Happy hacking, and post questions in the comments!