SOD: Sidebar Diversion
I couldn’t get the idea out of my head that the Avatar rendering cluster required 1 petabyte of storage. However, this slide show of the facilities used for filming the actors opened my eyes. [eye opening slide show]
The petabyte is required not just for the finished product. It’s needed to store all the sensor and camera data as well. Okay. I accept that Weta needed 1PB. How does one go about creating a petabyte storage facility? What are the tradeoffs? How much does it cost to build and then to maintain?
I need to get this out of my head and free up some brain cycles to continue with my Seeds of Discontent series. This article is a sidebar.
Disclaimer: I’m using a server build derived from the Seeds of Discontent serial (see also, Seeds of Discontent: System Benchmarking Hardware). It isn’t a traditional server room server but it serves as a baseline talking point. Besides, I can get budgetary pricing off newegg.com. It’s good enough for the purposes of this discussion. I’m sure the staff at Weta (or ILM or Dreamworks or Pixar) would have more insight into what *really* works.
I’ll choose the 4U chassis for this exercise since it better approximates the airflow of a desktop chassis. The 50U rack is taller than usual but it allows for 12 4-U servers plus a switch.
In the Seeds of Discontent serial, the HDD array uses the cheapest drives available; The final size is not important. In this sidebar discussion, final HDD capacity *is* important. This is not a single desktop box but a compute and storage cluster.
Each server functions as a compute unit as well as a storage unit.
A commodity multi-core CPU together with two commodity PCI Express GPU subsystems comprise the compute unit. Primary storage and swap for the compute unit is a two-disk RAID 0 SSD. The GPU cards have their own RAM while the CPU has two triple-channel banks of DDR3.
The server also hosts an HDD array which is not private storage but part of a larger storage cluster. For this exercise, I arbitrarily choose GlusterFS.
I’ve budgeted $2,500 USD per server (less the HDD array).
(circa mid-November 2010)
case+PSU $ 200
mainboard 230
SSD 135
SSD 135
DDR3 24GB 550
CPU 850
GPU 200
CPU 200
--------------------
$25,000 USD
$2,500 USD
In addition, I’ve budgeted $1,000 per rack and $1,000 per switch. A fully constructed rack sans HDD is $32,000. YMMV.
rack $ 1,000
switch 1,000
servers 30,000 (2,500 * 12 servers/rack)
--------------------
$32,000 USD
Let’s add the HDD to build out a petabyte cluster. Since I’m using the Asus Rampage III (admitedly not a server mainboard), the two GPU fully consume the PCI Express lanes. There aren’t any lanes left for a RAID card. The two SSD drives occupy the two SATA III channels leaving the seven SATA II channels for the HDD array. Each server then adds seven HDD (and each rack adds 84 HDD) to the cluster.
As seen from this list (circa mid-November 2010), the lower capacity drives are not the cheapest drives per gigabyte.
SKU $USD cnt GB $/GB $K/PB HDD/PB --------------- ---- --- ---- ----- ----- ---- WD5002ABYS-20PK 1750 20 500 0.175 175 2000 WD7502ABYS-20PK 2500 20 750 0.167 167 1333 0A39289-20PK 2700 20 1000 0.135 135 1000 WD7501AALS-20PK 1550 20 750 0.103 103 1333 WD6401AALS-20PK 1300 20 640 0.102 102 1563 WD5000AADS-20PK 900 20 500 0.090 90 2000 0F10381-20PK 900 20 500 0.090 90 2000 WD1001FALS-20PK 1800 20 1000 0.090 90 1000 WD5000AAKS-20PK 880 20 500 0.088 88 2000 WD6400AARS 55 1 640 0.086 86 1563 WD7500AADS-20PK 1250 20 750 0.083 83 1333 ST3500418AS 40 1 500 0.080 80 2000 WD7500AADS 55 1 750 0.073 73 1333 WD20EVDS-20PK 2800 20 2000 0.070 70 500 0F10383-20PK 1400 20 1000 0.070 70 1000 WD10EALS-20PK 1350 20 1000 0.068 68 1000 WD10EARS 65 1 1000 0.065 65 1000 WD10EARS-20PK 1200 20 1000 0.060 60 1000 WD15EARS-20PK 1700 20 1500 0.057 57 667 ST31000528AS 50 1 1000 0.050 50 1000 ST31500341AS 60 1 1500 0.040 40 667
Downselect the cheapest drive at each capacity point greater than or equal to 1TB.
SKU $USD cnt GB $/GB $K/PB HDD/PB --------------- ---- --- ---- ----- ----- ---- WD20EVDS-20PK 2800 20 2000 0.070 70 500 ST31000528AS 50 1 1000 0.050 50 1000 ST31500341AS 60 1 1500 0.040 40 667
I want to minimize the number of drives while minimizing costs. The Seagate 1TB drive is both more expensive per GB and requires more drives per PB than the Seagate 1.5TB drive. It is immediately eliminated. The competition is between the Western Digital 2TB and the Seagate 1TB drive.
If cost were the only issue, then the Seagate drive would win. If drive count were the only issue then the Western Digital drive would win. To get closer to an answer, let’s build out the storage cluster.
Petabyte Cluster with Just a Bunch of Disks
======= count ====== ========= cost ==========
Drive HDD Servers Racks HDD Rack Total
----- ----- ------- ----- ------- ------- -------
2.0TB 500 72 6 70,000 192,000 262,000
1.5TB 667 96 8 40,000 256,000 296,000
Even though the 2TB drives cost $30,000 more than the 1.5TB drives, the total cluster cost is $34,000 for the 1.5TB drive choice. Furthermore, there is no redundancy to protect against drive failure. How likely is a drive to fail? It’s not just likely to happen. It will happen. If a single drive will on average fail in five years, then a single drive in a pool of 500 drives will on average fail in 1/100 of a year (or roughly two drives a week).
Without debate, I posit that it is not possible to make nightly backups of a petabyte storage cluster. Firstly, we’d need a second petabyte. Secondly, that’s a lot of data to move and the cluster needs to continuously run compute jobs (rendering). The solution is redundancy either through local machine (e.g., RAID 6) or through GlusterFS replication (analogous to RAID 10 but at the cluster level).
Petabyte Cluster with RAID-6
======= count ====== ========= cost ==========
Drive HDD Servers Racks HDD Rack Total
----- ----- ------- ----- ------- ------- -------
2.0TB 700 100 8.3 98,000 268,000 366,000
1.5TB 1,087 156 13 56,000 416,000 472,000
Petabyte Cluster with RAID-10
======= count ====== ========= cost ==========
Drive HDD Servers Racks HDD Rack Total
----- ----- ------- ----- ------- ------- -------
2.0TB 1,000 144 12 140,000 384,000 524,000
1.5TB 1,334 292 16 80,000 512,000 592,000
The drive count starts to really add leading to more frequent drive failures. With a RAID 10 (or the equivalent GlusterFS replication scheme), the operations team can expect to replace five to six drives per week.
But the larger question is, how many servers are needed for the compute cluster? What if rendering needed no more than 60 compute nodes? If we fixed the compute nodes count to 60, we would need to add more drives per server. For the sake of discussion, assume we could load 24 drives per server but that doubles the cost per server before including drive costs (i.e., 2 * $2,500 = $5,000 per server sans HDD).
Furthermore, assume we’re using GlusterFS replication for the storage cluster redundancy. This presses the drive count up but lowers the complexity of building and maintaining local RAID systems each with 24 drives.
Petabyte 60 Server Cluster with RAID-10
======= count ====== ========= cost ==========
Drive HDD Servers Racks HDD Rack Total
----- ----- ------- ----- ------- ------- -------
2.0TB 1,000 60 5 140,000 160,000 300,000
1.5TB 1,334 60 5 80,000 160,000 240,000
There is a $60,000 capital cost difference between the two clusters. Drive failure rates are a third lower for the 2TB drive cluster but the larger drive costs more per GB.
Drive Failure Rate
(5 Year time to fail)
Petabyte 60 Server Cluster with RAID-10
== failure cost == HDD failrate === month ===
Drive HDD labor subtot units (days) fails cost
----- --- ----- ----- ----- -------- ----- ------
2.0TB 140 70 $ 210 1,000 1.825 16.4 $3,518
1.5TB 60 70 130 1,334 1.368 21.9 2,851
These number presumed that both drives had the same MTBF. Digging a bit further we find that the WD20EVDS claims 1 million hours MTBF and the ST31500341AS claims 750,000 hours MTBF. That is, the operations staff can expect the Seagate drives to fail at a rate 1/3 greater than that of the Western Digital drives.
Sidenote: The two drives are in different classes. The Seagate drive spins at a faster rate (7,200 RPM) and claims performance. The slower Western Digital drive (5,400 RPM) claims consistency and lower power. However, the slower rate is fine for the storage cluster which acts as secondary storage.
I will not attempt to sort the apples to oranges comparisons between the two drive manufacturers. I shall take my previous calculations and adjust the 1.5TB drive cost up by a third.
Drive Failure Rate
(Adjusted Fail Rate)
Petabyte 60 Server Cluster with RAID-10
original adjusted
fail HDD === month === === month ===
Drive cost units fails cost fails cost
----- ----- ----- ----- ------ ----- ------
2.0TB $ 210 1,000 16.4 $3,518 16.4 $3,518
1.5TB 130 1,334 21.9 2,851 29.2 $3,801
A reversal of recurring costs. Does it matter? No. Not really. In my opinion, it’s more important to minimize the day-to-day operation hassles. The three hundred bucks a month (one way or the other) is noise. The $60K difference in initial capital costs is significant but not as significant as reliable operations.
Rumor has it that both Seagate and Western Digital will soon release a 3TB drive.
Update 2010-12-17: Xbit Labs reports on Hitachi’s new sixth-generation perpendicular magnetic recording (PMR) which “enable 3.5″ hard drives with 4TB or even 5TB capacities.”
My final ponderings on this fantasy cluster looks at the impact of future HDD capacities. For this, I simply speculate on the cost per gigabyte. If you have pricing information more closely tied to reality, please let me know. 🙂
Drive Failure Rate
(unadjusted 5year rates)
Petabyte 60 Server Cluster with RAID-10 (or equivalent)
$/GB == failure cost == HDD failrate === month ===
Drive HDD labor subtot units (days) fails cost
----- ----- --- ----- ----- ----- -------- ----- ------
5.0TB 0.100 500 70 570 400 4.564 6.6 $3,762
4.0TB 0.090 360 70 430 500 3.651 8.2 3,526
3.0TB 0.080 240 70 310 667 2.737 11.0 3,410
2.0TB 0.070 140 70 210 1,000 1.825 16.4 3,518
1.5TB 0.040 60 70 130 1,334 1.368 21.9 2,851
This isn’t the end of the tradeoff line. At some point, 3.5 inch HDD will yield to the 2.5 inch form factor. The larger larger drives just won’t be available. That dynamic will change the equation. GPU cards will become increasingly more capable. CPU core counts will increase. RAM costs decline. Fewer servers will be needed. Fewer racks. Less power. I’m sure in my naïveté I’ve underestimated much here. However, I do believe one day the entire data center used to build Avatar will fit inside a 40 foot shipping container. And then, some time later–but not much later–that compute power will shrink to fit in a 20 foot container. And so on. The important point is that the capital costs (exluding facilities) for this fantasy cluster is under a million dollars. And it gets cheaper by the day.