Persistent and Spillover Storage (Hard Disk HDD or Solid State SSD)
Disk I/O is extremely slow, and for most of my work, disk and network I/O are the main bottlenecks unfortunately. If you notice that your jobs are disk I/O bound and they do not need to be, rewrite them to rely more heavily on RAM. In most cases, there is no workaround. One method I recently came across to reduce disk I/O is to write file data to Redis, and then let Redis write all of the data to disk at once when RAM is exhausted. Workarounds like these require some cleverness to implement.
There are some metrics you can consider when purchasing a hard disk, but I have not noticed huge performance gains when optimizing these metrics. I mostly stick with commodity hard disks (due to disk failures being common) rather than spend hundreds or thousands on special disks.
Spillover storage for swap. In desktop systems and commodity servers, the amount of RAM that can be installed is limited. Some boards will only allow 8GB with 16GB being a more common upper bound. My 2008 MacBook Pro only accepts 4GB! If you find yourself maxing out your system board, and are not ready for an upgrade, consider purchasing a solid state drive and configure it to be used as swap. SSDs are much faster at reading data, and in most cases writing data as well.
When RAM is exhausted, the OS starts to swap. That is, the operating system attempts to use hard disk space to store contents of RAM that are the least used. When the data stored on the hard disk is needed, it is swapped into RAM, and another older piece of data is swapped out to disk. Swapping is extremely slow, and it will kill the performance of any running jobs. If swaps occur too frequently, the system will begin to thrash; the hard disk makes lots of noise, and the system becomes non-responsive. By using an SSD for swap instead of the hard disk, swapping becomes much faster because there is no read head that must search and seek for bits over multiple platters.
The downside of SSDs is that they are more expensive than standard hard disks; SSD space is about $3/GB whereas HDD space is about 15 cents/GB. Coughlin Associates have shown that HDD and SSD prices tend to decrease at the same rate, about 50% per year. Unfortunately, this means that the gap in prices between HDDs and SSDs will remain for some years. Since SSDs have no moving parts, they are less likely to be damaged if dropped and less likely to fail due to mechanical failure. On the other hand, it has been suggested that once the controller on an SSD goes bad, the data is lost forever. Due to the price gap between HDDs and SSDs, I refer to SSDs as “spillover storage” for common users because it enables swapping to become much faster. Many users, though, choose to install SSDs to hold the operating system as well as applications for fast boot and system throughput, and then use a standard hard disk as “spillover” for their files.
There are several performance metrics printed on hard disk drives and SSDs that you may want to consider:
- Interface type. The interface is the bus that connects the drive (either HDD or SSD) to the system board. The speeds listed for the interface specify the transfer rate for data transmission between the output side of the device and the rest of the computer. Higher is better. The interface speed does NOT specify how fast the drive is itself. It is theoretically possible to have a very fast interface speed, and a very slow drive, and the slow drive would be the bottleneck between the two. Common interface types include
- Serial ATA (SATA): SATA III (max 6 Gbits/s = 600MB/s) , SATA II (max 3Gbits/s = 300GB/s), SATA I (max 1.5Gbits/s = 150MB/s),
- Parallel ATA (PATA) (sometimes marketed listed as EIDE, ATA-33/66/100, UDMA etc.). Considered obsolete, but noteworthy.
- SCSI (up to 5120Mbits/s = 640MB/s). Typically much more expensive than SATA/PATA, and were at one time superior drives but SATA has caught up.
- SAS stands for Serial Attached SCSI and combines the high performance of SCSI with the high capacity storage of SATA. SAS configurations can generally cater to more disks than SATA alone. SAS drives are much more expensive than SATA drives and are typically found with spindle speeds of 10,000rpm or 15,000rpm.
- Cache. Higher is better. Disk cache is used to store “files” or pieces of files, that are accessed frequently. Reading from the hard disk is very slow, so by caching frequently accessed pieces of files, the user can theoretically achieve quite a large speed boost. Due to the number of frequently accessed files however, it is unlikely that the cache helps at all for HPC, let alone typical use.
- Seek time is the delay associated with the head moving between tracks on the surface of the disk. Lower is better, although SSDs record seek time differently since there are no moving parts.
- Spindle speed is the speed at which the platters of the disk spin about its, well, spindle and is reported in rotations per minute (rpm). Higher is better. Slower spindle speeds induce much higher latency in transferring data from the disk to its interface (and thus the rest of the system). Common spindle speeds are 5400rpm, 7200rpm and 10000rpm. The difference in price, performance and heat is significant between 5400rpm and 7200rpm drives, but I have found 7200rpm drives to perform very well for their price. 10000rpm drives are pretty expensive and I have not found a use for them yet.
Brand is mainly personal preference. I have had very good experiences with Seagate, and more recently Hitachi drives. Personally, I am not fond of Western Digital, yet you will find plenty of people that prefer WD over Seagate etc. I strongly suggest purchasing drives retail. For several years I purchased hard disks at the LA Computer Show at wholesale prices. These drives had ridiculous failure rates, some failing within 3 months of purchase; it was rare to find a drive that lasted longer than 1 or 1.5 years.
My new system will reuse the hard disks in my current system. It will have a 2TB Hitachi HDS722020ALA330 (Deskstar 7K2000) 7200rpm SATA revision 2.5 32MB and a 2TB Hitachi HDS5C3020ALA632 (Desktar 5K3000) 5900rpm SATA III 32MB. The slower disk is only used as a backup for the faster disk.
Some Notes on RAID
I do not want to go too much into RAID because it would take up pages and pages. RAID (Redundant Array of Independent Disks) is a way to combine several disks into one or more logical volumes and provide some level of data integrity protection. RAID requires a RAID controller card, or it must be supported by the system board. I recommend using a controller card because if the system board fails, you could potentially lose access to the data in the RAID. The major downside to RAID for me is that it requires several disks that must be close to identical. Most RAID configurations will be limited by the size of the smallest disk in the array if the disks are not identical. There are several variations of RAID.
- RAID 0 provides no fault tolerance or redundancy and simply allows the user to combine several disks into one large volume.
- RAID 1 uses additional disks to mirror data. For example, with two disks in RAID 1, both disks contain identical content.
- RAID 2 distributes sequential bits onto different disks and uses Hamming-code parity on these bits which is stored on dedicated “parity disks”. The parity disk configuration requires disk space and decreases the total usable disk space by the amount used by the parity data.
- RAID 3 is similar to RAID 2 but sequential bytes are located on different disks.
- RAID 5 distributes data and parity across the array, but one disk can fail in the array with a performance penalty, but no data loss.
- RAID 4 is identical to RAID 5, but parity is stored on one dedicated disk.
- RAID 6 is similar to RAID 5, but allows two disks to fail in the array.
- RAID 0+1, 1+0, 5+1 combine features from two different RAID configurations.
So how much did this set you back?
About $2500, which isn’t too bad for me. A MacPro would have been close to $3000 for a low end. I was eyeing the MacPro…but I am kind of over Mac. I am still not settled on ALL of the hardware in this purchase. I’ve considered exchanging for a faster CPU clock speed, or a different motherboard, but we will see.
I kinda read this as “just buy top of the line everything except maybe the motherboard and you’ll be good.” Any experience with tradeoffs? How would you prioritize cpu vs ram vs disk io speed?
In my experience multiple cores don’t necessarily give a huge boost in HPC if you aren’t writing for it. I don’t tend to write multithreaded code because of the complexity and debugging issues that go with it. In that case, more cores won’t help my code finish faster; it’ll only help if there are multiple jobs running at once. Xeons are great if you’re switching between a boatload of threads like a web server might or you’re concerned about power/overheating, but in my experience they aren’t worth the extra cash you pay. In that situation, clock speed can be a lot more important than you give it credit for here.
Well, to be frank, this upgrade has been way overdue, and I did a bit of “throwing money” around. The purpose of the post was to document the decisions I made and what else was out there. Of course it will come across that way, because in an ideal world, if everyone could afford the top of the line hardware, we would all be good computing-wise. It would be too difficult to tailor this post to everyone’s needs.
In terms of priority, *for my research use* I put RAM as the most important, followed closely behind by the CPU and the number of threads it can run concurrently (not so much the clock speed), followed by disk I/O. I find that for my work investing in more or better RAM gives the best bang for the buck, followed by the CPU. Although a lot of my work is disk bound (crawling), faster disks are so much more expensive and are out of my budget. Because of this, I had to make the tradeoff to favor upgrading RAM over disks. At least there are some tricks that can be done with RAM to prevent overuse of disks.
To me “high performance computing” goes hand in hand with parallelism. Of course if the code has not been written to take advantage of the cores, the extra cores are useless. I would expect that if someone is buying a multicore processor, they intend to program to use the cores. For my research, not programming to take advantage of these extra cores would qualify as *not* high performance computing. I never suggested that CPUs with more cores will make things run faster; just that more work is done at a time. Hadoop is the most trivial example I can give where code takes advantage of multiple cores, OpenMP is another.
Everyone will have their own opinion, but for my use, based on experiences I have had using servers containing Xeons, and the issues I faced without a higher-end processor, the Xeons were the way to go without a question.
why wouldn’t you drop that money on several cheap computers, instead of one expensive one?
for $2500 you could have got 14 motherboards, 14 AMD Phenom quad-core CPUs and 14 x 1 GB ram
Sigh. I’ve heard that. While a cluster would be cool, I tend to use AWS when I need a cluster setup. Also just too much of a pain to have to build that many systems. I don’t spend money often, so I was ok with putting out a one time large purchase ;). This machine really serves two purposes: for development of jobs to be shipped to AWS (time = $), and to prevent me from having to use AWS for high-memory, high-speed applications.
Over the past 20 years though, I have collected a lot of old machines that might an ok cluster, just obviously not high end.
Heya
Is SATA3 Working on this motherboard?
I cant find any info saying that this Motherboard can support SATA3. Only Sata 2 in official notes.
Let me know if u got Sata 3 to work.
Thanks bye
I got D18 too, works like dream so far!
It may not support SATA III. Drives seem to lag far behind the interfaces though. Most drives can barely perform at SATA II interface speeds.
Btw forgot to say, U might be better off with EVGA Server motherboard- They can OC CPU = 12 cores x 4-5 ghz after OC…. That would be pretty much power…
I am having trouble receiving the D18. It seems to be a rare beast. It’s taken almost a month now. Considering cancelling. If I do I will probably go with the EVGA or Supermicro.
I can sell u mine D18 and jump to EVGA tbh. I could use some extra OC for my work 😀
Which OS do you plan on running?
I finished the system 🙂
I put in the same HD that was running on my old server. I am running 64bit Ubuntu 10.04 (Lucid). Runs great!
Considering upgrading to 11.04, or switching to CentOS, but I really don’t have too much of a reason to switch to CentOS.