jimmys69
MOODerator
Good question. I am curious myself.
Choosing an external hard drive for recording
- Thread starter warartistrecord
- Start date
Good question. I am curious myself.
S
still_fiddlin
Guest
If you've got a slot in your desktop to spare, eSATA should be faster than USB 2.0, and competitive with USB 3.0. USB 3.0 has the advantage that you can always plug your 2.0 device into that port if you need it, though. With eSATA, you'll be limited to the hard drives that come with it. I don't know what else uses it, offhand. I have no idea how the cards compare in cost, but external drives with eSATA might be cheaper now - likely to change over time, though.
On your laptop, I'd say use it if you've already got that connector, but anything external has to be plugged in to something, so what does that gain you? Just protocol conversions back and forth, and transfer rate will be limited by the weakest link. Either would be an upgrade over USB 2.0 if you are shopping for a new system, with the same thought that USB 3.0 will always be usable for more devices. (I think most newer notebooks have switched entirely to USB 3.0 - even my 2+ year old HP Envy has USB 3.0.)
On your laptop, I'd say use it if you've already got that connector, but anything external has to be plugged in to something, so what does that gain you? Just protocol conversions back and forth, and transfer rate will be limited by the weakest link. Either would be an upgrade over USB 2.0 if you are shopping for a new system, with the same thought that USB 3.0 will always be usable for more devices. (I think most newer notebooks have switched entirely to USB 3.0 - even my 2+ year old HP Envy has USB 3.0.)
dgatwood
is out. Leave a message.
eSATA will pretty much always spank USB 3.0, both in throughput and latency. Even if you ignore the fact that eSATA has a 20% faster theoretical maximum bandwidth, there's too much overhead in the USB protocol, and the silicon has to do way too much work. For that matter, as long as the silicon does more than zero work, it will be slower. When you use an eSATA drive, you're using a SATA controller that's hanging off a PCIe bus. So you get one conversion from PCI to SATA, and the drive speaks SATA. When you use a USB 3.0 drive, you get one conversion from PCI to USB 3.0, then you get a second conversion from the USB protocol to SATA. No form of USB will ever be able to match eSATA in terms of latency unless either we start seeing external hard drives that use USB mass storage commands natively (ick) or they stop updating the eSATA standard, and in all likelihood, the same is true for throughput.
USB drives are also much more likely to have firmware bugs in my experience. My favorite are the USB drives that tell the OS that they've finished writing data long before they actually do. If the OS pays attention to what the drive says and shuts down power to the port early, you get data loss. Or if the user pays attention to the drive icon going away instead of the drive's activity light going out, you get data loss.
Or the USB flash controllers that corrupt large block reads and writes over a certain size, resulting in bizarre, banded images when you use them on operating systems that actually know how to read in chunks larger than 512 bytes.
Or....
Incidentally, nothing uses SATA or eSATA other than hard drives and other storage devices (optical drives, etc.). That's why it is better at the task than USB—it does one task and it does it as efficiently as it can.
S
still_fiddlin
Guest
Just a couple of points here - both eSATA and USB 3.0 are suitable for video transport - the distinction for audio processing shouldn't be a concern to most of us. I wouldn't get hung up on the speed difference, but as always, it's good to keep the storage bus and audio interface bus independent, so if that helps you pick, either should work.
The OS doesn't do I/O to a USB device, it does it to a storage device; i.e., it's not *USB* saying that the I/O has completed, but the storage controller, which is embedded in the drive (in anything being considered for home use).
Yes, there's a protocol translation, same as with FireWire, and, as you point out, eSATA has the advantage of removing that. But, I wouldn't blame the drive or firmware for the kinds of faults you see - same drives and embedded controllers are used in all configurations. If you have problems, I'd be more suspicious of a poor quality I/O stack component, such as a driver. The file system is not going to say the I/O is complete until the *device*, not the bus, communicates that it is. If the device says it's complete when the data is in the buffer (and most do), you will have a problem if you unplug at that point, regardless of the transport. Always wait for the light to go out, and on any OS, always "eject" the device before unplugging. That way the data is guaranteed to be written.
dgatwood
is out. Leave a message.
Video is actually relatively easy. It's a single compressed stream running at roughly a fixed data rate, usually measured in double-digit megabits per second, or single-digit megabytes per second, usually from a single, nearly-contiguous file. Compared to that, audio work is serious abuse. When you are reading from thirty or forty files at once with limited-size buffers, latency can make a big difference.
To give you an idea of the problem, I've seen measurements on a lot of drives that are dual eSATA/USB, and the USB side has typically an extra 5-10 milliseconds of request latency, and sometimes even more than that. This is not just a problem with certain configurations; USB is simply inherently heavyweight for mass storage purposes. So you're potentially doubling the amount of time that the drive takes to return data whenever your DAW requests it. That's a huge difference, and when your DAW is waiting until fairly close to when it needs the data before requesting it, that extra 10 milliseconds per request can make the difference between a functioning audio recording setup and a sputtering, stuttering mess.
Now I'm not sure how typical those numbers are—some USB drives might be significantly better than that—but the point is that those are real-world numbers from actual hardware that you might pick up at your local computer store.
That's simply not true. In the case of a USB-attached hard drive, the OS sends USB mass storage requests to a USB-to-SATA bridge chip inside the enclosure. The OS doesn't know anything at all about the drive itself beyond what that bridge chip tells it. The storage medium behind that bridge chip could be anything from an actual hard drive to a Compact Flash card (in IDE mode) for all the OS knows or cares—or even a big pile of RAM chips and a battery. As far as the OS is concerned, it is a USB mass storage device, not a SATA or ATA device. It has absolutely no visibility into anything beyond that bridge chip; USB-to-SATA bridges are not transparent bridges, or even semi-transparent. Anything the OS knows about the drive beyond that bridge chip is at the discretion of the bridge chip, which can just plain lie if it wants to.
Now if we were talking about Thunderbolt, you'd probably be right. I'd expect a Thunderbolt disk to look like a SATA drive hanging off a SATA controller, which in turn would be hanging off a PCIe bus. The fact that the PCIe bus is on the opposite end of a long wire is largely an implementation detail.
From the computer's perspective, the only device it knows about—the USB-to-SATA bridge chip—is saying that the I/O is complete. That's my point. Because the OS cannot actually talk to the SATA device directly except by sending requests that the bridge chip must interpret and translate, the bridge chip can lie as much as it wants to. And to maximize performance, some of them do. BTW, it's not the I/O that's an issue; all hard drives lie about that, claiming to be finished as soon as the data is in the write cache. It's the USB controllers lying about the "flush cache" command that can cause problems.
You see, at some point, somebody realized that cache flushes take a long time, and thought, "Hey, saving files would be a lot faster if the enclosure just ignored cache flush requests and told the OS that the command had finished executing, or perhaps sent the command to the drive and then immediately told the OS that it was done without waiting for the drive to actually perform such a slow operation." You see where this is going.... And then the drive gets unmounted, the computer issues a flush cache command, and upon the USB-to-SATA bridge chip replying to indicate that the command completed successfully, the OS shuts off power to the port. And now you just lost the bits where you marked the file system as clean, and probably some other recent metadata writes. Oh, and your metadata writes are no longer synchronous, so you pretty much can't trust journaling at that point, because there's no way for the OS to have any idea whether a metadata change actually made it out to disk or not. And so on. It's a level of brain damage that pretty much blows away all the assumptions that OS vendors have to make in order to make computers robust....
I won't name names, but this was a real problem.
S
still_fiddlin
Guest
I'm sorry, I don't know you, and you don't know me, but I've been paid to work on, with and in computer operating systems since 1983, and worked on computers and programmable devices since the 1970s. This information is misleading.... The OS doesn't know anything at all about the drive itself beyond what that bridge chip tells it. The storage medium behind that bridge chip could be anything from an actual hard drive to a Compact Flash card (in IDE mode) for all the OS knows or cares—or even a big pile of RAM chips and a battery. As far as the OS is concerned, it is a USB mass storage device, not a SATA or ATA device. It has absolutely no visibility into anything beyond that bridge chip; USB-to-SATA bridges are not transparent bridges, or even semi-transparent. Anything the OS knows about the drive beyond that bridge chip is at the discretion of the bridge chip, which can just plain lie if it wants to.
...
An application writes data to files, and in rare cases, devices. No application writes data to USB. In the case of storage devices, files reside on those storage devices. Yes, the actual hardware is not exposed to the file system, and hence application, but the file system "talks to" a storage device driver, which is designed to communicate with the actual storage device. Just because that device is connected to a USB hub does not mean that USB has anything to do with the processing of the data being transported over it, except insuring that both ends of the communication (driver and device) use USB properly. It's the device driver that decides whether to cache data in system memory, or send it to the device, and it's the device that decides whether to tell its driver that an I/O is complete, whether that means the data is really stored or sitting in a cache buffer. USB does not participate in that communication, except to pass the data back and forth. That transmission can be asynchronous or not, but the driver does not hear from USB that the I/O is complete, because it is waiting for that information from the device. So, the driver does not tell the file system that the I/O is complete because USB says so; it's because the device says so, or possibly the driver cached the data locally in system memory and never sent it in the first place (yes, there are drivers that do things like that). It is *not* USB doing this.
On Windows, if you open Device Manager, and "view devices by connection" you can trace down from the PCI Bus through the USB root hub to the hub the storage is attached to. (NB: do not connect your important external storage through an external hub.) The storage device is controlled by a disk driver, not a USB driver.
Having said all that, it's important to understand that in all these complex I/O chains there are a lot of moving parts, and asynchronous I/O is the normal way things are done, i.e., you drop something in the mailbox, and go about your business, expecting the return receipt sometime later. Stuff can go wrong. But, it's up to each layer of the stack to insure it manages that, either by filling in any missing bits or resending, whatever, and never close the device until a flush has been done to insure everything in the pipe gets to its final destination. That's where bad drivers and applications get to screw things up, and they have, especially MSFT's USB stack pieces in the first couple of iterations. I get the hesitancy to embrace it, but I've moved on
, and they seem to have it pretty together for USB 2.0 at this point. (FWIW, I use a MacBook with USB 2.0 for the record i/f, and FW for the external storage - I'm not dogmatic about this stuff.)Edit: Let me clarify this a bit - each piece in the I/O chain can "complete" the I/O. So, if at the point the data packet passes down the cable to the USB interface that's part of your external drive, that interface/chip, which is the last physical device seen says "done," then the result is the I/O completes. However, this is no different from FireWire, or anything else for that matter, since there's always a terminating device behind which the hardware & firmware do some magic. Regardless, the file system only gets notified that the I/O is complete when the very last "device" says so, assuming, of course, it asked for completion status. There's always a little bump in the cable somewhere that can "lie" I suppose, but the good actors don't.
Last edited:
dgatwood
is out. Leave a message.
I've been working in operating systems for more than a decade, and have done a fair amount of block-storage driver development in that time.
You must have misread what I said. I never suggested that an application did anything of the sort. An application writes data to a file. The operating system's VFS layer translates that into a series of block operations. Down one layer further, the block storage driver stack translates that into a USB mass storage request. Next, somewhere way out on the USB bus, a bridge chip translates a USB mass storage request into a SATA request.
What I said was that the OS doesn't know that it's an ATA disk hanging off the USB bus. As far as the OS is concerned, it is talking to a USB device, not a SATA device. It has to trust that the USB-to-ATA bridge chip is translating requests sanely. The problem is that the translation step is very complex. USB mass storage packets don't even resemble ATA packets. As a result, there's significant potential for bugs, some of them rather severe.
If it were simple encapsulation of ATA packets, it wouldn't be a problem. But it isn't. USB mass storage is an unholy nightmare of a protocol (just like pretty much all the USB protocols, IMO). ATA is a very simple protocol (relatively speaking). Talking to a SATA drive using USB mass storage requests is a bit like having a conversation in which you say "I'm about to say a word. Hello. Did you hear that word?" followed by "I'm about to say a word. My. Did you hear that word?" and then having the person on the other end putting the words back together into a sentence and translating it into French before speaking it to somebody else....
The actual hardware (meaning the SATA drive) is not exposed to the drivers, either. The OS sees a completely opaque USB-SATA bridge, which translates USB mass storage requests from the drivers and turns them into SATA requests, or translates them into addresses and writes them to a flash chip, or translates them into any number of other things.
That is misleading. First, I never said anything about USB hubs. Those are passive devices that just pass packets. I said USB-to-SATA bridges, which are active devices that take a USB mass storage request from the OS and translate it into an ATA request that they then send to the actual device (the hard drive), and translate the responses from the actual device and send then over USB to the OS.
It's like having a wall wart power transformer plugged into the end of an extension cord. Although the power cord is a passive component that just passes power on, the transformer isn't.
Yes and no. The device driver decides whether to cache data in system memory, but there can be multiple levels of cache between there and the actual magnetic medium. A typical hard drive has its own cache, called a track cache, that caches reads and writes in whole-track increments. The reason for this is that if the drive handled every single-block request in order, performance would suck horribly because each request would, on average, have to wait for the disk to spin halfway around. That track cache, when dirty, must be properly flushed to disk before the drive is powered down. If that doesn't happen, you get data loss.
If by USB, you mean the USB controller on the motherboard, you're correct. However, the USB endpoint (the hard drive enclosure) most certainly does participate in the communication.
Notice that in that photo, the actual device—the ATA hard drive—is not visible in the device tree. The only thing you see is "ST315005 41AS USB device". In this particular case, the device name happens to be similar to the name of the SATA drive beyond it, but what you're seeing there is the USB-to-SATA bridge. If you use some other bridges, such as an Oxford 912 with the stock firmware, you'll typically see "Oxford Semiconductor" there, and the OS will have no idea of what the hard drive beyond it is. The only reason you see anything Seagate-specific at all is because Seagate provided a custom firmware image for its USB-SATA bridge that was programmed to call itself a "ST315005 41AS USB device".
That's true for FireWire. It's not true for eSATA because AFAIK, there is absolutely no translation being performed in the enclosure—only signal amplification and pulse reshaping. There's basically no way for an eSATA enclosure to lie because they're dumb as a brick.
And that was what I was trying to convey.
warartistrecord
New member
I'm definitely getting a new hard drive (probably several), but I'm not connecting my computer to the internet. I'm too paranoid about it getting viruses and dying.
warartistrecord
New member
2 Questions, how can I know if the cooling system is adequate? and is there a way to isolate the sound of the PC without covering it in something and thus increasing the heat?