The 24-bit challenge

[Shailat]

Another thing to think about........

On what kind of system are the judgers listening to.
Is it on computer speakers?.

 

[pchorman]

when trying to discern the subtlest differences in sound quality it goes without saying that quality speakers, or better yet headphones, are required. if people don't care to use the right tools for the job they not only invalidate the test but they aren't paying enough attention to detail anyway for it to matter. that's the way I see it anyway.

The test results were documented in a separate post and have now dropped off the front page, but they essentially show that very few could tell what was what, and the few that nailed the highest quality snippets fell within the expected distribution of random lucky guesses, if I understand the crazy probability calcs applied.

Somehow I think a simple binomial distribution should have been used - you know, that simple equation with p and q (ref. either Papoulis or Stark and Woods, or any popular graduate engineering probability text) that yields the probability of picking objects from a set in specific order without replacement.

 

[Ethan Winer]

Shailat,

> I have heard a 20 bit card kick a 24 bit card's ass more then once. <

Agreed.

> I'm not sure your short clips can actully prove your point. <

But it's a start.

> And.....your cat is way to fat. <

No kidding!

--Ethan

 

[Ethan Winer]

bd,

> There was no 24bit file represented in the test. <

Yes, but there's no consumer standard for distributing 24-bit files. So any advantage 24-bit could offer is realized only after dithering down to 16 bits! And if you can't tell the difference after dithering, then why bother in the first place?

--Ethan

 

[Ethan Winer]

Shailat,

> On what kind of system are the judgers listening to. Is it on computer speakers? <

I hope not! Some of the respondents said they used headphones and a few others mentioned various speakers.

--Ethan

 

[bdemenil]

Yes, but there's no consumer standard for distributing 24-bit files. So any advantage 24-bit could offer is realized only after dithering down to 16 bits! And if you can't tell the difference after dithering, then why bother in the first place?

1. As 24 bit is only dithered to 16 at the end of the mixing process, this does not address the value of using 24bit source files.

2. When sending your mix to mastering, you would want it to be in the best possible format. If mastering in analog (as many do), your 24bit mixdown would be converted to analog by a DA capable of taking a 24bit input - thus you would benefit from having your mix in 24 bit. If mastering in digital, you could concievably even mix down to 32bit, as many audio applications are capable of reading 32bit files, and process internally at 32 bit anyway. The digital mastering process would surely also benefit from working from a higher resolution source.

3. DVD audio is a standard which accepts 24bit - although it is not much in circulation yet.


Once again, nothing in the test compares 24bit to 16bit. It only compares dither to truncation - which is not the same.

 

[Ethan Winer]

BD,

> As 24 bit is only dithered to 16 at the end of the mixing process, this does not address the value of using 24bit source files. <

I already acknowledged that! If you'd like, I'll withdraw the test, erase all of the threads that discuss it, and never bring it up again.

> The digital mastering process would surely also benefit from working from a higher resolution source. <

Maybe, maybe not. Until that is tested specifically, we'll never know for sure.

--Ethan

 

[bdemenil]

I already acknowledged that! If you'd like, I'll withdraw the test, erase all of the threads that discuss it, and never bring it up again.

OK...OK...I didn't mean to offend - if you already said this somewhere in this long thread, i missed it - sorry.

 

[ccryder]

1bit=6.0dB? or 6.02dB?

I do believe it's more like 6.02dB, not 6.0dB. (It might even be more than 6.02dB, that I'm not quite sure of.)
So the test is flawed, and instead of playing the Sound Forge volume game to to create truncation of bits for the test files, something like Samplitude's *2 and /2 Amplitude functions should be used. Simple bit-rolling is all that is required to create proper test files.

'Twas a valiant effort though...

-Dan

 

[Ethan Winer]

Re: 1bit=6.0dB? or 6.02dB?

CC,

> I do believe it's more like 6.02dB, not 6.0dB. <

I don't know why that would be the case, but I'm not a math head either. However, if anything, the extra calculations needed only help to prove my points, since another common myth is that any math done on a Wave file is damaging. I mean, sure, it's damaging in a strictly technical sense. But in practice, a little math is not nearly so bad as so many people say it is.

--Ethan

 

[Ethan Winer]

BD,

> OK...OK...I didn't mean to offend - if you already said this somewhere in this long thread, i missed it - sorry. <

No problem, and I'm not offended at all! Sorry if I overreacted!

--Ethan

 

[Bobro]

Originally posted by ff123


One more point, If File4 and File5 were compared directly against each other, with no other files in the test, then only 6 people in agreement would be required (using the LSD formula) to say the two files are significantly different. The moral? Fewer files to compare at the same time means more powerful results.







ff123







Edit: before somebody asks, I already tried eliminating File4 and File5 from the analysis of the 10 sensitive listeners. It does show then that File2 is preferred over File1 with a confidence > 95%. Problem is, I'm not real comfortable with doing this. The proper method in statistics is to conduct another experiment to confirm results like this.

Thanks, ff123! These statistics are precisely as I expected them to be. As I posted in long and annoying detail on Craig Anderton's forum, the response I sent to Ethan was this:





File 2: the best one


File 3: like File 2, but with the life sucked out


File 1: something odd about this one


File 4: noisy


File 5: very noisy





and then I wrote "so I guess it would be..."





File 2: 24>16 dithered


File 3: 24>16 truncated


File 1: 13-bit


File 4: 11-bit


File 5: 9-bit





So technically I got 4 and 5 right, and, if I understand the statistics correctly, made the same technical mistake as other "sensitive" listeners by assuming the best sounding file, File 2, would be the same as the "best" according to theory. Musically but not statistically of interest, my wife (who administered the blind test over headphones and took the test as well) , rated File 2 as "the only one not poo-poo" (translated and censored ) and 4 and 5 as "like an old radio" but didn't care to rank further than that, knowing even less about dither etc. than I do.



The conclusion I draw from this test is- the less processing, the better, and Ethan perhaps you should consider different dithering settings or algorithm?

I just tracked the background noise of an old analog synth through an analog filter, just zuzzzzing softly away, onto some new stuff and I'm quite pleased how it "pre-dithers" and makes a soft bed for the final 16/44 file.


As far as the Delta. I own two Delta cards, the older DiO 2496 and the new Audiophile 2496. With both, I get "microclicks" when using the onboard convertors. Little one-sample flat spots in the waveforms, audible to some and visible to all in the excellent spectrum view in CoolEdit as ultrathin columns of noise all the way up the frequency spectrum. I got the external A/D Flying Calf and haven't had one of these microclicks since. Judge for yourself. Just bought a second-hand but still in the box SEKD ADA external unit, it'll be here in a few days and I'm curious as to whether the difference will be audible.

-Cameron Bobro

 

[ff123]

The proper way to interpret the group statistical results is to say only that files 4 and 5 were distinguishable.

There is the suggestion in this data that file 2 was rated better than file 1; however, that was not proven by this experiment -- a confirmatory experiment must be run to test this question separately.

An individual listener can "prove" with high confidence that he actually can hear a difference (as opposed to just thinking that he can hear a difference) by performing an ABX test.

See: http://www.pcabx.com/

If an individual listener can demonstrate that he can achieve significant ABX results for file 1 vs. file 2, I would like to know about it.

If someone decides to perform an ABX test (it won't be me, because I didn't hear a difference beyond the 11th bit), I would recommend using my ABX probability calculator (http://ff123.net/abx/abx.html) to figure the probability that you arrived at your results by chance. Once you perform enough correct trials to get to a p-value of 0.01 or lower, you should immediately stop (you might mess up your results by continuing!) and crow about your highly sensitive ears.

It is possible to achieve p=0.01 or lower with as few as 7 trials (as long as you get them all correct).

And no fair performing multiple ABX sessions to "cherry-pick" the best one. You must count all trials in every session in figuring out your overall p-value. Of course, this doesn't preclude you from breaking up ABX sessions over time to prevent listener burnout. In fact, short listening sessions are recommended.

ff123

 

[Bobro]

The proper way to interpret the group statistical results is to say only that files 4 and 5 were distinguishable.

There is the suggestion in this data that file 2 was rated better than file 1; however, that was not proven by this experiment -- a confirmatory experiment must be run to test this question separately.

An individual listener can "prove" with high confidence that he actually can hear a difference (as opposed to just thinking that he can hear a difference) by performing an ABX test.

See: http://www.pcabx.com/

If an individual listener can demonstrate that he can achieve significant ABX results for file 1 vs. file 2, I would like to know about it.

If someone decides to perform an ABX test (it won't be me, because I didn't hear a difference beyond the 11th bit), I would recommend using my ABX probability calculator (http://ff123.net/abx/abx.html) to figure the probability that you arrived at your results by chance. Once you perform enough correct trials to get to a p-value of 0.01 or lower, you should immediately stop (you might mess up your results by continuing!) and crow about your highly sensitive ears.

It is possible to achieve p=0.01 or lower with as few as 7 trials (as long as you get them all correct).

And no fair performing multiple ABX sessions to "cherry-pick" the best one. You must count all trials in every session in figuring out your overall p-value. Of course, this doesn't preclude you from breaking up ABX sessions over time to prevent listener burnout. In fact, short listening sessions are recommended.

ff123

The ABX program is great! Thanks. Well, I can hear which one is which in the ABX, I'm really grateful you've removed any doubts I had that I might be imagining these things. Because when my wife conducts blind tests on me it is entirely possible that I know her well enough to second guess what "random order" she's using.

Here's the BIG BUT: I've been firing up the ABX with File 1 and 2 on and off since yesterday morning, just in between other things with long breaks. A quick listen to first few seconds of each file, that one's that, no prob.

BUT those were spaced sets of 1, 2 and 3 trials over a whole day.
AND I wasn't really ABX'ing, I was listening for File 2, which has more pizazz than File 1, and going from there. In other words saying which one was which and then matching X to A or B. If this
was with files I'd never heard and whose character I didn't know beforehand, I guarantee that I would have done much worse.

I first tried a longer session, 13 in quick sucession without pause (that's the "same" riff 39 times in a row, LOL). This kind of trial is horrible. My ears crapped out after 5 trials. From number 6- couldn't hear one damn bit of difference and just went on feeling, plain and simple.

Masochistic run of 8 out of 13 plus widely spaced runs of 1, 2 and 3 adding up to 11 out of 11- altogether, 19 out of 24, .004.

The real-life answer AFAIC- I can hear the difference with fresh ears but NOT with the slightest fatigue or loss of concentration.
I would say that many people may be deluding themselves into thinking they can't hear differences or even more likely just don't have detailed enough monitoring systems. I'm listening over Sennheiser HD-600's.

It bothers me a great deal to be scientifically shown how quickly my perspective goes out of whack, so much for "crowing about hearing".

Forgetting the numbers, what are the MUSICAL conclusions? Well for one thing I'm very glad that I'll soon be getting some very nice and non-fatiguing studio nearfields. Ethan really is a consumer advocate, with this test he's placed the last stamp of approval on my decision to lay down some fairly serious cash.

Thanks again, ff123, for the ABX program. The playback isn't as detailed as Samplitude's (in the ABX the telltale "waffle" 3/4's of the way through File 1 is not audible) but it's not a mastering program and doesn't need to be. In another test with my own stuff, the ABX debunked an illusion I had about two other files- those particular files are really either the same or "might as well be", which is good to know.

-Bobro

 

[ff123]

I'm convinced you are actually hearing a difference between files 1 and 2!

But can you explain the number of trials again?

13 masochistic runs of which you got 8 correct, plus some more sessions spread out over time which added another 11 total trials (which you got all correct)? This would add up to 19 of 24.

This is the first such demonstration I have seen of someone "proving" with high confidence that he can hear the difference between truncation and dithering at 16 bits.

ff123

 

[ff123]

Tonight I will verify that Files 1 and 2 are within one bit of each other. I took a quick look at the files in Cool Edit to make sure the music wasn't clipping (it's not) and to make sure that they were reasonably aligned in time (they are, within 5 milliseconds of each other).

ff123

 

[Bobro]

I'm convinced you are actually hearing a difference between files 1 and 2!

But can you explain the number of trials again?

13 masochistic runs of which you got 8 correct, plus some more sessions spread out over time which added another 11 total trials (which you got all correct)? This would add up to 19 of 24.

This is the first such demonstration I have seen of someone "proving" with high confidence that he can hear the difference between truncation and dithering at 16 bits.

ff123

First I did 13 trials right off the bat in quick succession. The first were clear, I thought it would be a cakewalk, but the last of those 13 I heard no difference at all, and just guessed. Then I realized what was going on and did the 11 over the rest of the day (actually number 11 first thing this morning before listening to anything else, that one jumped right out). A set of 3, then 2 sets of 2 then four times just one listen. I knew that if I did sets of say 6, it would take a looong time, if ever, to get the "stat" because after hearing a file more than a dozen times in a row, it's a just a blur.

An unfamiliar full orchesration would be a better test I think, especially as there are jangly metallic sounds in these examples, which betray any monkey business in the upper frequencies. But even then there's the huge variable of "which dither? What depth?". I'd be up for doing a test like that with the ABX, though.

Anyway the biggest problem is trying to match up everything you hear with knowledge and practice. Until recently, my mixes sounded, well, like there's a thick blanket over them to be extremely polite. Doesn't do any good at all to hear high frequencies if you don't know that you have to mix in mono to get translatable mixes and no phasing problems, LOL.

To keep things in perspective- at this time I'm not investing in better dither but in some seriously good monitors. IMO the value of these kinds of tests isn't to prove some kind of universal truths, but to help find what's important for the individual to improve their own sound.

Well I've changed my mind about Ethan's motivations. It does take quite a circus to analize the subtilties, and even though they add up, the wise consumer should invest first in more important things than better gadgets. He's obviously invested in building a good room before an expensive soundcard, which is the musical approach IMO.

Whew, back to work.

-Bobro

 

[ff123]

Thanks again, ff123, for the ABX program. The playback isn't as detailed as Samplitude's (in the ABX the telltale "waffle" 3/4's of the way through File 1 is not audible) but it's not a mastering program and doesn't need to be. In another test with my own stuff, the ABX debunked an illusion I had about two other files- those particular files are really either the same or "might as well be", which is good to know.

Arny Krueger, the PCABX author, says this about his program (in rec.audio.pro):

"I don't know what Samplitude does to .wav files, but I know for sure that PCABX is absolutely bit-perfect. I tested this by playing 16 and 24 bit files with PCABX.EXE through the digital output of one sound card and into the digital input of another. BTW, I repeated this test in the analog domain to get an idea of how long it takes to switch files with the switching delay set to zero.

PCABX.EXE is a Visual Basic program that never gets any closer to the files than their names. It starts and stops the Windows multimedia .wav file player on cue, running in millisecond increments.

Samplitude may do some resampling of .wav files when it plays them, as I know for sure there are situations where CoolEdit Pro resamples to play 24 bit files on computers with 16 bit sound cards. It doesn't change the files unless you formally ask it to, but it does do some processing that might be hidden to some when it plays files. AFAIK it won't change sample rates except when formally asked to."

ff123

 

[Bobro]

Arny Krueger, the PCABX author, says this about his program (in rec.audio.pro):

"I don't know what Samplitude does to .wav files, but I know for sure that PCABX is absolutely bit-perfect. I tested this by playing 16 and 24 bit files with PCABX.EXE through the digital output of one sound card and into the digital input of another. BTW, I repeated this test in the analog domain to get an idea of how long it takes to switch files with the switching delay set to zero.

PCABX.EXE is a Visual Basic program that never gets any closer to the files than their names. It starts and stops the Windows multimedia .wav file player on cue, running in millisecond increments.

Samplitude may do some resampling of .wav files when it plays them, as I know for sure there are situations where CoolEdit Pro resamples to play 24 bit files on computers with 16 bit sound cards. It doesn't change the files unless you formally ask it to, but it does do some processing that might be hidden to some when it plays files. AFAIK it won't change sample rates except when formally asked to."

ff123

Is Windows multimedia player the most accurate program for playing back files?

Samplitude lets you select between 16-bit, 24-bit, and 32-bit playback. I don't know how this works, but do have it set to 24-bit playback, as I record at 24 bits. I've been listening from an Audiophile 2496 soundcard.

So it's definitely not 24 bit files be processed down to play from a 16 bit soundcard, rather the other way around. How this affects or doesn't affect the sound, I have no idea.

I've noticed debates about playback accuracy on forums. Except for the noticably (and famously) bad summing and eqs of Steinberg Cubase (which may have changed in the interim), all the audio programs I've heard sound just fine so I don't worry about it.

Maybe it's just a slight difference in playback volume in this case, I wouldn't be surprised.

-Bobro

 

[ff123]

Well, I just compared file1 with file2 using Cool Edit. I trimmed the beginning and end of the files to time align them with each other, then I mix pasted and inverted one file into the other. The resulting difference file looks like the picture below

The differences are plus or minus 1 or 2 bits. The last second or so shows large differences, I'd guess because Ethan applied a fade out.

Presumably, the differences would have been only plus or minus 1 bit if rounding had been used instead of truncation for file 2.

The last second is of some concern because the differences here are more than just the difference between dither and truncation at 16 bits.

I don't believe Bobro's results are invalidated since he writes that he made his comparisons based on the first few seconds of the files and not the end.

ff123