Blind listening tests

famous beagle

Well-known member
I wasn't sure really where to put this, but I figured here would be as good of a place as any, because the guitar world has plenty of "clones," whether it be amps, pedals, guitars, etc.

I was in a music store the other day, and I heard a couple of people talking about analog synths. Neither of them were employees, but the conversation was clearly framed within the scenario of the greenhorn and the veteran. The greenhorn asked about the Behringer clones that have flooded the market this past couple of years.

For those that don't know, Behringer has released very affordable "clones" - for the most part ... there are differences, such as the inclusion of MIDI, USB jacks, and a few other little tweaks here and there - of several classic analog synths from Moog, Sequential Circuits, ARP, etc.

Now, I'm primarily a guitar player, but I do love all things analog, including tape and analog synths. So I immediately perked up when I heard this conversation. Veteran guy didn't even bat an eye. As soon as the word "Behringer" was out of Greenhorn's mouth, he replied, "Oh God no. That Behringer stuff is total crap. Don't go anywhere near it. I can tell the difference between it and a Moog in my sleep."

After hearing this, I immediately thought about all the YT videos I've seen that compare the Behris against the originals and how impressed I was. Forgiving the fact that the originals were sometimes 40 years old at this point --- and most likely sounded slightly different than they did when they were new --- I was still blown away by how similar the Behris sounded.

In the comments of these videos, there were, of course, the purists like Veteran guy, who dismissed them right away as crap. But the vast majority of the comments seemed to be among the lines of "Even if this doesn't sound EXACTLY like a Moog, it still sounds amazing," or "I've played many Moogs over the years, and none of them sound exactly the same. This (Behri) sounds within the range of variation I've seen from Moog to Moog."

In short, they've gotten very positive reviews, generally. And when you consider the price, the tag line from most is always "it's a no-brainer." To give you an idea, Moog's reissue of their Model D monosynth costs about $3,500 new. Behringer's Model D sells new for $300. Granted, the Berhi doesn't have a keyboard -- it's just the sound module -- so you have to add a MIDI controller for it. But still, it's crazy the difference.

I know this is the guitar forum and I've been talking about synths, but the reason is because there are, of course, thousands of A/B videos on YT like this for guitar stuff as well. Pedals are probably the most common.

And it was Veteran guy's comment ... that he could "tell the difference between the two in his sleep" ... that got me thinking about blind listening tests in general.

Now ... maybe he's right. Maybe you could get as close to matching the sounds as possible between the Behri and the Moog and you could play the same exact phrase back to back. And maybe Veteran guy could guess which one he was hearing 10 times out of 10. I personally don't believe he could, but I'm willing to give him the benefit of the doubt.

But I started thinking ... "So what?" Even if that's true, what does it prove? That basically has no application in the real world. Taking it back to guitar land, for example, you never hear a track in which one lick is played with an original vintage Tube Screamer and then immediately afterwards with a JOYO TS clone. And then the track stops and someone says, "Ok ... which one was the original TS?"

What's a more important test, IMHO, is whether you can tell the difference between the original and the clone in completely different contexts, because that's the way they're going to be used. If a clone is good enough to fool you when you don't know what to listen for, then isn't it good enough?

In other words, let's say you're listening to a track and you say, "Wow! Great tone! What were you using?" And the other guy says, "I plugged my 61 Strat into a vintage Tube Screamer and went straight into my 68 Super Reverb. And then you believe him. But in reality, he was using a Mexican Strat running through a JOYO TS clone and into a reissue Super Reverb. Does the universe collapse on itself? Does time and space unravel?

Anyway, it just got me thinking about it. I know the A/B comparison is helpful to some people because it's clear and concise. But the more I think about it, the more I think it misses the whole point.

Any other thoughts?

BTW, I don't own a JOYO pedal, but I do own a Behri Model D, and I really love it. :)
 
I think you need to look at it from two perspectives...that of the audio engineer, and that of a typical listener.

Some people will say that most listeners can't tell much in the way of subtle quality differences, and wouldn't care even if they did, so why bother about it, and for an average listener, it may be fine to have that perspective...but I think from the engineering perspective, it really is about the subtle differences, the specs, the minutia...because audio engineering often goes to great distances to see a 10% or even 5% increase in quality....and rightly so.
It would be like telling an architect that you don't see or care if the left wall is out of level by a 1/4"...yet to him, that's a negative thing.

AFA just "tone" goes...I think it's too subjective, and while you might hear a tone difference between pedal A or B (or whatever)...the next question would be, is one tone any more better or worse to YOU.
IOW...even with the difference, both may be quite useable in context. Of course, if you, the engineer are very familiar with tone A...and that is the tone you want, but pedal B gives you some other tone that my be quite useable, but different...well then, as an engineer, you're not going to be satisfied with B...and you are justified in picking A over B. Someone else may not care, or may prefer B.

I don't much care what other people hear or care about...it has to satisfy me and my expectations, regardless what I choose....so if I do hear a difference in quality or tone, no matter how subtle, I am justified in dealing with that and choosing what works for me...without thinking "no one else will notice or care".

So I may definitely do an A & B both for quality and tone choices...as long as I'm hearing a difference, and even though I know that in the context of some mix, that difference may become less relevant, the fact that it's there initially, I still want to make a conscious choice...and yes, it may be only a 5% difference.

It's not something you need to do with everything, constantly, sit there and A&B and test and compare...but if it's something that you think about, then it's there, it's on your mind, so I don't want to be thinking about AFTER the fact, wondering should I have taken the time to pick one over the other.
Of course....there are times when either choice is close enough and good enough for Rock-n-Roll. :)

AFA gear choices go...I buy both expensive and inexpensive...so it's not a blanket mindset for me.
 
"A/B tests" you see and hear on YT are in fact nothing of the sort...Well, nothing a serious audio engineer would consider valid.

To be valid virtually every variable must be the same for the 'A' and the 'B' devices. Then the recording devices, microphones for instance, must be as beyond reproach as possible. Not good enough to say "but it is the SAME mic in both cases!" Because any colouration could hide or accentuate differences between sources. Similarly the monitors must be as accurate as possible.

In other words, you cannot test a system with crap kit! Whale ***t bit rate MP3s also do not help.

The scientific world does 'double blind' tests which YT stuff never is. Then, at the end of the day the results of such tests are statistical since you cannot rely on just one person's opinion* IIRC you need at minimum of 24 subjects before a result can be called statistically valid? Who has the time, money and kit for that fekker? In any case, the purveyors of kit these day can hardly write a meaningful specification.

*Present company excepted Miroslav. YOUR opinion is totally valid for you.

Dave.
 
I'm inclined to agree with both lines of thinking somewhat. To be fair, I'm a bottom feeder who can't afford the 'real' thing, so you might want to take that into account.

To ecc83's point, most of the YT A/B tests I watch, the conditions are close enough for me to consider the tests valid.
However, as he also pointed out, I don't know how much you can tell from an mp3 over the internet, on what are probably computer speakers.

As to miroslav's observation, for someone like me, under the conditions I'm using them and for the purpose I'm using them - yes, close enough for rock and roll is good enough for me.
If I were a pro, performing and especially recording, my guess is probably not.

My problem from a usage standpoint, with a lot of the cheap clones ala Behri and especially Joyo, is that they usually aren't nearly as well built as the originals, and you get what you pay for.
My problem, from an ethical standpoint, is that the major companies do their R&D to invent these things and improve them, and the clone companies just copy what they've done. Pretty much theft, imo, plain and simple.

That being said, I wouldn't be able to have much of anything if I didn't get clones, and used clones at that. I have both Behri and Joyo stuff. Just an FYI, the Joyo stuff I've tried is pretty noisy, but that's not to say that the originals aren't also, but that's not my understanding.
 
I agree with Beagle's assessment. People will do comparisons, and pass judgement because some piece of equipment doesn't exactly match some predetermined target. The thing that this doesn't address is that the predetermined target is not an absolute. Its not a measurable target like a lap time around a race track, or the height of a high jump. They will also pick a target which has been manipulated, adjusted and processed in the first place. Its like trying to buy a Les Paul and a Marshall and then being disappointed because you can't get the same sound that Eric Clapton got on Hideaway in 1966. Throw out all the variables like the mic used, the recording gear, the mixing and mastering, the phono cartridge, turntable and speakers used to play the record, not to mention that they're no "Eric Clapton". Its the fault of the new guitar not being made to 1959 specs with "old grown wood and vintage correct wire and PIO caps" that's the cause.

I got a real kick out of Sound Pure's microphone comparison. They do them as an a-b-c-d choice. You have to contact them to find out which is which. They want you to listen to the mic and then choose what you think sounds the best. Then you start reading the comment and you see disagreement as to which is the better unit, follow by complaints that they should have identified the samples so they could know which was the best. You can't have a perception bias if you don't know the answer first.

I do comparisons of my gear, so that I can gauge the differences, and to see if one sounds better to me than another. I can file those characteristics away so that if I've got a thin sounding instrument, i can add some bottom end with this mic, or avoid using a noisier mic with an especially quiet situation.

For me this is a hobby, and unless I win a $50million lottery, I'm not buying a bunch of U87s and vintage U47s for thousands of dollars. There are other mics available which have a pleasant, clean accurate sound that can capture a wonderfully nice sound of my guitar.
 
We're in an art industry, that uses science. As such, everything we do is measured, tested and analysed by what we like. It can never be subjective because objective testing is either flawed or just impossible. Take a pure sound, add processing and we have a nice pleasant distortion of the original. Change some parameters and we have a nasty version. How can you measure good and bad distortion? We try to find science terms that fit. We analyse again and determine the balance of harmonic distortion, or other altered components and then assume the differences we find are the magic element, but we really don't have answers. Behringer have shown that they are true pioneers and that the country of origin nowadays really doesn't matter. The virus situation in China shows just how much they produce quietly for the well known brands. Just watch over the next few months when quality equipment suddenly gets short because the insides are filled with Chinese components. The usual made in America, assembled in Mexico arguments should really be Made in America, assembled in Mexico using Chinese manufactured components. This has happend already in the UK with JCB who build heavy plant equipment - diggers and stuff. They import the electronics and hydraulic components. Behringer and their allied other brands make music accessible to far more people. The old arguments are fading away. A genuine Moog will still be sought, even if it goes badly out of tune with a weather or temperature change, but the idea of being able to control a very similar one from your DAW meens as a working instrument, the real Moog will remain enthusiasts only.

I have a couple of Chinese chibsons that I play all the time, because they're gorgeous looking and nice to play. I've had others that went in a skip. It's quite fun buying on spec in the expectation of buying rubbish and being surprised.

I buy large amounts of chinese moving lights for theatre work - most is capable and reliable, but in that world we have snob value too. For some of my theatre contracts we hire the big brands in big quantities, and these things really are just as unreliable. Somehow it's OK for a moving head costing 6 grand to repeatedly fail - because of haze, or pyro dust, or aircon etc etc but the chinese ones are no worse when i buy them for lower budget productions.

These lights generate the same conversations that we're talking about in this topic. Nobody can objectively compare them (like sound) in areas where the physics is appreciatedly as art. The out of focus gobo with those colour fringes - is that because the optics cannot produce a flat field without colour fringeing due to poor quality glass, or is it an efffect that looks nice? Like sound, we can measure light output, and the colour temperature, but shining one on a drumkit at 30ft away can we actually say A is better than B? Maybe that yellow colour has a hint of orange, or maybe that cymbal doesn't seem to sparkle quite so much? If the damn thing misses the cymbal every third time - then that's a bad light.

If we all view the same painting by an old master, do we all like it? Do we all even have the vocabulary to talk about it, and from a distance could us, as viewers tell which is the fake or the original? The veteran synth player is like the old painting collector. Perfectly normal and valid points of view. But all opinions.
 
"A/B tests" you see and hear on YT are in fact nothing of the sort...Well, nothing a serious audio engineer would consider valid.

This is, of course, right, and I didn't mean to imply that the "pedal shootoffs" on YT are legitimate listening tests. There are some videos that will at least try to take the bias out of it by not revealing the A and B until later in the video. But I don't know how consistent they are with all the other variables.

At any rate, I know that people --- engineers as Miro pointed out --- can get extremely picky when it comes to blind listening tests.

And that was kind of my point. I don't see the value in those for anything other than the minutiae that delights engineers. Not that there's anything wrong with that, of course. If they enjoy making those comparisons and all, then that's great, and more power to them.

My point is that these are musical instruments, first and foremost. And when people rag on the clones, they don't normally do so with scientific engineering terms. Some people do, of course, but the vast majority of the comments you hear are more like "The Gibson wins hands down!" or "I could tell the Fender every time!"

And my point is, even if you can tell the Fender (or Gibson or Moog or whatever) from the clone 100 times out of 100 in a blind test when they're playing the same exact things (and I would bow down in deference to anyone who can truly do that with a reputable clone), then what does that really say?

Like I said, in the real musical world, you're never going to have to do that. What you are going to have to do is tell the difference between the Gibson and the clone playing different things in different situations and contexts.

So ... if some expert can listen to 20 different tracks and tell whether it's Moog or clone every time, then I will shake their hand and buy them a six pack, because that's truly amazing.

But if they can't, then hasn't the clone adequately and indisputably done their job --- on the sound end --- anyway?

I understand that people have their reasons for owning a vintage Moog (or even the reissue), and that's cool. I have my own reasons for preferring to record to tape, and it's not because I "think it sounds better than digital" or anything like that. It's more about the process for me. In fact, it's the same reason that I like to use analog synths instead of soft synths. I'm sure I could be fooled in a listening test, but again, I prefer working with tangible knobs, and I like the idea of an analog piece of gear. And, of course, I understand that someone may just prefer the idea of a Moog over a Behringer.

But I'm not talking about that. I'm talking about the people that don't use clones simply because they say they don't sound the same/as good as the real thing.

And my whole point is that the idea of a listening test that pits clone against original with the exact same set of variables is not an effective means of determining this in real-world musical scenario.

---------- Update ----------

Great discussion here everyone, BTW. I've really enjoyed reading everyone's posts.
 
I bet I could tell the difference between the Nord mellotron samples and a real thing. The Nord would be making sound, and the Mellotron would be broken down again, with dirty tape heads, and a malfuctioning tape drive!
 
I am going to have to defer to Rule 1: If it sounds good, it is good.

My philosophy is to blind test everything- I don't feel that anything doesn't have some use, but I just use my ears.

FWIW I whole heartedly agree that no two instruments are going to sound alike anyway, nor mic's either so I just go straight to try it and see mode
 
I am going to have to defer to Rule 1: If it sounds good, it is good.

My philosophy is to blind test everything- I don't feel that anything doesn't have some use, but I just use my ears.

FWIW I whole heartedly agree that no two instruments are going to sound alike anyway, nor mic's either so I just go straight to try it and see mode

All of that ^^^^

In life I am a vigorously oppose the idea of the end justifying the means.

In recording I am the opposite . . . my primary interest is the final result, and I don't particularly care how I get it. It's a variant of Rule 1: if it makes it sound good, do it.
 
I am going to have to defer to Rule 1: If it sounds good, it is good.


. . . my primary interest is the final result, and I don't particularly care how I get it.


I've had a graphic hanging on my studio wall that is an image of Joe Meek, with the caption "If it sounds good, it is good"...for like 20 years now. :)
Yes...that's what counts.

That said...there is still a process for getting to the end. There are still decisions that have to be made, some analytical and some subjective.
So whether we consciously do it (and/or admit to doing it)...we are A/B-ing constantly in the studio, and we make choices based on that...again, sometimes analytically, and sometimes subjectively.

I've said it in the past and I still believe it...that if YOU sometimes just think "this" will make it better, then it will, even if your perspective is purely subjective...even if you have biases. It's part of the process...having/creating/believing in a certain vibe that may not have anything directly to do with the sound of something...yet it has an impact on it. Like listening to the same song on a gloomy rainy day VS on a sunny day...it will feel different.

You also have to question yourself and what you are doing, and we all do...even if it's done internally purely with thoughts.
It's rarely some haphazard, "toss it against the wall", and if it sounds good, it is good...kind of thing.
 
Whilst I agree that everyone is doing subjective comparisons and then you pick "what you like best" that, IMHO goes away from the title of the thread? "BLIND listening tests"? What then followed cannot remotely be considered 'blind' because peeps are doing the tests themselves and KNOW what the 'A' and 'B' are in most cases. In a proper double blind test a group of people are invited to "listen to some stuff please" and they don't know the source of the music, the equipment reproducing it, not even WTF the data collected will be used for! Are they testing amplifiers? Speakers? Audiophool cables???

The closest I ever got to a DB test was The Firm were concerned that a double triode brand then in production of guitar amps might become rare and would a substitute sound the same?
I built a rig that allowed an experienced player to switch valves instantly in the amplifier's signal path. One original and the other a different brand.

The test was blind to some degree because once the player had flipped the valves a few time he was totally bllxed and did not know original from sub and thus was JUST listening for differences. (I knew but was out of his sightline)

The result, from several sets of experienced ears (none mine!) was that yes, under hard drive, one brand was slightly different but clean nobody could tell. The difference was so small though that punters would never know which valve it was and in fact nobody ever did!

In a production process you HAVE to test against a standard from time to time. Component suppliers WILL switch bits without telling you and although one LED in a feedback circuit SHOULD sound the same as any other you have to test to make sure. You also have to do routine frequency response tests and gain checks because wrong values CAN get stuffed into a PCB or even missed out altogether.

Dave.
 
Whilst I agree that everyone is doing subjective comparisons and then you pick "what you like best" that, IMHO goes away from the title of the thread? "BLIND listening tests"? What then followed cannot remotely be considered 'blind' because peeps are doing the tests themselves and KNOW what the 'A' and 'B' are in most cases. In a proper double blind test a group of people are invited to "listen to some stuff please" and they don't know the source of the music, the equipment reproducing it, not even WTF the data collected will be used for! Are they testing amplifiers? Speakers? Audiophool cables???

Thanks for that story; it was an entertaining read. :)

I suppose I should have titled my thread differently or chosen my words more carefully in my original post. I didn't mean to imply that any of those YT pedal videos were legitimate blind tests.

I only meant to say that the idea of a blind test is, IMHO, kind of flawed, with regard to judging the musical tone and/or worth/effectiveness of an instrument. This is why I say that:

It seems to me that the idea behind a blind test is, by keeping all other variables the same except one, it should somehow be harder to tell two products apart.

To me, though, this should make it the easiest to tell the two apart, because you're essentially putting the two under a microscope, thereby making even the slightest variation that much more discernible because there's nothing left to cover it up.

And the problem with this, especially with regard to analog gear (purely digital sounds would probably be different I assume), is that these instruments are made up of individual components which themselves vary from unit to unit. In other words, resistors have variable tolerances, etc. Of course, they're supposed to be within a certain tolerance to do their job. But I'm sure most of us have had the experience of playing/listening to identical pieces of new gear which, when set identically, still don't sound quite identical.

So my point is, why should a clone be held to the standard of one particular original when it would often be a challenge to find two originals that sound indiscernible? And this is not even taking into account the idea of aged components in a 40 year-old unit. However, I think if you even compared several identical reissue Moogs, you'd likely find some slight inconsistency between them.


Yes? No?
 
I am going to have to defer to Rule 1: If it sounds good, it is good.

My philosophy is to blind test everything- I don't feel that anything doesn't have some use, but I just use my ears.

Your doing experiments without measuring ? Crazy man.

I am a scientist. With all my tools and ambitions . I need to build a giant robot that fires death rays to enslave the world. Then make a team to stop it. So I become the world-hero that saved everyone. It would be fake, but everything is. Dreams , some are small. Some are big.
 
Last edited:
.

And the problem with this, especially with regard to analog gear (purely digital sounds would probably be different I assume), is that these instruments are made up of individual components which themselves vary from unit to unit. In other words, resistors have variable tolerances, etc. Of course, they're supposed to be within a certain tolerance to do their job. But I'm sure most of us have had the experience of playing/listening to identical pieces of new gear which, when set identically, still don't sound quite identical.


Yes? No?

No .

They all sound the same. New in the box items at GC all sound exactly the same. Unless they dropped it on the way to you. The sound comes from the circuit . Not the individual component. The product is greater than the sum, of its parts.
 
I built a rig that allowed an experienced player to switch valves instantly in the amplifier's signal path. One original and the other a different brand.


Dave.

That sounds cool Dave. In the 90's they had a product called 'Yellow Jackets' . You could switch the power valves to small ECC83's or even smaller Mosfets. Changing a 100watt Marshall into a 1 watt Bedroom midnight screamer. No ridiculous power brakes, or even a need to re bias. Ever use the Yellow Jacket system? It is awesome.

They might still sell them. They have a manual.
 

Attachments

  • yjnos_product_manual.pdf
    511.2 KB · Views: 6
No .

They all sound the same. New in the box items at GC all sound exactly the same. Unless they dropped it on the way to you. The sound comes from the circuit . Not the individual component. The product is greater than the sum, of its parts.

That last statement is not one a scientist would use. Components DO affect the sound of equipment.

Take the Cs and Rs in an RIAA feedback network. Just a 5% tolerance band will change the response by several dBs and in a side by side test a pre amp flat '20-20' within 0.5dB will sound different from one + or - 3dB . The same argument applies to any circuit that has frequency modifying componets in it such as filters and EQs. Then there is stereo! You MUST keep tolerances to very fine limits or images will shift all over the shop.

Dave.
 
That sounds cool Dave. In the 90's they had a product called 'Yellow Jackets' . You could switch the power valves to small ECC83's or even smaller Mosfets. Changing a 100 Marshall into a 1 watt Bedroom midnight screamer. No ridiculous power brakes, or even a need to re bias. Ever use the Yellow Jacket system? It is awesome.

They might still sell them. They have manual.

Sorry I might have misled you? The device was for 'lab' use only and to be used only by technical personnel. Bloody dangerous with exposed 300V DC!

Dave.
 
That last statement is not one a scientist would use. Components DO affect the sound of equipment.

in an RIAA feedback network.

RIAA has a network? She sounds really sensitive.

I don't know about that. The SH-1 seymour duncan pickup will always sound like the SH-1 even bought from 2 different stores. If it didn't why bother at all? You could grab any old boxed pickup and go with it.

Could we say they sound 96% the same?
 
Back
Top