On Judging Public Mix Contests

[mshilarious]

As some may have read, the forthcoming PMC #11 will be judged by yours truly Never fear, the tribunal of judges will also include xfinsterx and farview.

However, in light of the fact that I've never even entered a PMC before , I figured it would behoove me to undertake a study of my responsibility. And so, having read most of the PMC threads, I wish to share with the Clinic my observations with the goal of documenting the judging techniques that have been developed and encouraging discussion of their merits.

Stay tuned for a series of following posts:

 

[BentRabbit]

They probably weren't covered in the threads, but most of the contestants were aware that all of the winning entries were stolen from me and renamed...
Since it was all in fun, I never really made a big deal about it, but since you're going to be judging, I thought you should know...

Oh... and 'Stairway to Heaven' was mine too... I whipped it up one day in 'Creative Writing' and the teacher said it never made it to her desk... Imagine my surprise when i first heard it on the radio...

 

[corban]

Oh... and 'Stairway to Heaven' was mine too... I whipped it up one day in 'Creative Writing' and the teacher said it never made it to her desk... Imagine my surprise when i first heard it on the radio...

That was my bad, I swiped it from her desk. The bridge was in some awful need of a rework, so I fixed it up some, added some spice, and voila!

 

[mshilarious]

History of PMC, Part I

The first PMC was judged by poll; professional judging began with #2 and continued through #8. I am too stupid to find #9, and #10 was not a contest. Therefore, I restricted my survey to #2-8.

BlueBear was not only the original judge, but the creator of the judging system that continued largely unchanged through the PMCs. In #2, BlueBear used 6 categories and a 5 point rating system:

LOW END -- how well the low-end was balanced and sat in the mix.
AIR/TOP END -- how well the high-end was balanced and sat in the mix.
DEPTH/SPACE/AMBIENCE -- how well the tracks sat together... was each individual track clearly sitting in its own space relative to the other tracks?
MIX BALANCE/LEVELS -- how the tracks were blended... were mix elements blurred, jumping out uncontrollably, or just right.
IMAGING -- how well the soundstage of the mix was laid out between the speakers.
OVERALL MIX -- this was a bit of a catch-all -- very subjective - did something in the mix speak to me or not.

In PMC #3, the Bear changed to a 10 point scale, added a seventh category, and set an important principle regarding evaluation of material:

I expanded the scoring range from 1-5 to 1-10 instead...... and I added another category - OVERALL MIX IMPACT - mainly because there were certain songs that sounded good sonically, but lacked punch, and vice-versa.....

One more point, and actually, I'm guessing this may generate some groans. You all know from my critiques in the MP3 Clinic that I never judge on the subjective creative merits of a song, only on the sound quality itself. So in this case, you'll notice that other than the catch-all OVERALL SOUND QUALITY property, I did not specifically take into account whether more tracks were added or changed compared to the original tracks. All mixes were eval'd strictly on the sound qualities listed above, so any "enhancements/changes" would still fall under those characteristics.

As you will see in a minute, the categories and rating scales have served quite well. There have been occasional accusations of bias, and while I did not study scores from an individual point of view (nor did I listen to any of the mixes), statistically I find that the system has been used in a proper and effective manner.

Next up: pretty (or ugly, depending on your taste) graphs!

 

[mshilarious]

History, part II

I hope that last post was helpful for new entrants who don't know, and I didn't, the descriptions of the categories.

Anyway, Blue Bear also added descriptions to the categories as follows:

1 - Yikes!
2 - 4 Poor
5 Fair
6 - 9 Good
10 Massenburg

Here's one minor quibble: the descriptions don't really match a normal distribution (and the distribution of scores is very good--see below). Thus, I would suggest:

1 - Yikes
2 - 3 Poor
4 - 6 Fair
7 - 8 Good
9 Excellent
10 Massenburg

Very, very, very few category 10s have ever been awarded--no easy grading here! In fact, until PMC #8, nobody had ever scored an overall average 9 (more on that later). Thus, I believe it's appropriate to upgrade the description of rating 9, and also broaden the "fair" description to match the distribution of scores:

 

[mshilarious]

History, part III

One more chart: while the participation in PMC has dropped steadily, hopefully we can reverse that as prizes will be offered for PMC #11!

However, the average score has sharply increased in the last few--are poorer mixers dropping out?

 

[mshilarious]

The Matrix

Finally, my analysis of the scoring grid. The seven categories have proven very durable. One thing I did to enhance my own understanding of the categories was to reorder them in order to show the relation between technical and artistic categories, quantitative vs. qualitative, science vs. art, etc. The major change here was to move the mix balance category from the middle of Blue Bear's chart to the first position.

Whereas Blue Bear has described the categories well, I have two questions of the participants:

1) Does this order roughly represent workflow? That is, throw up the faders, get the balance right, compress, EQ, pan, effects . . . Obviously these categories aren't discrete functions, and mix elements will overlap categories--

2) Do the categories impact your workflow? Does anyone mix for these seven items, or perhaps keep in mind a previous low score in a subsequent contest? Does this stuff mess with your head

Well, here's my matrix. Bottom line is I am going to aspire to judge the same way as previous judges; I hope this thread will be a useful bit of historical record when we're on PMC #76 and somebody asks where all this came from anyway?

 

[NL5]

However, the average score has sharply increased in the last few--are poorer mixers dropping out?

Actually, I think the mix contests are doing what they were designed to do - make better mixers out of us all. I know it helped me immeasurably (sp?).

That is one of the main reasons scores are going up.

 

[xfinsterx]

I personally dont think about the stats when i mix.
I just mix till its done.

It dosent f%^& w/my head.

Cool spreads Mshilarious.

Very cool!

 

[ds21]

I personally dont think about the stats when i mix.
I just mix till its done.

It dosent f%^& w/my head.

Cool spreads Mshilarious.

Very cool!

yeah, I could care less about the stats or scores or what anybody thinks on my mixes in these contest, well that's the way I think when I'm mixing them, but once I'm done I like feedback and see what areas I need work on. Work flow has to do with my habbits/methods or what catches my ear in the rough "throw up the faders" e-val.
can't wait to hear what the song is

 

[Blue Bear Sound]

That's very interesting analysis, mshil - excellent!

 

[ocnor]

I think when you get to the qualitative side of the chart it becomes much more of a gray area. Artistic merit is highly subjective because it is determined by the judges specific musical taste. Judges "A" and "B" might like bone dry vocals while judge "C" likes reverb and delay. So if you mix the way judge "C" likes it then you will have a lower average score. One remedy for this would be for the judges to agree on and give specific instructions as to how the mix should sound. This would be much like the artist saying that they want it to be reminicent of a certain band.

 

[mshilarious]

I think when you get to the qualitative side of the chart it becomes much more of a gray area. Artistic merit is highly subjective because it is determined by the judges specific musical taste. Judges "A" and "B" might like bone dry vocals while judge "C" likes reverb and delay. So if you mix the way judge "C" likes it then you will have a lower average score. One remedy for this would be for the judges to agree on and give specific instructions as to how the mix should sound. This would be much like the artist saying that they want it to be reminicent of a certain band.

I think the control for that has been that the factors are equally weighted. In fact, the first four factors you could almost score straight with a quantitative analysis, although I don't believe that is appropriate. Nevertheless, if a mix comes in bassy, nothing above 8 kHz, -6dB RMS with dozens of overs, and no width, I don't think it will matter how nice the reverb on the vocal sounds.

For experienced mixers who don't have those problems, yes I imagine the distinguishing characteristics will be, and should be, purely subjective.

If the artist in question, in this case xfinsterx as they are his tracks, wants a certain style, then I would score that under any and all appropriate criteria, not just the qualitative ones.

I might publish quant analysis on the submissions, but only after results are final and maybe anonymously--that it not matching stats with entries--purely because I am interested on a statistical level in the correlation between quant analysis and qualitative judging. I'll hypothesize that there is a strong correlation between quant and whether you are in the top or bottom half, but little correlation between quant and position in the top half.

Any interesting results I'll publish here and not in the PMC #11 thread.

 

[Llarion]

I thought this thread merited a bump so those rookies to PMC (like me, for instance) can read the analyses presented herein...

 

[Llarion]

*Pwang*!!!

Up we go one more time... Great info in here, PMC 12 people...

 

[mshilarious]

Darn it, I lost the spreadsheet with these charts. If I get time today, I'll try to recreate them with PMC #11 added.

 

[mshilarious]

Revised chart #1:

 

[mshilarious]

and #2:

 

[Blue Bear Sound]

WHAT???? Still no "Massenburgs"?????????????????????

 

[mshilarious]

WHAT???? Still no "Massenburgs"?????????????????????

Well, you could enter

It would be pretty funny if Massenburg entered incognito, and we gave him like a 9.4--"You just weren't quite as good as yourself today". But I ain't gonna hold my breath.