First if we could get a solid definition of spectral balance. I understand you want a nice curve, and no rigid peaks.
Sorry, this is incorrect. There are no rules or guidelines or any other such thing as to how a spectral graph should look. There will be peaks, sometimes more, sometimes less, and the idea of a "nice curve" only works if you're talking about pink noise, which most of us don't like to listen to.
Spectral analysis cannot be used to determine if something is right or correct. It can only be used by trained eyes to help diagnose some very specific causes of problems one already hears.
You want to add distinction and clarity to individual components b finding a space in the spectrum for each instrument to dominate.
This is kind of closer to the truth, but still is somewhat oversimplified.
First there are many ways to add distinction and clarity to individual components that have little to do with which part of the spectrum they dominate. Sometimes, in fact, one needs to cut bad stuff away from the section of the spectrum that an instrument dominates in order to increase clarity and definition.
Second, there may often be more than one instrument that are dominant in a specific part of the spectrum, but because of the song's arrangement or the instruments' locations in the stereo soundscape (or both) they cooperate with each other instead of fight against each other for their part in the mix.
That said, though, it is generally correct that you want to use the frequency spectrum available to you in a fairly democratic way by letting the instruments stake out their own sections and their own roles in the mix.
A huge chunk of this should be handled in the music composition and arrangement, and the tracking and mix should be designed support this arrangement. Where there is no arrangement - e.g. a newb garage band where every single player wants to play full-bore lead through the entire song - the mix engineer has to have the discipline to know when less is more and when to use gain or mute automation to arrange the song on the fly to make spectral sense.
Are there rules of thumb when it comes to where specific components typically sit in the spectrum? I know every mix is different. I just want to know about the process, and where you start etc.
Like already said, you start with the music composition and arrangement. Just like Beethoven with an orchestra, the instrument lines should be written/performed to give each instrument it's role spectrally as well as emotionally. Then you sound design the tracking to support that. For example, you might tune the guitar amps and select the guitars and guitar pickups to support the idea of a high guitar vs. a low guitar, or select bright and dark microphones/preamps for different instruments to support the roles in a similar way.
There are no rules as to which instruments to use for which roles. There are obvious trends, but there are so many exceptions to these trends that they cannot be considered rules. The obvious ones would generally be to have bass and kick drom dominate the low frequencies and cymbals and shakers and such up on the high end, but everything else - guitars, keyboards, horns, vocals, etc. - tend to all fall somewhere in the middle and can be used for just about any spectral role you desire (of course a barrie sax will be lower than an alto sax and so forth).
Follow the arrangement. If the arrangement sucks or is non-existent, then use your ears to figure what make sense. If git 1 sounds like it has a little lower of a timbre than git 2, then you might EQ to reinforce that difference. If the left hand on the piano sounds really cool, you might make room for that more on the low end, but if that's boring or muddy, but you have a great big tom, then you might let the left hand of the piano fade just a bit and emphasize the noodling on the right hand instead. And so forth. This is what is often called "letting the mix tell you what it wants".
G.
