Can a song be so badly recorded and mixed that it can't be mastered ?

Perhaps if you are serious about getting a great master spending a bit and take one songs multi tracks along with your mix to a pro studio and see if you can watch them do a version (for pay of course). Still, IMHO one the best things you can do is listen to your mix alongside a mixed and mastered track in the same or similar genre.
Otherwise, start with one song and see if you are happy with the result.
Is there a guide to not messing up your track so Massive Master would not reject it?

Need the info, sorry to bump old thread
In general, it's difficult these days to mess up a mix so bad that it can't be mastered with some level of improvement. Still, you don't have to mess up in a massive way to have issues that can't or shouldn't be resolved during the mastering stage. Keep your levels balanced, leave a bit of headroom, export your mix in 32 bit floating sync. Take extra time to clean up frequencies that are clashing. If you hear something that doesn't sound quite right in your mix, address it before mastering. Your final mix should sound somewhat competitive before it's mastered. If the mastering engineer has to make drastic adjustments, then the mix wasn't ready. The adjustments in mastering should be very conservative. Mastering should be the polish that glues it all together and gives it a competitive level. When you export in 32 bit sync, even if your mix is clipping a lot, it can be normalized to preserve the dynamics and provide room for mastering, so long as the dynamics were there to begin with. No need to do it that way really, but just as an example that it's pretty difficult have a mix that can't be mastered with some degree of success. If your levels are all off and you have overly processed everything, mastering could just make it even worse. Of course, it would be pretty obvious in the mix in that example.