A bit of drift can be good though.
I think it's more correct to say "A bit of drift is unintentional" ! Sometimes, drift sounds awful and slack. Or at least it has when I've done it.
If the drummer is going to slow down, I'm going to have to too. That's why it's important to at least get the drummer record to a click track. Even more so if they often wander through tempo. But once that instrument is down, everyone has a real point of reference. My internal clock syncs to it and then I'm pretty much playing on my own.
In a way, your point kind of contradicts itself. If the drummer has played to a click and has really nailed it, then if you play to that drummer, you are in essence playing to the click anyway.
Like you, if I'm putting down the initial bit with the drummer, I don't use a click. If I'm starting off on my own, now I do. It's good for any percussionists coming in later and it provides a time reference.
Time reference.
That's an important phrase because in all the debates about clicks and not using clicks, there is always the implication that a click takes away the 'feel' or groove or humanness. Which is actually a lot of raffifia. Aside from the fact that use of a click doesn't
ensure metronomic beat/time {people may still have problems hitting a consistent beat}, what a click does is provide an anchor point around which vocalists and instrumentalists can dance. If you know your onions, you can skip about the metronomic beat and skilfully weave in and out
without throwing off the actual timing and anyone else that's going to play in that song, even if they're sometimes thrown by what you're doing, they've got a reference they can stick with. And that's all it is. A reference.
It's like dancing to disco music. The thing that many disco songs from the late 70s/early 80s aimed for was 120 beats per minute. But if you were dancing to said track, a good dancer could weave in and out of that strict timing while never departing from the rhythm and groove of the song.
So it goes with a click.