A real simple test you can do for yourself is to record a click (or any solid/steady reference beat) at some BPM tempo of your choosing and record about 3-4 minutes worth.
Now, cut/split the click track after the first 2 measures (8 beats) and then MUTE the remaining portion of the click track.
Then set yourself up to record to a new track and play whatever instrument you play, using those 8 click track beats as your count-off, and then keep on playing WITHOUT the rest of the click track (you muted it) for the remaining 3-4 minutes.
Don't even look at the tracks for visual cues...just turn away and play/record using your...."internal clock".
Stop recording at the end of 3-4 minutes and compare where notes/strokes/beats of your playing are relative to the muted click track beats, and also un-mute the click and now play them back together and listen and be amazed at how your "internal clock" is off relative to the click track.
Do that for 10-15 more tracks, and watch how the drift errors multiply...and more important...how the drift errors vary as to where they drift relative to the other drifts.
It might not be too bad if you COULD drift the same way in every spot on every track for 10-15 tracks, as that might not get noticed, which is what often happens when a band plays together, because they are simultaneously giving each other cues, so if they are relatively tight, it wont be too noticeable when they drift together in spots, but playing/recording alone...you will have sloppy tracks that will never sound like a tight band played them.
A solid reference is the way to go...IMHO.
Even after I get the drums recorded...I don't stop using the click. It's there for every track, I simply turn it down and pan it off to the side so it's not the most prominent thing.
And like Armistice said...if you find a simple "click" sound to be distracting...use a drum loop or some sequenced MIDI drums.