Exactly,
For me, I am a self taught drummer. I am primarily a guitarist, which lead to bass, then keys, and eventually drums. I hate clicks just like the next guy, but what I used to do at the very beginning, was program something into my keyboard, including any drum ideas (so I could learn how to play them). After that I would pop on the headphones or run it through the PA (Pair of Cerwin Vega 15" 3-ways at the time) and I'd just try to keep up.
Like any other skill, you have to put in the time and effort to get used to it. I wouldn't expect any drummer to be comfortable with a click if they never did it before.