Not familiar with Macs but in XP there is an advanced volume control accessible through the Control Panel...
Start > Control Panel > Sounds and Audio Devices > Volume > Advanced
At this point select...
options > properties >adjust volume for recording
A small record control window opens up allowing you to select which input to use for recording as well as to adjust the input level.
I'm sure that your Mac OS has a similar setup scheme. Good luck.
Also, make sure you're running the turntable through a phono preamp stage which will provide the necessary RIAA EQ curve. Without it, your vinyl will sound like s#it.