Well, a few things.
First and foremost, the mixer meters are calibrated in dB(VU) while Audition (and pretty well any DAW) is calibrated in dB(FS). What this means is that 0dB on the mixer should be equivalent to roughly -18dB on your DAW so you're not that far off.
Second, the driver download for your mixer says it's for Steinberg so that implies it's probably ASIO. If so, the Windows Audio settings should be completely bypassed so if they're adjusting anything, something is set wrong. Assuming you're on Audition 5.5 or later, go to Edit/Preferences/Audio Hardware and make sure the device class is set to ASIO and that the driver is set for the Yamaha one.
Some drivers have a control you can call up in Windows to set sample rate, level, etc. I don't know if the Yamaha driver has this but it's worth checking
Third, there's an attenuator for the USB output from your computer into the mixer. This doesn't sound like your problem but have a read of page 28 of the manual anyway.
Finally, in Audition are you monitoring the input levels or the playback. If playback, make sure the mixer isn't turned down for some reason.