Are you sure it's not the latency offset you're adjusting?
The offset's a little tricky - if you're recording a softsynth real time w/ 7ms latency, you might want the offset to be zero - because you're adjusting to what you hear when you play. But if you're recording a midi track and monitoring the keyboard, or recording audio and doing direct monitor, then you want the offset to match the actual latency.
I finally hooked a monitor up to my controller (yes it has sounds too) so that I could leave the latency offset alone.
BTW, the last time I upgraded drivers, I lost the ability to manually set the ASIO latency on my Delta 66. But I can still set the offset. For other protocols, I can still set the latency itself...
Daf