Pretty close to what?
10ms is about the time required for your ear to hear two distinct sounds. Can a 4ms offset have a major impact on your recordings if you're layering a lot of tracks? I wouldn't say major, no, but some impact quite possibly....and of course it's always best to be in control of the things that can have an impact on your sound.
Another thing to consider is outboard effects. Think about delay's and reverbs, to which you're basically adding a 4ms predelay, which is quite a lot when you're fine tuning! It's something you can always adjust for, but you need to be aware of it.
Then, if you ever have to crank your buffers up for any reason and you forget to reset them, you can definately run into problems. I used to switch between ASIO and WDM apps and I like the ASIO performance I get from a 2048 sample buffer. Well, when I'd switch back to WDM and forget to reset the delta buffer, suddenly I was contending with a 50ms offset!! Ouch.
Using ASIO, your track offset will always be about a millisecond. Yeah that's only a 3 millisecond difference in your case, but its four times better.
So anyways, for most of us this problem hasn't been a show stopper, but it is a problem, and it's somebody's fault! The trick is finding out just who's fault it is and getting them to fix it.
Slackmaster 2000