I think I know what you're looking for on the drums, it's the snare. It's too hot, and later it's dynamic when the rest of the kit is relatively static. The overall effect is very jarring on an otherwise mellow track.
I hesitate to suggest compressing it more because I don't think that is the answer. You seem to be going for a mellow jazzy feel on the kit, with a syncopated snare accent, and I think you are most of the way there. Maybe cut the level in the beginning, and bump it back up closer to the end. Actually listening again it's more involved than that. Hmm, maybe a compander.
Consider shifting the pitch up a notch or so.
OK it's 12:30 here which means Conan O'Brien is coming on, which means that Max Weinberg is about to play the theme music. Listen to his snare on that, I think that's what you need.
Otherwise I thought the tune sounded great.