I am watching the setup for Mixcraft, and they recommend using Core Audio. Have you done that or are you using the ASIO driver? Also, you will find lower latency with higher bit rates. its the nature of USB. Its a burst mode system, as opposed to a streaming mode like Firewire or Thunderbolt. That's why you buffer the data. When the computer polls the system, the interface will dump the data down.
If your buffer size is 256 and your sampling rate is 44,100 times per second (Hz means cycles per second) then your latency will be (256/44,100) seconds which is 0.0058 seconds or 5.8ms. Obviously if you double the sample rate, the latency drops by half. However this is one way. If you are monitoring, you will double that, plus you have overhead in your computer, so you might see a slightly higher latency for full round trip. Thats also why lots of interfaces have a "Zero latency" option, where the audio is rerouted directly to the output, to eliminate that round trip.
The thing that concerns me is that you are getting drift. That's not normal behavior. That suggests a possible bit rate issue. I might try downloading the Mixcraft demo and try that.