Woah, that's weird. Maybe when direct monitoring was on, software monitoring was on as well? There really shouldn't be any noticeable latency with direct monitoring.
I think, from what you've described, that you'll need to bring up the ASIO control panel for the Focusrite and lower the buffer...