Thanks for everyone's contributions, but it isn't any of the suggested causes.
I think I have found the answer, but it is very specific to my setup, so read on if your audio interface is a focusrite saffire pro 24. It's a great interface, but it comes with some demonic software called saffire...