Yes, to some extant you will lose some quality with anything patched in between the gear and speakers, it may or may not be noticable, there's only one way to find out!
There are other solutions - monitor controllers. These usually provide a switch to pick what source to monitor, a switch to pick a choice of speakers, a volume control and a headphone amp. Cheap ones can be had from Samson and Behringer, but these won't be better than a small mixer at the job, just more convenient and take up less space. More expensive ones like the Mackie Big Knob do the same thing with better quality. There are also some really expensive ones from top end makes like Cranesong that use passive resistor networks to control the volume precisely and repeatably.
The really simple and cheap solution if all you want is to pick one source at time to feed the speakers is a "Hi-fi" componant selector switch box.
The main problem a lot of people find with active monitors is they can be too loud without turning the source volume down a lot. Ideally, the sources output fader should always be left at neutral 0db - neither cutting nor boosting - and the speaker volume should then be adjusted to give a reasonable choice of working levels for the engineer. Stereo volume controls (faders or rotary) are very poor at matching the left and right channels especially at lower settings and this is why the more professional monitor controls use switched resistors to attenuate the signal.
My own solution is a home build switch box providing a choice of monitor levels, so if you're handy with a soldering iron I can give you some links to some design ideas.
