You need proper near-field (sometimes called "closefield") studio monitors to do serious mixing. Before I got good monitor speakers, it took me a long time to figure out how to set EQ for the mixes so that things would sound good on random cassette players...and it was tons harder when I attempted to mix with headphones, even good ones.
The purpose of good mixing monitors is not for them to sound good or to make your music sound good. It's so you can hear what your music really sounds like, so you can make it sound good!
It's not that they're supposed to sound bad, it's that they're not supposed to sound good. They're not supposed to sound bad either. They're just there to reproduce accurately how the music sounds, especially at close range (because you normally sit a lot closer to studio monitors than listening-type speakers).
Most high-end "speaker systems" are set up for theoretical flat response in anechoic chambers and other details that impress the hi-fi buffs. And in the real world, most people who are listening (rather than mixing) diddle with their EQ settings to make the music sound the way they want to hear it. And they rarely sit a meter or so away from both speakers at once as we generally do when mixing.
Near-field monitors are made to reproduce music in your studio in such a way so that when you hear it sounding good, it will sound good on boom boxes, stereo systems, and truck radios too.
You almost have to use published reviews to narrow things down to 2 or 3 competing systems in your price range, because experienced reviewers have tried the monitors while mixing, whereas any Jo Schmoe with a computer can post "these things sound awesome"...oblivious to whether that sound translates to mixes. Again, the important thing is not do they sound cool, but do they mix well!
I suggest bringing CDs to the store that you're real familiar with, and at least one that "everyone" agrees is well-mixed. Here's my personal list:
Elton John - Goodbye Yellow Brick Road
Assuming that you're listening to monitors that have been well-reviewed for mixing, I'd listen for clarity and shimmer and good bass tones, and I wouldn't want to hear distortion, buzzing, or excess thumping. Make sure you hear good stereo imaging in more than just one tiny "sweet spot"...you don't want to have to keep your head still forever while mixing.
Crank it up and turn it down. Good speakers can take the former without flinching (make sure that nothing rattles or buzzes, even at VERY LOUD VOLUMES) and will sound good (although you won't hear everything) at lower volumes too.
When you hear good monitors, instruments may jump out at you without warning, you may hear subtle things you never heard before, or hair may suddenly grow in strange places on your body. But they won't necessarily make you want to dance, because they don't emphasize frequencies at either end of the spectrum (or the middle, for that matter) the way "listening speakers" tend to do.
Monitors for home studios come in two main types: active and passive. The active types have an integrated power amplifier that has been optimized to work properly with the speakers. Passive monitors (a.k.a. "speakers" :-) need an external power amp. If you happen to have a good one laying around, it's generally cheaper to buy passive monitors.
...are ordinary monitors with a fancy name and a jacked-up price to deceive
the unwary. Speakers can't tell the difference between analog and digital,
other than the fact that loud sounds played through good digital equipment
will be equivalent to playing the original instruments, and will therefore
announce the speakers' poor quality by having the cones rip right out...