Not a hell of a lot to add that hasn't already been said, but I'm bored and on my lunch break, so...
Greg's right, the appearance does matter when you're playing rock music. You know how you go see, ohh, Kiss or someone, and they have a wall of Marshalls behind them? Most of those are just for show, they'll be plugged into one but they bring a ton of dummy 4x12s (most not even loaded with speakerS). Why? Because it's part of the show. So yeah, there are visual reasons to gig with a real amp over and above the viability of a direct signal for guitar tone.
That said, even if you don't care about visuals, you have two remaining problems, one a matter of taste/not a huge deal and the other a VERY big deal.
First, my experience with modelers is that while they're adequate, they're not THAT good. The feel is off, the sound is usually a bit grainy in the presence and a little smeared in the midrange (line6, anyhow), and while they certainly sound like a guitar, it's rare that you hear a modeled tone that absolutely blows you away. As an added risk, most models are usually over-gained relative to the real thing, and with all of those on-board FX, the danger is that you'll be tempted to dial up a live sound with too much gain and a ton of chorus, delay, flanger, reverb, or whatever, both of which can make it awfully hard for a guitar to "cut" live (in fact, I think my biggest problem with modeling is considering all the programable options, it makes user error VERY tempting, and people tend to dial up tones they'd never bother to go after with an amp).
Second, the bigger problem is you're SO much at the mercy of the sound system. I don't know what kind of places you usually play, but I've played my share of gigs in places where the house PA was only up to the task of reproducing vocals and maybe a bot of acoustic guitar. If you roll into a club expecting to go direct and either the PA is underpowered, or they simply don't have enough channels for you, you're going to have problems. Second, going direct means you have to rely on monitors for your guitar sound, if you want to hear yourself play.Not only can you not take it for granted that every club will HAVE monitors, much less decent ones, the other thing you have to worry about is getting enough signal of you that you're comfortable playing, without having the guitar so hot that your singer feels like you're overpowering him.
I just think that while your rig is going to be heavier if you keep bringing an amp, it's more flexible, more practical, and safer, even aside from the fact it looks cool.