I've never heard about this in a 'headphones' context, but in general, you can fry amp channels by connecting too small of a load. As the load drops, the amp can push more and more power. That means the amp will draw more amperage because the voltage is constant. Your headphones are probably not designed to take that extra current either, so you could have problems on that end too. If the load drops too low and the amperage gets too high, you'll have problems. Connecting too HIGH of a load is never a problem for the gear, but it'll sound quieter.
IME you'll be ok if you don't push your headphones too loud. Your headphones will be louder, you wont have to turn up the volume too far. If you do, you could blow the little headphone amp pushing the jack. Effectively, instead of the volume knob going from 0-10, it now goes from 0-20, and it's up to YOU to make sure you dont turn it up past 10. You're sorta losing a little safety feature and introducing a potential problem because of the mismatch.
