I'd say that the best thing to do in that case is to re-record the drum tracks (if you have the option) Why? Because drumtracks are a whole on their own, they carry the energy, the pulse and the feeling of the song. If your drummer loses the tempo in some fills, you'd have to re-record them and just leave them as they are. Most early bands like Led Zeppelin had a sloppy style, and they left all the mistakes in the recordings and it sounded very human and it added to the rock'n'roll feeling.
If you try to cut, stretch or edit a drum fill, it will kill most of the feeling, the groove of the drum track. Because you'll be listening to a drummer playing all nicely when suddenly you hear this fill which has been played on by a bunch of codes.
But unfortunately I don't know how to make slow fills faster or fast fills slower. It probably has something to do with time stretch.
Sorry, not much help I guess..
