Not dust, but moisture. While traveling, our speakers went from hot to cold/then cold to hot many, many, times a week. Didn't hurt them a bit, they are designed for such scenarios. In a perfect world, it's a good idea to let them warm up slowly.
Don't fret about a little dust.