I would carefully use a low power vac with a fabric attachment (the one with the long loose brissles sticking down).
Another option is to hold the vac a few inches away, and drive the speaker with some strong bass. That will sometimes make the dust fly off, and the vac will be there to catch it.
Ed