Yes it does come from the tape, and your description of tape is essentially correct also, but the two points are not mutually exclusive.
If tape didn't retain magnatism at all, it would be useless as a recording meduim. It is that very retention that allows it to work. When they say it doesn't retain it's magnetism very well, they're talking as compared to something like an iron magnet.
The magnetization of the heads isn't so much from the tape dust itself, but rather from the prolonged exposure of the heads to the magnetic fields on the tape itself.
Have you ever, when you were a kid, magnetized the tip of a paper clip with a magnet just by letting it stick to the magnet for a while (this is a classic way of making a homemade compass needle)? Just the prolonged contact to the magnet causes the clip to acquire some magnetic properties itself.
It's a similar principal with tape heads. Just the prolonged contact with the magnetic tape, over time, can cause a small build up on magnetism on the tape heads themselves (especially the playback head , which has no field of its own.)
As far as the noise, not only can the magnetism affect the performance of the playback heads, but they can serve to slowly erase the tapes themselves, like lightweight, lower magentism versions of an erase head. This can actually increase the noise level on the tape itself.
This is why there are indeed
tape head demagnetizers of all different sizes and shapes (and levels of quality too) ffor everything from casette decks to 2" open reels.
G.