Wear a good mask, gloves and long sleeves when you are working with the material, and use as tight of a fabric covering as you can find that still "breathes" (literally, test breathing through it). Anything inorganic that lodges in your lungs is going to be unhealthy in the long run, but you are not working with it on a daily basis, and unless you are jiggling and moving the panels around all the time, it's not going to be degrading by itself and spewing out into the room, at least IMO. (Your body's respiratory system is designed to move that kind stuff out of your nasal passages and trachea *before* it gets to the lungs, and is pretty efficient at doing that, given the chance, so keep those parts healthy!)
My little music room is the most dust-free area in the house, and it's got 24 of the batts stuffed into panels, covered (but not tight against the wall) on both sides. It's not going to be what kills me, if it's bothering me at all.
It might be somewhere here that it was noted that you can even cover it in very thin plastic, maybe the thinnest dropcloth stuff you can buy, if you want, without seriously degrading its [sonic] absorption, but I don't know the physics of that. And then, of course, you'd have the microplastics to worry about, because that stuff *will* break down...