By tuning your guitar down, you are puting less tension on the neck, so you will have less bow (or even a back bow) if you leave the truss rod the same as with standard tuning. If you are moving back and forth between the two, you will have to either compromise on the setup, or you will have to have it setup for one, and it will be off for the other. I would chose having it setup for the lower tuning. That way the action will be a litle high in standard tuning, but it wont buzz either way. If you set it up for standard, it is likely to buzz in the lower tuning.
As for physical damage to your guitar, I wouldn't worry about it. If you were tuning up a step, there might (MIGHT) be some problems, but I doubt it.
Light
"Cowards can never be moral."
M.K. Gandhi