I have no illusions whatsoever about the garment being "magical" in some way. I see the "protection" promise as spiritual, not physical. They are powerfully symbolic to me, however, and I absolutely love the concepts of:
1) Being reminded constantly of the covenants I have made - the basic ones of the temple that I think are important and powerful. When I am wearing it, I hardly think about it. It is subconscious in practical terms. However, when I put it on and take it off, I am reminded tangibly of what I am striving to become. That's important to me, so I wear it as a token of my own personal commitment to God - not as some kind of requirement.
2) Being clothed in the Priesthood power of God - and that concept applies to all men AND women who have been endowed. Women who are endowed in the temple carry with them actual symbols of godly power and covenants, and I personally think that is important in the Church - that women literally have the symbols of the Priesthood in a tangible form even though they can't exercise that in the administration of ordinances. To me, there is great power in that symbolism - perhaps even more so for women than for men in our current culture.
3) Progressing toward godliness and entering into the presence of God. That is the symbolic center of the temple for me - that I have the right to approach God and be in God's presence - that I ALWAYS am clothed properly to enter the Bridegroom's feast and commune with them. Even if I didn't find other deep meaning in the temple, I would continue to attend simply to participate in that specific symbolism.
I'm not saying everyone should see these things as I do - not at all. However, I caution people to not throw out the available symbols and their potential power when those things can be used for tremendous enlightenment if only adapted slightly for each individual. I wouldn't think seriously about not wearing the garment right now, because I have a way for it to help me personally. Even if someone else thinks "it's just underwear", to me it isn't "underwear" at all. It is a sacred symbol, and that is important to me.
BTW, as a follow-up to the last sentences, I personally am not opposed at all to the idea of wearing the garment OVER regular underwear - even though I know that is not the Church's default standard. I think that would remove much of the "weirdness" for many people - as it then would be seen as "clothing" without the automatic, negative association with "underwear".