The garment (especially for women) is one of the things that I really love about the temple and Mormonism, even as I understand that many women struggle to wear such visually non-pleasing "underwear".
First, it is the one thing that says starkly to me that women actually do "have the Priesthood" in an important and powerful way - even if they aren't "authorized" in our current time and culture to "exercise keys" and perform ordinances outside the temple. Seriously, the visual representation of men and women being "clothed in the garment of the Holy Priesthood" and carrying Priesthood symbols as they leave the temple and enter the world is wonderful imagery to me.
In other words, if I can say it in this manner, when you get past the outward appearance and how the world sees us and get to what is "below the surface" and how God sees us, all men and women who leave the temple carry the exact same Priesthood symbols with them - just as they both can perform Priesthood ordinances while they are in the temple. Wearing the garment, to me, is more about taking the temple out into the world (being protected from evil in the world as if you still were in the temple) than it is about anything else - which is the main reason I personally don't struggle at all with the concept of wearing the garment.
For what it's worth, I also would have absolutely no problem whatsoever with someone wearing regular underwear beneath the garment, particularly in order to keep the symbolism of the garment more sacred in their own eyes and not let it come to be seen as nothing but weird underwear.