I was struck by something years ago about women actually "passing" the sacrament. I have a wife and four daughters, I am gone many Sundays, my oldest son has been in college for the past two years and my second boy often is administering the sacrament, so on our pew each Sunday the sacrament is passed by women more than by men - and often exclusively.
There are two aspects that get melded together because of our imprecise terminology. It requires the Priesthood to "administer" the ordinance of the sacrament; it does not require the Priesthood to "pass" the sacrament. We use "pass" to describe what the deacons do, but, in reality, they are assisting the Priests in the "administration" of the sacrament. The Priesthood "hallows" or "sanctifies" and oversees the ordinance (grounds the practice in divine authority), but all are allowed to participate in the process of receiving AND giving - even young children, who are the only one's inherently "worthy" to represent the sinless one.
The key, in my opinion, is not who "passes" the sacrament to each and every member, but rather who "administers" the ordinance - who "oversees" the distribution to the body of believers. That administration can take any number of forms, including our current method of involving the women, girls and children in passing the sacrament.