It is our responsibility to explain the promises we make when we partake of the sacrament to those who are visiting. It is their decision whether or not to partake, based on their willingness to make those commitments.
We sometimes forget that, while the sacrament is a renewal of our baptismal covenants, it also is accessible to those who have not been baptized (like our children) - and, therefore, for them it is not a renewal. For our children, it functions more as a preparatory promise. The sacrament is not limited to our unbaptized, as if it was something only for the "righteous children's club". It is a sign of commitment, for us, to baptismal covenants already made, and for others, non-baptismal promises already made or willing to be made. If I allow my 5-year-old to partake and not my devout, adult, Christian friends, I think that says more about me than about them.
Should non-members partake? Let me ask a different question:
If they are willing to make the promises inherent in the sacrament prayer, and if they are living their best to the extent of their knowledge, why should they not partake?