Sometimes we assume it is because they have been offended or lazy or sinful. Actually, it is not that simple. In fact, there is not just one reason that applies to the variety of situations. Some of our dear members struggle for years with the question whether they should separate themselves from the Church. In this Church that honors personal agency so strongly, that was restored by a young man who asked questions and sought answers, we respect those who honestly search for truth. It may break our hearts when their journey takes them away from the Church we love and the truth we have found, but we honor their right to worship Almighty God according to the dictates of their own conscience, just as we claim that privilege for ourselves. (Dieter F. Uchtdorf, October 2013 General Conference, "Come, Join with Us")
This is one of my favorite quotes of all time, from any apostle. If we say we grant all people the privilege of worshiping God according to the dictates of their own consciences, we need to accept it for "our own", as well as for "others" - and that also applies to those who stay in the LDS Church but have different views than ours.