Personally, I would say:
"Thank you for asking me. I really appreciate it. I will need to talk with my wife and pray about it. I'll give you an answer by _______ (insert your acceptable time table)."
Then I would do exactly that - talk with my wife and pray about it - with a full understanding that I might or might not "get an answer" but I've done all I can to ask.
If I feel like I get an answer, great. Problem solved.
If I don't feel I've gotten an answer, I would make whatever decision with which I am comfortable.
If that decision is, "No" - then so be it. I would respond with:
"I have talked about it with my wife and prayed about it, and I just don't feel like I can accept it right now. There's no "worthiness" issue involved, but I just can't accept it."
If pressed, I simply would repeat that same basic message. I've thought about it and prayed about and just can't accept it right now.
If the decision is, "OK, as long as I can do it my own way in harmony with my own conscience" - then so be it. I would say what I said to my own Bishop once (with regard to being called as the Primary Chorister) - which was, essentially:
"I don't think I can magnify the calling in the way that you want it done, but I am willing to do it as long as you understand that I will do the best I personally can do."
In essence, I was saying:
"If you still feel inspired to call me, you are getting ME - and I will do the absolute best I can, but I still will be ME."
That basically was my response, but I got called into the High Council at the same time, so it was irrelevant in the end. (*grin*)
I believe in accepting callings as a basic rule, but I also believe in making sure those who are considering calling me understand my situation fully before they formally and officially extend the calling. I also believe in and have done that whenever I have been the person approaching someone about the possibility of extending a calling.