Time to let him go if he only is attracted to you for your weight.
I'm sad for you that is just not right.