Hi...know how you feel..really
Until a couple of years ago I was slim and muscular,working out at gyms etc My conversion to the fat life (I had always had a fat fetish and like my food
)was fraught with your sorts of doubts..it took several tentative starts into gaining before I accepted it fully.It's not easy..all the people commenting in a politely (or even not politely!)way about your weight gain and the automatic assumption that you want to lose it;society's general prejudice (work etc)against fat as a sign of weakness and uselessness.Can you live with that?If you think not and that it will make you unhappy,then don't do it.
As for girls,as I know,there is no denying that it will limit those who are sexually attracted to you.However,my experience is that most women are a lot less superficial than men and though they might not be looking for a big man,they will not rule out a man just on that..personality,intelligence,sense of humour etc meaning alot more to most women in my experience.
Well,might not be much use,but those are my thoughts