You have to go with your gut
I also struggle with worrying about what others think, but have let myself get up around 250 lbs to enjoy food and fattening without reaching a size that would be very concerning to some of my family. I have dieted down (like right now) but usually just get fat again... it would probably be better to just stay fat... certainly more fun. Maybe try gaining a bit more and see how you like it. Usually others are not thinking about your weight nearly as much as we imagine. Plus they have already seen you chubby.