I say go for it! I weigh about 370 pounds and I'm loving it. Yeah, the concern about what other people think is definitely a thing, but if you learn to love yourself and put yourself first, you'll find that it doesn't last that long.
You have to go with your gut I also struggle with worrying about what others think, but have let myself get up around 250 lbs to enjoy food and fattening without reaching a size that would be very concerning to some of my family. I have dieted down (like right now) but usually just get fat again... it would probably be better to just stay fat... certainly more fun. Maybe try gaining a bit more and see how you like it. Usually others are not thinking about your weight nearly as much as we imagine. Plus they have already seen you chubby.