There are many different kinds of fitness. Fitness does not simply mean muscles, it may mean just fitter than a total potato. As erotic as pure sloth is as a fantasy, it is dangerous and boring as a reality. Being muscly may be a certain turn-off, but being fit enough to survive an orgasm? Only in the most extreme fantasies is that not a good thing (though in all honesty they can be perversely and disturbingly hot). The truth is, as others here have mentioned, some of the sexiest things about being unfit is having to move around and feel the weight, or to watch your lover become excited as you try to move around and do even normal things. Also, swimming is a seriously erotic form of fat exercise: The body moves, its wet and soft, waves of flesh in waves of water, floating as it belies its weight, moving as it belies its encumberance, and its decent light cardio that doesn't build muscle. Also, having to have help to get out of the pool can be very sexy and embarrassing.
I am a fan of fantasies like those of
www.deviantart.com/piritaargenta, or my own for that matter, or even ones like Wicked Stepfather, truly one of the darkest stories out there, but fantasy of any type tends to be limited by words and length, one-dimensional if you will, and so misses the depth of the eroticism of reality. To be fit enough to enjoy your life will not decrease your pleasure at your general unfitness, and the value gained thus will far outweigh any erotic fantasy.