Gram of fat has over double the amount of calories than sugar/carb (9cal vs 4cal). And diving into the sugar will, no joke, lead to diabetes pretty fast. If you're going to splurge hard, stick more with your fats: butter, cheeses, cream, oils, peanut butter, nuts, etc.
6 years