I feel you on this 100%.
All my life, I've been put on diets, ignored by the opposite sex, and hated every minute of it. But I have recently come to find out that being heavy is my comfort zone. And even though I don't like it, losing weight cares the bejeesus out of me because I'm not going to know what to do. At least in my current state, I know what to expect.
I'm not sure if I'll ever get over the idea of not being this size. I just know that also under the threat of diabetes drugs and insulin (which I am super opposed to taking and don't currently need) I am going