There is no doubt it is a "Weight" issue. I came off A Stratoliner curb weight fueled and packed at aroung 860 lbs. Then throw my 195 lbs. on top of that and you cruised at 95 mph with virtually the calm of sitting in your easy chair.
That said, at 25 mph on the twisties, the big cruiser handled like a garbage truck.
In parking lots if a car backed out unexpectingly, you might be going down.
In and out of the garage, a Royal pain in the ***.
Gravel, u-turns, up hill, etc. all a nightmare.
I will take the "Wind Gust" pushing me around a bit for the Sportster's "Nimble Handling" anytime.
I have learned bigger is not better (not for me any how)...

 

Weight problem... yes. Just gain a couple hundred pounds and you'll be set.

I hate riding on the interstate for that same reason of feeling like I've been beaten up by the wind. Every time I turn my head to look at the lane next to me and I'm going 70+mph, I have to hold the bottom of my full face helmet for fear my head might be torn off!