I think we all want what we don't have (myself included). The Street Glide guys add tourpaks and lowers and the Ultra guys get the quick detach stuff and want to take stuff off.

I have the quick release for my tourpak, I took it off once or twice, and really missed its function, and didn't really notice much difference in handling or weight. I removed my bag rails, put a bullet signal bar on and added filler panels in the back so I could have that Street Glide / Limited look. After my bike went down in the garage (gently, while lowering my jack), the rails are now back on and I had to remove my fillers so they could fit.

If you are wanting a new bike, and have the cash to spare, go for it. But you can have the best of both worlds by adding the detachable hardware, and switching back and forth.