Rocks and sand will quickly destroy your belt and rear pulley. With a 105,000 kms. on my Sportie, I've replaced my belt 3 times. The road maintenance crews use sand to provide traction on icy roads. I do ride year round and have to replace my belt and pulley yearly.
I took mine off about three years ago or more, and regularly ride 2-up with no issues. I even had my GF sit on the bike with bell bottoms on to see if it would be OK, and she would have had to add something like an additional 4 or 5 inches worth of material to the pants for them to get behind the cover & caught in the front pulley.