For me, I can't stand being pushed forward out of the sweet spot on my seat. I also need to be able to squirm around just a little, rearward. I want to feel just like I'm riding without the backrest on at all, but have it be there with just a little bit of lean backwards. OTOH, if I'm going to be riding in twisties, the backrest isn't invited along for the ride.
I like mine to just touch my back. It seems to urge my back to be upright without forcing it. But it still can ease the strain on long rides. Wouldn't want to be without it.