You need a size larger than you currently have. You shouldn't have to wear something really thin (silks) under the jacket liner.. They are too thin and you will get burned...Don't ask how I know. I just went from an old large wire to a new micro wire liner and let's just say you need something on your neck under the zip up collar to protect your neck.
If you can't wear a medium thickness shirt under the liner, it's to small.
I have a couple of shirts, that I call mouse hair. They are a really short pile fur that feels good next to your skin. Your liner should accommodate wearing that type of shirt.