Take some sort of wire brush style angle grinder attachment and just lightly go over it. Steel wool or high grain sandpaper would work well too. Just don't take too much off or you will mess up the balance of the crank assembly.
Thanks for all the quick responses. The flywheel has been sitting in a box while I have been distracted during the rebuild. I'll post some pics as it gets going.
I say leave it alone, it's not hurting anything and your much more likley to cause a problem getting sandpaper grit & rusty bits into the bearings.
Well I would just like to add that whenever you are rebuilding an engine KEEP THE PARTS CLEAN AS i SPRAY THE HELL OUT OF EVERYTHING WITH BRAKE CLEANER..