Use a center punch or similar tool to displace material in the bore to reduce the internal clearance. I use 3 “dimples”, 120 degrees apart.
If worst comes to worst loc-tite blue works wonders
While a big hammer will work, the chances damaging the bore with an “out of square drive” are greatly...