Sounds like someone has done a botch job. Those hinge pins ONLY run directly in the gimbal ring. Maybe PO/mechanic has 'bushed' them to be able to sell it, but it's definitely not right.
Can you continue to use the boat?... Short answer, no. With those hole/pins damaged/worn to the point where you're seeing it, it means the drive is SERIOUSLY out of alignment with the engine and engine coupler. It will be a very short time before you damage the coupler and will be replacing the coupler, as well as the gimbal ring.... In fact, the drive just sitting out of alignment is putting a side loading on the coupler and gimbal bearing. I'd be getting the drive off there TODAY! (Check my videos to see how quick and easy it is to remove a drive. Link in my signature.)
Chris.........