What you're describing is referred to as slow speed wander. Very common in boats this size. It has to do with pressure differences on the sides of the hull when off plane. There's almost nothing you can do about it but live with it. Trim tabs will help but not eliminate (in my experience)...