Assuming the boat is in gear when your full throttle rpm's are too high, you need to know how high, and using that info, get the correct prop for it as the one you have now is too small!
Insane to test an engine out of water at full throttle while on muffs, could test at idle/fast idle or bit more while at loaded when geared forward, just watch out for spinning prop...