Opinion: this is mine others will differ, just wait
The motor does not have enough power to do what you want for the load placed on it. Reducing load by off loading folks and running with less weight works.
To fix you either need more HP or less pitch prop. Reducing pitch from lets say 19P...