The noise and vibration is a thought, but I am more concerned with the thrust of the motor.
I am doing some searching here and a couple issues that appear to come up are weight and thrust.
In regards to thrust - there is usually reference in these threads about the "hull speed" or "displacement speed" of a hull, which in doing some more searching roughly means the speed at which the frequency/wavelength of the wave created by the bow of the boat moving in the water matches the length of the boat. Though, that seems to be a bit misleading because I found references to that figure and there not being any "special" increase in drag at that moment. Obviously, as you speed up the drag increases. However, what I found with the displacement/hull speed was nothing like the difference between a boat plowing and jumping on step - where there is a "jump" in power vs. speed in most cases. In reference to the hull/displacement speed, though, what usually comes up is that around that speed there is a law of diminishing returns with motor hp. The difference between a 5hp motor and 10hp motor, in terms of increase in thrust and forward speed, is a moot point. Going between displacement/hull speed and step/planing speed would take way more than a 10hp motor, but that isn't the point. Adequate thrust/control is - as emergency power to get home, but most often used for trolling.
Some references I found were in regards to greater than 20ft boats in rough seas where with much wind and larger waves a 5hp motor would be taxed pretty hard. In calm seas the 5hp would work fine. In these cases the usual recommendation is more power.
Then there is the added weight on the transom to contend with. When I make my rounds on the lake I always have a spare motor with me - either a 9.9 or a 4hp, mostly the 9.9. Just to try it last year I put the 9.9 on the transom just like I would if I was out on the lake and the main (25hp) quit. I moved the 25 over and dropped the 9.9 on. It usually rides up front to distribute the load better. With the 2 motors on the transom the bow went up to about a 45deg angle. I didn't give it a good run or adjust the ballasting, I did not have any other gear in the boat as I was just testing it. The bigger the boat the less this would be an issue, but the other side of it is transporting the boat and the road torture. If the kicker is permanently mounted this is extra weight the transom has to endure.
I suppose a kicker wouldn't be too hard, even if it is a bolted-on mount, to take on and off the boat. So transportation aside I think the 2 questions are 1. the amount of thrust/control and whether, with a smaller boat, a 5hp motor would give adequate control; and 2. if the added weight of a larger motor would have adverse affects on the performance of the boat. With a light empty boat on a calm day (16ft, 64" beam) the 4hp we have will scoot it pretty good. Quadruple the weight and add 12-20" to the beam and what happens?