You have to pull the drive off the boat... at least you should pull the whole drive off and not just the lower half... to service the impeller.
There are other things to check and service while the drive is off, like greasing the u-joints and gimbal bearing.
Check the gimbal bearing for any roughness by spinning it with your fingers, should feel smooth like silk.
Check alignment of the gimbal bearing to the coupler which is bolted to the back of the engine.
Check the bellows for any tears or cracks in the rubber which can fill the boat with water if they fail.
Check out the "sticky" at the beginning of this forum labeled "Adults Only" it's chocked full of good info...
Here are some videos to check out:
Removing the outdrive from the boat:
Separating the upper from the lower
Water pump removal