You have a wiring problem (you should get battery voltage, or close to it, when you energize the yellow/red wire at the solenoid) whether you are doing this with the ignition key or the remote starter switch.
When troubleshooting a starter problem the first thing you should be doing is measure the voltage between the big positive connection on the solenoid and the battery negative, this should be battery volts, if not you have a problem with your positive or neg cable or the negative ground stud on the bellhousing for the negative cable. Next, you connect a voltmeter between the yellow red wire at the solenoid and an engine ground, and have someone turn the key, same thing battery volts or close to it, If your are getting 5-7 volts something is most definitely wrong! You need to get a V/P wiring schematic and follow how the current flows for the starting system and find out where you are losing voltage.
First check for batt volts at your big pos terminal on the solenoid
then check for batt volts between your yellow/red wire at the solenoid and ground (or negative return path more correctly stated). Then get the schematic and starting checking each connection. I know its a pain but that's what you gotta do
the only other thing I can think of, is if a mechanical problem is causing the engine to be hard to turn over and that is sucking so many amps through the starter that your voltages are reading low. But, if that were the case the starter would be getting real hot real fast. Sometimes you have to pull the spark plugs and try to turn the engine over by hand, water in cyl can cause this, and so can problems with the outdrive, gimble bearing etc.
did you bench test the starter off the engine?