If the linked head unit and pair of speakers drained your battery in two hours to the point where it wouldn't crank the motor over, then yes, your battery was not healthy to begin with.
Just to make sure, I'd charge the battery fully on a charger, and test again. Since you didn't mention the state of charge the battery had to begin with. Outboards, older ones especially, aren't always the best at charging batteries.
For perspective, I've ran my stereo (two amps, 6 speakers, 1 subwoofer) off of a group 34 odyssey battery for at least four hours and probably more, and then used it to crank my boat's engine right up. So something is definitely not right with your setup.