There's no reason that there should be such a pronounced difference between the cable types. It sounds like you either have a bad cable, a bad coaxial output on the box, or some really odd interference condition that's corrupting the data over coaxial. Of the three, I think one of the first two is most likely.

Either connection (assuming no faults with source or cable) should work just fine. If you need to get a new one of some sort, I'd look at a few online sources. For optical, I'd go with either Outlaw's PDO ($20 in 1.8-meter length), MonoPrice optical, or BlueJeans. For coaxial, I'd look at BlueJeans or one of the higher quality (not the cheapest) MonoPrice. I've used MonoPrice for some of the longer runs in my system (mostly sources like DVD recorder, VCR, and AppleTV) for stereo analog, component video, optical, and HDMI. I've used Blue Jeans or Outlaw for just about everything else, including a lot of Blue Jeans component video, some analog audio, and some coaxial digital audio. You'll get more quality for less money from any of these compared to Radio Shack or someplace like Best Buy.