What sources were you playing? DVD-Audio, SACD, and Blu-ray will sound better via analog than coaxial because they will give you the lossless audio instead of lossy.

For DVD's and CD's, I've found that the BDP-83's analog section is at least as good as the Onkyo 885 (which is itself comparable to the 990). My wife actually preferred listening to CD's via analog over HDMI, and I've found myself bouncing back and forth between HDMI and multichannel analog a fair bit.
Source direct outputs the original video format of the disc or file being played. For DVD's, that means 480i. For Blu-ray, that can be 1080i or 1080p. The TV has to scale that signal to its native aspect ratio - there's no setting to adjust. The TV won't show that it's doing this, because it will probably list the input resolution.

The benefit to source direct comes if you have a video processor in the signal path that is equal to or superior to the BDP-83's ABT 2010. Maybe you've got a DVDO Edge (which uses the same chip) or VP50 video processor handling all signals, or an Anthem Statement D2 with its Gennum processor - in which case having the BDP-83 output an un-processed signal lets you get optimal use out of that separate video processor. It also is smart enough to switch to 1080i for SACD and DVD-Audio playback, since with HDMI you need an HD video resolution to allocate enough bandwidth for multichannel audio. For most of us, though, the best bet is going to be to set the BDP-83 to the output resolution that is closest to the native resolution of our TV.