I've been meaning to reply to this post for a while now. Let's see what I can offer...
When I now run the THX optimizer with the Samsung set to PCM using optical out and the 990 on Dolby PLXII movie it does not send the test to the rears or sub but instead sums the surround and rear to the front.
This sounds about right. PCM over optical or coaxial will always be limited to stereo, as there is not bandwidth available for more than two channels of PCM audio. Doing test tones in this manner is not going to be useful at all.
Finally the question. What should the 990 be set on for 6.1 and what should the Samsung be set on?
I'd have to dig into the Samsung manual to see what options there are for optical and coaxial output, but you will probably want to set the Samsung to bitstream output. You will probably also want to connect the multichannel analog output. The BD-P1500 will decode Dolby Digital Plus and TrueHD, so for those and multichannel PCM tracks you will want to use the player's multichannel analog output. For DTS-HD High Resolution, DTS-HD Master Audio, and plain old Dolby Digital and DTS, you can use the optical output just as well as the multichannel analog.
Question two. When I run auto setup it sends a signal to the rear center but does not recognize it in the setup. I have to go back and manually insert it. Should I run a Y connector to the amp from the suround back output to fool the 990?
This one I'm not so sure on, as I've not ever set the 990 up for 6.1 and I recall there being something quirky about it. You might touch base with
Outlaw support to see what suggestions they can offer.