Just wondering:
Does setting the Denon to bitstream cutoff your analog signal? That could explain your RCA input not working.

The Denon is sending a "raw" bitstream, right?

Another option while you figure this out is to select DTS NEO:6 Music in the 970 or Dolby PLIIx-M. However, I do like the 5-Stereo/7-Stereo modes better.

Good luck.