Just wondering:
Does setting the Denon to bitstream cutoff your analog signal? That could explain your RCA input not working.
The Denon is sending a "raw" bitstream, right?
Another option while you figure this out is to select DTS NEO:6 Music in the 970 or Dolby PLIIx-M. However, I do like the 5-Stereo/7-Stereo modes better.
Good luck.