I've got a vanilla blue-ray player connected to my 975. The BR is set to 5.1 sound. I THINK the 975 is as well, but it won't play out the rears.

Now, with an Xbox One it works fine (using the speaker test, where the grunt points at the current speaker as he talks through it), so I know it's connected correctly.

The manual is absolutely impossible to figure out. What scrolls by on the 975's display bears little resemblance to what's printed in the manual. The abbreviations are too hard to figure out.

So, what setting(s) do I have to use to watch a movie with 5.1 sound?

Are you using the On screen display? I think the manual is written to assume you are. Without it it could be much more difficult to set the system up. If you have a computer screen which inputs HDMI or DVI you can use that if you don't intend to do video. Once the OSD is on, go to the speaker setup menu and make sure the surround mode is selected as 5.1. Then go to the speaker calibration screen and see if the system sees all of your speakers. Use the test tone to calibrate the system.

The display screen scrolling for 5.1 audio, for example, would show (3/2).1. The 3 is for the front sound stage, left, center, right. The 2 is for surround left,right. The .1 for the sub, naturally.
What can be confusing are the parenthesizes before and after the 3/2. The first one looks like the letter'C'.
The Edit boxes, and Letters, to the right of the display show the discreet signal format (all the letters), and your active speakers (all the boxes).

Check the icons on the right side of the 975 display. Page 15 in the manual shows the options for how these should look for 5.1 (and 7.1) inputs.

If it doesn't look like one of those revisit your setup.

Correct. The letters are the incoming signal, from your source device, and the boxes are the outgoing signal from the 975. If you see:

[L] [C] [R]

[SL] [SR]

[LFE]

Then you are receiving and sending out 5.1. If you don't see boxes around anything but L and R, then you have the unit in stereo mode and will need to use the SURR button on the remote to remedy this.

I tried both the OSD and the mini-menu.

And, remember: it works for other devices.

I'll check those. I remember seeing them, but I'll exhaust the choices until I see them all.

All 6 have boxes around them. The three front speakers all have sound coming out. I can't tell if the sub is working. The two rears have a very light static if you put your ear really close, so they're getting power.

When I try surround mode, I get (3/2).1. If I hit it again, I get "mode unavailable" over and over. Which makes no sense, because earlier I got 5 or 6 choices (with the same movie in the same blue-ray).

Remember: from another device (an Xbox with the Halo speaker test), all six speakers work correctly.

What Bluray player are you using? If one source functions and the other doesn't I would suspect the source being the issue. Have you tried swapping HDMI ports to see if the port is bad?

Some blu-ray players will send limited channels out to HDMI when the analog outputs are also used - any chance you have both HDMI and analog outputs from the player set up? Check the audio configuration of the player - it might not be configured to send the 5.1/7.1 bitstream.