For the record, in case anyone else has this problem, I think I figured out what's going on. The crossover for my mains is set to 60hz. The crossover for my center and surrounds is 100hz. That means in movie mode the sub is playing freqencies from 60-100 hz--but in stereo mode it's only below 60hz.

I think because of room acoustics or a sub problem I have a lot of extra energy between 60 hz and 100 hz, and a bit of a null below that (at least, it seems that way from pink noise sweeps). So when I calibrate the sub so that it's not too loud from the 60-100 range, it doesn't really fill in the low frequencies that well.

The folks at Outlaw recommended I switch my center crossover and the mains to 80 hz. That way the sub is producing a bit more of the bass range in stereo mode. That does seem to help a bit so far.

Of course, my main issue is to try to fix the boominess in the upper bass. I'm not sure if I need a bass trap, or if there may be something wrong with my sub such that it doesn't reproduce ultra-low frequencies anymore (it's a decade old, and it's been moved 3 or 4 times).

Hope this helps someone!