I haven't spent much time messing with 5.1/7.1 Stereo, but I'll offer a theory based on my limited experience and understanding. The 5.1 Stereo mode has always been louder than normal - same signal is being copied around to other channels, so you have more speakers playing at the same time. When you've got 5.1 Stereo mode going with small speakers all around, you have five channels redirecting low frequency information to the sub rather than just two, so it makes sense for the sub to be more energetic. I'd be curious to know what you find if you go back to stereo or upsample and simply dial the subwoofer trim up four or five dB.

You have the mains set to small and the sub set to L/R+sub? I didn't think that was possible. The L/R+sub mode is supposed to be used with large mains (it's a form of "double bass" in which the mains are getting a full range signal but a copy of the data below their crossover point is also handed off to the sub to reinforce the low end).