I wonder if you're hearing a difference in phase between your mains and the subwoofer in the cross-over frequencies when you listed to music? If they are out of phase they could be fighting each other.

Does your sub have a phase adjustment that you can tweak or just a 0/180 degree switch?
_________________________
975/7075/SMS-1
Aperion Verus Grand Towers & Bookshelves, Verus Forte Center, Infinity Surrounds, Ultra-X12
Oppo BDP-203, XBOX360, Xbox One
LG 65" OLED
RR2150 w/Klipsch SCR-2