This can be a pretty complex issue (both subjectively and mathematically). However, the link that follows is to a pretty well-written piece on the subject of crossovers. Please keep in mind that there are some issues that are unique to n.1 bass management that are not really addressed by the material in the link, but overall, it is a good primer on crossovers:

http://www.rane.com/note160.html

Actually, if you think about it, there is but one way where the crossover function could perform without producing any ripple in the transition bands - matching 4th order slopes, and only those slopes controlling the magnitude response (like what is done in professional multi-amplified systems).

That is, the 990 is equipped with a 4th order low-pass to route content to the sub output. This equates to 24 dB/octave, or 80 dB/decade (i.e. the filter's output will be 80 dB down at 400 Hz if the crossover frequency were to be set to 40 Hz). On the other hand, its high-pass filter is a 2nd order network, that is 12 dB / octave (or equivalently) 40 dB / decade; one of these...is not like the other. I'm pretty sure that 2nd order crossovers always result in ripple in the crossover region (though I think it's on the order of 3 dB). I could be wrong about that though...

So as things are in the 990, it seems that in order for the crossovers to sum (without ripple borne of differences in the filters' magnitude responses) properly in the transition bands, you would need to match the crossover frequency (i.e. - 3 dB LF response) of the (main) loudspeakers...and...it would have to be a sealed box enclosure, because that is the roll-off of a sealed box (2nd order). Effectively, this would match the (net) orders of the filter systems - 4th order for the low pass, and 4th order for the high pass. Now... if you could select 4th order as a high-pass shape, then you could indeed select a crossover higher than the - 3 dB point of the speaker - effectively the high-pass in the 990 would be controlling the main speakers' LF roll-off rather than the spring (compliance) of the air relative to the compliance of the driver. There would still be some interaction between the 990's high-pass and the box's natural roll-off, but it should be pretty easy to calculate where the additional poles would come into play (as a function of frequency). Just how audible this is...who knows?

As it stands, the high-pass roll-off of the 990's crossover is 2nd order, and only 2nd order. This means that if you want to duplicate the magnitude response of the system (that you would observe at a given frequency with a 4th order high pass), then you'd need to set the 2nd order crossover frequency a full octave up as compared to that of a 4th order alignment. That is, for the 990's high-pass magnitude to be at the same value at 50 Hz that would be realized in a 4th order high-pass, you would need to set the 2nd order's crossover to 100 Hz - because the slope of the 2nd order is exactly half of that of the 4th order. So, it seems to me that setting the crossover to a value higher than that of the speakers' roll-off is likely to cause problems due to the 2nd order nature of the filter in the 990. Thus, a welcome addition would seem to be to allow the high-pass order to be switched to 4th - maybe this is being incorporated into the new Outlaw processor.

Things get a bit murkier when you consider the ubiquituous ported enclosures often used for main speakers. So, what would happen in a similar scenario? Let's say the ported speaker has the same - 3 dB point of 50 Hz as the sealed variant. So far, so good. However, a ported enclosure will have (by nature) a 4th order rolloff. So, by setting the crossover to 50 Hz, we end up with a 4th order low pass (to the sub) and a 6th order high-pass from the mains system (speaker's 4th order roll-off + filter's 2nd order roll-off). Thus, it's not mathematically possible to have flat summed response in the crossover region, because even though the crossover points match, the rates at which the filters' magnitude responses change (as a function of frequency) are not the same.

It would seem that (ironically, or perhaps paradoxically) the best approach one could take with a ported enclosure - from a magnitude response case only - would be to set the low pass of the sub output in the 990 to the - 3 dB point of the ported speaker, and then...not high-pass filter the signal going to the ported speakers. This would result in a 4th order low-pass (for the sub) and a 4th order high-pass (for the mains, created solely by the speaker's natural 4th order roll-off). But again, even with such an undertaking, room gain / sub placement / mains placement will play a major role in what results from all of this. Thus, fundamentally...the most important thing is whether or not you find the manner in which you have set your system's crossover frequencies yields a pleasant sound quality - to you. I want to stress that aspect...because all that I'm really addressing is the mathematical side of the issue (and I'm not even speaking to phase anomalies that result from traditional active crossovers).

Note to self: I should really measure the complex FRF of the crossovers relative to an input signal to assess the phase and magnitude characteristics of the 990's crossovers, unless another user has measured them and is willing to share these data...but I digress...

So this begs the question...how else could one work-around the inconsistencies borne of the differing slopes in the crossover region given what the 990 allows us to select (if we want to attempt flat response in the crossover region)?

If you are using ported enclosures, one way would be to seal the port of the vented (main) speakers as this would make the 4th order roll-off become a 2nd order roll-off. By doing so the crossover (in the 990) could be set to the 'new' - 3 dB point of the now-sealed speaker (but you would have to know what that frequency is), and because the speaker would now be a 2nd order function (as would the filter with the same - 3 dB point as the speaker), a net 4th order high-pass would result. I'm not sure how good this would sound (as merely changing the box tuning from a 4th order alignment to a 2nd is likely to introduce some magnitude anomalies), but from a theory standpoint, it seems to make sense (and no, I'm not saying that a 'correct' measurement necessarily equates to a good-sounding system), but again, anomalies created by merely sealing the port may make the exercise wholly academic.

In a perfect world, an 'intelligent' pre-pro could be designed such that part of the set-up procedure would be to use a supplied microphone, placed in very close proximity to the LF driver of the main speakers (i.e. a near-field measurement) and then a shaped random stimulus be sent to the mains. One could repeat this measurement process for the subwoofer as well.

Such a set of measurements would yield the magnitude and phase response of the main speakers and the sub fairly independent of the boundary conditions. That is, the measured pressure response of a near-field measurement will be governed by the speaker and not the room.

There appears to be a real advantage (setting the complexity required to implement this aside) to this approach in that the actual roll-off for the speakers that you actually have - not what's listed on their specification sheet) would be known (and this would address manufacturing variability related to the compliance equivalent volume (Vas)). As such, it would be relatively easy to provide the complement of that to a DSP such that whatever speakers were used for the mains, that the correct and complementary low-pass filter network (most likely an IIR, but possibly a FIR) could be provided.

That is, I can see how such an approach could allow the user to define what order system was desired, and then (I'm a little rusty here, but it might work) poles or zeroes added as necessary to define the filters such that a specific net desired result was possible. Taking things further, if the menus for such a beast incorporated something like a 'which alignment would you like?' type option (i.e. 2nd order Butterworth, 4th order Linkwitz-Riley) then some clever DSP programming should be able to do something like this, that is, allow the issues surrounding crossovers to be remedied. At least in this way, by measuring the magnitude and phase characterisitcs of the actual system drivers, the chances of defining known shapes with known results seems more likely.