Speakers are one of the few things I can believe might break in, although the changes the mechanical system experiences over time is minute and for the most part complementary.

In any case, if you don't match the levels of the new drivers the output might sound bright. Try building (or altering the existing) resistor network to tame the Audax.