You are missing something. The amplifier, even when powered off, does not present an open circuit to another amplifier.

When powered on, one of the design goals is to try and minimize the output impedance of the amp. In simple terms, the output terminals look almost like a short circuit to anything connected to them. This is not good if that something is another amplifier and there happens to be a signal present at one or the other set of input terminals.

Swapping speaker leads is cumbersome at best. You might try constructing a custom switchbox, using a couple DPDT break before make switches of suitable amperage rating. (DPDT - Double Pole Double Throw)
You would want to switch both legs of each speaker circuit which is why the double pole spec. Break before make assures you that there is no way to parallel the outputs of the two amplifiers. Add a box and 4 sets of dual banana plug receptacles and you're pretty much there. Digi-Key, Allied Electronics, and Newark Electronics will have the necessary parts in their on-line catalogs.

Good Luck.