ratpack, linearity also is not perfect and has errors. Power delivered is a function of impedance and a speakers load is NOT linear. Therefore the load the amplfier sees at 1W, 10W and 100W is different. Unfortunately it is also not purely resistive, it is reactive and has a direct impact on an amplifiers performance, particularly in the frequency response.
If everything were equal, loads were purely resistive, then yes it would be linear and we could all rest easy.
I am merely pointing out that specifications are done at a particular set of criteria, in the case of amplifiers full power out at an 8 ohm resistive load. Given a different set of criteria, the specifications would be different, and therefore probably sound different.