Whenever there is a question of whether there truly is a change, a double blind test is the only way to really sort out the truth. To properly do this in the situation you mentioned, you would have the person switch (or not switch) the configuration back and forth with you staying exactly in the same place, and without you knowing the configuration you are hearing. Score would be kept of correct identifications. Anything less than 80% correct identifications is questionable - 50% is what you would get by just flipping a coin.

The human ear is as easily fooled as the eye (ever seen an optical illusion?) and our acoustic memory is generally bad. Also, what you heard just recently (or silence) dramatically influences what you hear a minute later. A speaker for instance will sound slightly different to your ear if the thing you heard 5 minutes before was a truck verses silence, or another speaker. Also, if you drunk or ate something either hot or cold at that refridgerator, this will greatly change your hearing perception for several minutes afterward.

Personally, I've had probably thousands of instances where I think I've heard a dramatic (or slight) change from one day or minute to the next, especially after I've changed something in a system. More often than not when I've checked myself by having somebody change the system back to original and to the new configuration without me knowing which is which, I've been unable to reliably tell any difference.