There is a Q&A at Monoprice.com about this that says:

Question: Can I use this switch to switch 5.1 channel analog audio instead?
Answer: No, it won't work. Levels are not matched between the inputs.

Maybe it is because video connections are supposed to be 75 ohms, but the audio ones are not?