It'll just throw things slightly out of phase. How badly, I don't know... but probably not enough to make an audible difference.

A tone near the edge of human hearing, at 20 kHz, will be 180 degrees out of phase if you introduce a delay of 1/40,000 sec. Using Brianca's .3 number, that still leaves about 8200 feet difference.

So, it's still out of the ballpark by about two orders of magnitude.