I'm betting that it would work just fine. The switching circuitry won't send the electrons off in any certain direction. There will either be an electrical connection, or there won't. The directionality of which way you want the signal to flow is irrelevant. Just like you don't hook up interconnects with a 'to' and 'from' side (at least normal people don't -- some extreme, delusional audiophiles do), the switchin circuitry will make a connection regardless of which 'direction' the signal is flowing.

The only reason this wouldn't work if all of the switching was taking place inside a processor, like sampling the video, processing it, then sending it out another set of connectors. Other than video processors, like scalers, pretty much all video switchers just make an electrical connection.