While the chances for harm to any of the equipment connected that way is small, there are issues, such as impedance changes, that can affect the results of such a Y connection. It may be best to have only one of the sources powered on at a time so that the two outputs are never ‘bucking’ one another. Personally, I would look for another way, even if just a 'multiple selector switch' box of some kind.