If your speaker is totally un-shielded, it will distort the colors at the edges of the TV picture tube. You will need to eventually de-gauss the tube if this happens. Some sets do this automatically and some have a button to press to do it.

Your other speaker might be better shielded or simply more immune to picking up the signal. I would definately use a shielded speaker next to a TV receiver (or CRT computer monitor).