An unbalanced connection basically has the signal line (think the tip of an RCA connector) and a shield that completes the circuit, and helps absorb radio frequency interference before it gets into the signal line.

A balanced connection uses 2 signal lines, one with the signal's polarity reversed (so when the signal goes up, on the second line it goes down instead). A balanced line also has a shield to absorb the RFI before it gets to the signal. When a balanced signal gets to the next piece of equipment it goes through either a differential amplifier, or a transformer, which takes that second signal line and flips it back into polarity with the first. The two signals are then added together.

When the polarity is flipped at the end, any noise that was picked up by both signal lines is now reversed polarity between the two lines and is cancelled out. You also get a +6dB boost in signal when they are added together. It is also possible to disconnect the shield from the circuit in order to break ground loop issues as you have a complete circuit between the hot (+) and cold (-) lines.

For the most part balanced connections are desirable for runs over 20' long. Some equipment may have better sounding circuitry on the balanced side, but it can be a gamble as differential amplifiers and transformers all have distinct character which may or may not be helpful to the aesthetics of the system.

My personal feeling on the matter for hifi equipment is that if you don't have long runs, or special equipment that requires balanced, don't worry about it.