I wrote a long post on this subject that is buried somewhere in the archives. Basically, the more efficient your speakers are, the less gain you want from your power amplifier. A more powerful amplifier has more gain than a less powerful one, and the inherent noise from the amplifier is raised. A less efficient speaker will not make this noise as obvious as a very efficient speaker will, and horns are very efficient.

Also, the less of the available power your use from a class A/B power amplifier, the closer you are to the point where the "upper" and "lower" transistors hand over the signal. The area of this transistion can contain small amounts of what is known as crossover distortion, and this is made more audible by efficient speakers. A less powerful amplifier that is driven harder has less of it's signal residing in this crossover region. A powerful amplifier that is barely diriven spends most of it's time in this crossover region. The resulting sound can be harsher and grainier.

The best solution with speakers that are very efficient is a class "A" amplifier that has no crossover distrotion potential at all since a single output device or devices reproduce the entire signal. Vacuum tube class "A/B" power amplifiers are better than class "A/B" solid state amplifiers in regard to crossover distrotion because there is far more overlap between the upper and lower halves of the signal between the output tubes. The ulitmate solution is class "A" tube amplification for very efficient speakers, but this can be an expensive path.