Without getting really detailed and somewhat boring, it comes down to which metal has more valence electrons. Silver has more than Gold, so they can be torn away easier and it is more conductive for that reason. If you want more info, do a google search for Bohr's Atomic Theory. This differnece would still be very negligible.