This seems to be a common problem. I sent my unit back to have the pot replaced. Worked for a while, then the same behavior has resurfaced with the replacement pot.
I generally don't adjust volume at the unit (using the remote instead, which works fine), but if I did, I would say this is a major defect.