The loudness adds bass so it is changing the frequency level. The speakers are most likely for high & mid range frequency which is typical of round speakers. 5x7 & 6x9 oval speakers usually handle bass frequencies better.

I would look into getting some bas blockers (they wire inline with the speaker wire) to filter out bass. Crutchfield has them.