Unless you are playing it really loud, espeically on music that has high average power such as rock or organ, the amp isn't in any great danger.
Let's do a calculation. Say it's a 32W amp, designed for 8 ohm speakers. That means it's rated to deliver 16V and 2A at full power. I've chosen the numbers for easy calculation
It's the current that can damage the amp so it will safely deliver 8V and 2A into 4 ohms. Hence 16W. Unless you really wind up the wick, most of the time the amp will be delivering a fraction of this so don't worry unless you're an organ or heavy rock addict. The heavy bass in this sort of music can push amps (and speakers) beyond their limits very easily.
PS: Many years ago I blew the bass drivers on my BC1s when doing a frequency sweep test. The very low bass sounded a bit quiet so I wound up the volume. 100W of 40Hz from a Quad 405 quickly burnt out the bass units.