Audio lagging when no screen updates are happening in VirtualBox?

I’ve got a VirtualBox VM on Arch Linux running Mac OS X 10.6, and when it plays audio, if my cursor is still and nothing on the guest display is moving/updating, the audio starts lagging. If I shake my mouse really fast (in the guest window), it plays smoothly until I stop moving my mouse.
If there is a progress bar moving on screen, the audio will play smooth until it disappears or stops moving on screen.

Changing the audio format to 22050.0 Hz in Audio MIDI Setup seems to make it stutter/lag less, but there is still a lot of stutter.

So, how do I fix this? And is it an issue in the guest (Mac) or the host (Arch Linux)?