Unexplainable choppy audio during playback?

I’m streaming H264/Opus from an Axis M1135 camera via WebRtc. The audio RTP packets contains 20ms of audio, but the timestamps do not exactly increment by 960 (which from my understanding must be the case).  Sometimes the timestamps increment by a slightly  different amount, e.g. 953, 964, etc.  This results in a very high amount of concealed samples and choppy audio. Hardcoding and incrementing each timestamp by 960 results in smooth playback.

I’ve written a more in depth explanation of this issue:

I haven’t found the reason why the audio is choppy, when the timestamps do not exactly increment by exactly 20ms/960. Does someone know why and how one could correctly solve this?

roxlu