Certain frames in the video have a time stamp on them. For various retarded reasons, the timestamps can sometimes be incorrect, or set way ahead or behind what they're actually supposed to be. It causes the problem you are explaining.

So what you need to do is go through the whole video and redo the time stamps from scratch, there is an application that will do this for you at www.repairvideo.com/ it has worked for me many times.