x264 – Why does FFMPEG b-frames and b_pyramid offset start_pts (and start_time) in fragmented output?

It seems when transcoding something into fragmented or segmented output the start_pts is not 0, and no combination of options can make it 0 other than setting bframes to 0 and bpyramid to 0. This does not happen with regular nonfragmented output.

Does anyone know why this is, and how to prevent it as I believe this is causing weird timesync issues with playback in browsers (every fragment slightly delays the video more and more) where after 2-3 hours the stream can end up 15minutes+ delayed.

Example where each segment start_pts is not 0:

ffmpeg -i in.mp4 -pix_fmt yuv420p -an -f yuv4mpegpipe -frames:v 150 - | ffmpeg -f yuv4mpegpipe -i - -y -force_key_frames 1,2,3,4 -map 0 -codec:v libx264 -f segment -segment_list out.csv -segment_times 2,4 -segment_time_delta 0.05 -preset:v fast -segment_format_options movflags=+faststart out%03d.mp4

start_pts is 0 here if we add:

-x264opts b_pyramid=0 -bf 0 
#or change codec to
-codec:v mpeg4
#or output regular mp4
ffmpeg -i in.mp4 -pix_fmt yuv420p -an -f yuv4mpegpipe -frames:v 150 - | ffmpeg -f yuv4mpegpipe -i - -y out.mp4

EDIT: Looking into this further I am starting to think this is a bug with how empty_moov interacts with the negative_cts_offsets flag. (when empty_moov is used, negative_cts_offsets seems to be ignored, and we need empty_moov for full webbrowser support.)