FFMPEG mp3 to mu-law output delayed – Python subprocess w/ pipes

I am using a python subprocess and pipes to convert mp3 audio into mulaw. The basic problem is that i get about a 0.5second delay in the first output. The input is being piped in as one 45-byte ID3 header and 480-byte mp3 frames (each a “chunk” in the code). It seems to wait about 40+ frames/chunks before generating any mulaw output. After that it generates mulaw output pretty much with each input frame/chunk.

# This spins up a single ffmpeg process and uses pipes to write and read from the process.
#  Current issue is that it wants a bunch of mp3 frames before it gives an audio back.
def convert_chunks(self, chunks: Iterable[bytes]) -> Iterable[bytes]:
    # Start the persistent ffmpeg subprocess
    ffmpeg_process = subprocess.Popen(
        [
            'ffmpeg',
            # None of these options seem to help with the delay.  One of them causes the first few seconds to be cut off.
            # '-fflags', '+nobuffer',    # Disable internal buffering
            # '-flags', 'low_delay',     # Enable low-latency processing
            # '-flush_packets', '1',     # Flush packets immediately
            # '-max_delay', '0',         # Eliminate additional delay
            '-f',
            'mp3',  # Input format is MP3
            '-i',
            'pipe:0',  # Read input from stdin
            '-f',
            'mulaw',  # Output format is mu-law
            '-ar',
            '8000',  # Set sample rate to 8000 Hz
            '-ac',
            '1',  # Set audio channels to mono
            'pipe:1',
        ],  # Write output to stdout,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        # stderr=subprocess.DEVNULL  # Suppress ffmpeg output for simplicity
    )
    # this is important to avoid blocking the read from ffmpeg stdout
    os.set_blocking(ffmpeg_process.stdout.fileno(), False)
    # input('Press Enter to continue...')
    closed = False
    try:
        for chunk in chunks:
            ffmpeg_process.stdin.write(chunk)
            ffmpeg_process.stdin.flush()
            audio = ffmpeg_process.stdout.read()

            if audio:
                yield audio
        # we do want to block on the final read
        os.set_blocking(ffmpeg_process.stdout.fileno(), True)
        ffmpeg_process.stdin.close()
        closed = True
        audio = ffmpeg_process.stdout.read()
        if audio:
            yield audio
    except Exception as ex:
        print(f'convert_chunks exception: {ex}')  
    finally:
        if not closed:
            ffmpeg_process.stdin.close()
        ffmpeg_process.wait()

There are obviously some other things i can look at (including the python code, and maybe run some trace-s on the process) but maybe somebody knows if this is a fundamental limit to how quickly i can get the mulaw out?