ffmpeg – How to modify video framing while encoding at runtime?

I’m developing a webapp to direct volleyball matches from the gym taking score, scouting and live stream integrating all these features in a single panel rendered on a mobile phone screen.

My issue is related to live stream taken at 720p (due to lack of bandwith) from the gym and published to an RTMP server (MediaMTX docker image) where ffmpeg encodes upscaling at 2K (scale=2560:-1,lanczos) and publishing in a local RTMP new path to be accessed by SRT or hls from other clients. I have setup a portion of the screen of my app to direct the framing of what I want to show during my live stream at 1080p moving a cursor on the video that has 16/9 aspect ratio and represent a zoom view of the upscaled video. The move around of the cursor let me decide what portion of the original stream show to my viewers.

The ffmpeg filter “zoompan” is’t accessible at runtime using zmq (cfr ffmpeg flters documentation). I have try mpv too but I’m able to get the stream and move my cursor but I don’t know how to restream in a headless environment.

Someone can address me on how to build up this “system”?

My webapp is written in JS and I can use json to interact with a sort of api exposed by the video serve. I can use VideoJS show original video and “directed” video on my console (either at 360p/15fps for bandwidth issue).

Best
Filippo