MacOs audio is misreporting latency on my M1. – Logic Pro

I’ve been experiencing some weird latency related issues lately.
Today I had time to sit down and measure my Macbook Air M1. I tested the Mac’s internal audio and with my RME Babyface Pro FS. On my computer, the recordings get misplaced by about 45-55 milliseconds in Logic if I record via the computer’s built-in microphone. When I use my Babyface, the audio tracks end up exactly where they should.
Is anyone other than me experiencing this problem?
I have a MacBook Air M1 with 16 GB Ram and 512 GB storage.

FYI: The testing was done on a new user account. I also ran RTL Utility which showed the differences between measured and reported latency to be 2316 samples at 44,1 kHz sample rate on buffer sizes 64-128-256-512.