I use an aggregate device with my interface and the blackhole virtual driver. I seem to recall having issues like this, and when I checked the Audio MIDI Setup in MacOS, the blackhole was listed as the ‘Clock Source’. I changed it to my interface and it resolved things.
But it’s been a while, and my memory is hazy…