Welcome to the club! I’m a latency nerd, too:
In short: A bit of a constant latency is like some distance to the sound source and isn’t the main problem. Otherwise, no one could play Bach Toccata and Fugae on a church organ, ever.
The problem would be latency jitter, which is like a random timing or unsteady groove. This jitter could happen by improper handling of the audio blocks and incomming MIDI in the software, and this would go really worse with larger buffer settings. This was the case with the discontinued former VST host Forte, which forced my hunt for lowest possible buffer size, in the past.
As you found out, too, RME has exceptional low-latency interfaces and drivers. Some others too, just to say. And GP has a very low latency jitter MIDI to audio engine, so as long you use a resonable low buffer size, latency isn’t an issue.