This is somewhat related to another post on here where mixer plugins were used for routing, and the construct was massive. You might find it interesting to read and see what he did, especially the global screenshots, and no complaints on latency. Build what you want and try it out, I’m sure there will be no latency or resourcing problems.
Rocket Surgeons - Gig Performer in Action - Gig Performer Community