Hmm … with 64 samples buffer you should hear really no sluggish response.
If you dare you might check in here:
Some research on latency, jitter and buffer size
With a decent interface (RME Baybface Pro in my case) 96 or 128 samples buffer should be absolutely fine. And I’m really picky when it comes to latency and jitter …
I run my Babyface at 96 samples with no CPU problems playing heavy layers of Omnisphere, NI Kontakt and many more.