That’s a good question often asked of me. Depending on sample rate, the lowest buffer you can get to without pops/clicks or other issues (perhaps like what occurred with me spurring this post in the first place) is the best for drums. The impact on the sticks in the hands comparing it with what you hear can have a massive disconnect if there’s anything later than I’d say around the 6ms mark. If you can get it lower, even better. Once it gets closer to 10ms, then I personally consider it unusable.
There’s a little drum expander box called DrumPi, based on the Raspberry Pi computer, which I am helping create kits for (I have done two, there’s a 3rd coming) which latency is 0.4 ms, so incredibly quick. Not taking into account the latency of midi delay, which from an eDrumin is approximately 3ms depending on how you have it set up. But when you play it, it’s response and how that feels as a drummer is sublime. I may go down this route live in the future, depending on how it develops as it’s still quite a new concept and very much in its beta phase.