So even playing directly, with no scriptlets, those sample rate and buffer settings are going to give you 10.5 ms latency before you do anything (512/48000) which will be VERY noticeable.
You really need to be at 256 or even better 128, which is typical of most setups
If you’re getting glitches at those sizes then either your computer is under-powered or you’re trying to use plugins that can’t handle the processing in the time allotted. Your audio device drivers might also not be up to par (what audio interface are you using)