I think the problem, is that if you are jumping around it will always take a long time to load - ‘predictive’ loading is based on the rackspaces GP thinks you will move to, and the only way it does this is that it is the rackspaces that are next to the one you have selected; it assumes you are moving up/down in a list .
So even if it were possible to just have a single rackspace predictively loaded, if you don’t jump to that rackspace then GP has to load it anyway and, as you say, it can take a while. This isn’t really a flaw in GP as it can have no idea what song you are going to go to if you are moving around a setlist almost at random.
Also, the first rackspace that is loaded is the selected one, then the others around it depending on how many you have set in predictive loading - it doesn’t matter if the loading bar takes a while whilst it loads the others as the one selected will be ready to go as soon as it can - check for the little green bar next to the rackspace name.
Predictive loading is fantastic if you work through a setlist and it really does make loading time zero when selecting one of those setlists.