If you have 1,000 songs as mp3’s on your phone, do all 1,000 eat up cpu while you’re playing one of them? (In case you’re not sure, the answer is “no”.)
It’s the same principle in GigPerformer. If it’s not in an active rackspace, it’s not using CPU. (There are some minor caveats to that, but for the most part that’s the right mental model.)