How to detect when gigfile has finished loading? How to detect when rackspace is ready?

[Solomon]

I’m running version 4.7.

I have some code in the “On Variation” callback in my global rackspace script which calls GetRackspaceNameAtIndex(GetCurrentRackspaceIndex()).

I’ve noticed that when I first run the application and load my gigfile, this callback gets called many times: once for most of the rackspaces in the gigfile (but not all). I’m curious as to why it isn’t called for every rackspace.

I’ve also noticed that on some occasions (which I can’t reliably replicate), GetCurrentRackspaceIndex() returns -1 (which is accompanied by the error “GetRackspaceNameAtIndex() canot be called at this time”), and sometimes when it returns 0, which is a valid rackspace index, GetRackspaceNameAtIndex nevertheless returns an empty string and emits the same error message.

Now, none of my “On Variation” code actually needs to be run at gig load. I only need this code to execute when the user is actually switching to a variation for performance. So I have two pressing questions:

1. In order to speed up the loading of my gig file, is there a way that I can detect that a given “On Variation” call is happening during load (in which case it can immediately return without doing any work) versus during ordinary operation (in which case it needs to do what I’ve designed it to do?)

2. I’m supposing that the sporadic errors are happening because the “On Variation” is invoked in one thread, whereas the rackspace state that GetCurrentRackspaceIndex and GetRackspaceNameAtIndex are relying on is being set up in a different thread. How can I reliably know that the new rackspace/variation has been fully loaded before I proceed? My current best idea is to loop until GetCurrentRackspaceIndex() returns a nonnegative integer, and then loop until GetCurrentRackspaceIndex() returns a nonempty string. However, I’m not certain that even this would guarantee that I can safely proceed, and I’m a bit nervous about the possibility of infinite loops.

[Markus]

@Solomon

I’ve noticed that when I first run the application and load my gigfile, this callback gets called many times: once for most of the rackspaces in the gigfile (but not all). I’m curious as to why it isn’t called for every rackspace.

Just an Idea, maybe this is because you use predictive loading which limits the number of rackspaces beeing loaded on startup?

[Solomon]

Hmm, no, I don’t use predictive loading.

The precise list of rackspaces for which “On Variation” is called during load varies from one loading of the gigfile to another and seems to be random. It looks like “On Variation” gets invoked for approximately two-thirds of my rackspaces on each load, but not the same ones each time.

I’ve noticed that “On Rackspace” only gets called twice during load, only for rackspace 0, and only after all of the calls to “On Variation” have happened, so I can use that information to decide whether my “On Variation” needs to do any work or not: if “On Rackspace” has never been called then we must be in the middle of loading the gig, so “On Variation” can safely return without performing any tasks. Once “On Rackspace” has been called at least once, then I know the load is complete or nearly complete, so subsequent calls to “On Variation” will represent genuine rackspace/variation changes rather than being artifacts of loading the gig.

[Markus]

@Solomon

I’ve noticed that “On Rackspace” only gets called twice during load, only for rackspace 0, and only after all of the calls to “On Variation” have happened

Seems when no predictive loading is congured, than on startup all Rackspaces are loaded (and may be programatically also activated) once and when this is done then the first rackspace is again activated which is done by default on startup (just guessing). And since loading for sure means that each block in a rackspace is loaded, then all variations of this rackspace muss be activated before which causes the loading of all blocks in this variations.

Your idea to use the 2nd call to OnRackspace for rackspace 0 as “loading is done” event sounds good. But of course it would be more safe if GP once in future will provide a “OnGig” callback (or what ever it is named). And this should be called not only after loading is done on startup but also after other subsequent loadings of other gig files is done.

I edited this again: Just one more note, I’ve been looking through all the event handlers in indeed the event “gig is loaded” was somehow forgotten

No it has not been forgotten! As already recommended by dhj there is this GigLoded system event.
I really start loving GP scripting!

[David-san]

@Solomon

I’m curious as to why it isn’t called for every rackspace.

This is something to check, but probably because when you load a rackspace with only one variation, only the On Activation callback is called and not the On Variation. But, you can easily check if this assumption applied to your gig file…

@Solomon

on some occasions (which I can’t reliably replicate), GetCurrentRackspaceIndex() returns -1

This probably happens at loading time, when the Global Rackspace is initialized and no Local Rackspace is loaded.

@Solomon

sometimes when it returns 0, which is a valid rackspace index, GetRackspaceNameAtIndex nevertheless returns an empty string

It is perhaps that at loading time when the first rackspace is loading, the rackspace index is defined, but not everything is initialized.

If you can find a way to reproduce and give us the steps to do it, we could try to reproduce too.

@Solomon

In order to speed up the loading of my gig file, is there a way that I can detect that a given “On Variation” call is happening during load

It is always the first time is called, so you could detect using a initialized flag variable if it is the first time or not. But, I wouldn’t do that only if really really necessary. Don’t try to optimize the loading time by adding conditional statements which will be evaluated many many times and only be useful to you the first time.

@Solomon

I’m supposing that the sporadic errors are happening because the “On Variation” is invoked in one thread, whereas the rackspace state that GetCurrentRackspaceIndex and GetRackspaceNameAtIndex are relying on is being set up in a different thread.

My assumption was that it happens at loading/initialization time. Not the case for you?

@Solomon

My current best idea is to loop until GetCurrentRackspaceIndex() returns a nonnegative integer, and then loop until GetCurrentRackspaceIndex() returns a nonempty string.

Don’t do that! Avoid this kind of uncertain loop. But, you could perhaps only try it for testing purposes while debugging your GPScript.

[dhj]

FYI you can’t really tell when a rackspace is ready because plugins may be doing stuff without informing the host such as samplers loading samples asynchronously.

[Solomon]

@David-san

This is something to check, but probably because when you load a rackspace with only one variation, only the On Activation callback is called and not the On Variation. But, you can easily check if this assumption applied to your gig file…

Nope, it’s unpredictable. It’s different rackspaces each time I load the gig.

The vast majority of my rackspaces have only one variation, but the “On Variation” callback is invoked for most of them on gigfile load.

@Solomon

on some occasions (which I can’t reliably replicate), GetCurrentRackspaceIndex() returns -1

This probably happens at loading time, when the Global Rackspace is initialized and no Local Rackspace is loaded.

All of what I’m describing is at loading time, but the -1 only happens very rarely.

@Solomon

sometimes when it returns 0, which is a valid rackspace index, GetRackspaceNameAtIndex nevertheless returns an empty string

It is perhaps that at loading time when the first rackspace is loading, the rackspace index is defined, but not everything is initialized.

If you can find a way to reproduce and give us the steps to do it, we could try to reproduce too.

It’s not easy to reproduce because it happens only rarely, and apparently at random. This is why I think it’s a threads issue.

@Solomon

In order to speed up the loading of my gig file, is there a way that I can detect that a given “On Variation” call is happening during load

It is always the first time is called, so you could detect using a initialized flag variable if it is the first time or not. But, I wouldn’t do that only if really really necessary. Don’t try to optimize the loading time by adding conditional statements which will be evaluated many many times and only be useful to you the first time.

Yes, it’s always the first time it’s called for a given rackspace, but since it’s only an unpredictable subset of my rackspaces for which this happens, I wouldn’t want to ignore the first call for every rackspace. It’s different every time I load the gig.

@Solomon

I’m supposing that the sporadic errors are happening because the “On Variation” is invoked in one thread, whereas the rackspace state that GetCurrentRackspaceIndex and GetRackspaceNameAtIndex are relying on is being set up in a different thread.

My assumption was that it happens at loading/initialization time. Not the case for you?

Again, all of what I’m describing here happens at loading time. Yes. But the errors only happen rarely.

@Solomon

My current best idea is to loop until GetCurrentRackspaceIndex() returns a nonnegative integer, and then loop until GetCurrentRackspaceIndex() returns a nonempty string.

Don’t do that! Avoid this kind of uncertain loop. But, you could perhaps only try it for testing purposes while debugging your GPScript.

I use a loop counter to see how many attempts it takes. The vast majority of the time there is no error. Sometimes it works on the 2nd or 3rd attempt. One time it took about 5000 attempt before it got a nonnegative number, but even in that case it happens pretty fast.

Also, now that I’m ignoring all “On Variation” calls until the first “On Rackspace” call, the error hasn’t appeared, but I believe there’s still a chance it may happen during ordinary operation (as opposed to initialisation), so I’m still going to put the loop in for safety.

Solomon

[dhj]

This might be a silly question but is there a reason you can’t use the GigLoaded System callback?

[David-san]

@Solomon

The vast majority of my rackspaces have only one variation, but the “On Variation” callback is invoked for most of them on gigfile load.

How do you check this? E.g. not sure a Print function is the right solution for this at loading time when each rackspace has only a short period of time for initialization.

[Solomon]

@dhj

FYI you can’t really tell when a rackspace is ready because plugins may be doing stuff without informing the host such as samplers loading samples asynchronously.

That’s a good point. All I need from the new rackspace is that its widgets exist and have their correct initial values so that I can read them with BindExternalWidget, which in fact is the reason I’m trying to get the rackspace name in the first place.

[Solomon]

@dhj

This might be a silly question but is there a reason you can’t use the GigLoaded System callback?

Ooh, hey, that might help me a lot! Thank you for bringing it to my attention.

I don’t currently have a Gig Script, but I should be able to communicate from the Gig Script to the Global Rackspace script using external widgets.

[Solomon]

@David-san

@Solomon

The vast majority of my rackspaces have only one variation, but the “On Variation” callback is invoked for most of them on gigfile load.

How do you check this? E.g. not sure a Print function is the right solution for this at loading time when each rackspace has only a short period of time for initialization.

Hmm, good question. Are you suggesting that the On Variation callback may in fact get called for every rackspace, but for some of them which load especially fast, it gets aborted before my Print statement is reached?

[Solomon]

In order to isolate this behaviour, I’ve created a new gig from scratch. It has five rackspaces. I’ve thrown a bunch of widgets and plugins into each of them (not connected to anything) to make sure that each rackspace takes some time to load.

Here is the entire Global Rackspace script:

// Global Rackspace script

Initialization
  OpenLogWindow()
End

Function diagnostic(rackspaceIndex : Integer)
  var rackspaceName : String = ""
  var loopcount : Integer = 0

  While rackspaceIndex == -1 Do // wait for GP to finish initialising the rackspace
    rackspaceIndex = GetCurrentRackspaceIndex() // returns -1 if rackspace isn't ready yet
    loopcount = loopcount + 1
  End

  If loopcount > 1 Then
    Print("GetCurrentRackspaceIndex took " + loopcount + " tries.")
  End

  loopcount = 0
  While rackspaceName == "" Do // wait for GP to finish initialising the rackspace
    rackspaceName = GetRackspaceNameAtIndex(rackspaceIndex) // returns "" if rackspace isn't ready yet
    loopcount = loopcount + 1
  End

  If loopcount > 1 Then
    Print("GetRackspaceNameAtIndex took " + loopcount + " tries.")
  End

  Print("rackspaceIndex = " + rackspaceIndex + ", rackspaceName = '" + rackspaceName + "'")

End

On Variation(oldVariationIndex : integer, newVariationIndex : integer)
  Print("On Variation(" + oldVariationIndex + ", " + newVariationIndex + ")")
  diagnostic(-1)
End

On Rackspace(oldRackspaceIndex : Integer, newRackspaceIndex : Integer)
  Print("On Rackspace(" + oldRackspaceIndex + ", " + newRackspaceIndex + ")")
  diagnostic(newRackspaceIndex)
End

Here is some sample output on gigfile load:

On Variation(0, 0)
rackspaceIndex = 3, rackspaceName = 'rackspace_3'
On Variation(0, 0)
rackspaceIndex = 4, rackspaceName = 'rackspace_4'
On Variation(0, 0)
GetCurrentRackspaceIndex took 1066 tries.
rackspaceIndex = 0, rackspaceName = 'rackspace_0'
On Variation(0, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'
On Rackspace(-1, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'

Here’s another sample run:

On Variation(0, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'
On Variation(0, 0)
rackspaceIndex = 2, rackspaceName = 'rackspace_2'
On Variation(0, 0)
rackspaceIndex = 3, rackspaceName = 'rackspace_3'
On Variation(0, 0)
rackspaceIndex = 4, rackspaceName = 'rackspace_4'
On Variation(0, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'
On Variation(0, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'
On Rackspace(-1, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'

And here’s another run:

On Variation(0, 0)
rackspaceIndex = 2, rackspaceName = 'rackspace_2'
On Variation(0, 0)
rackspaceIndex = 3, rackspaceName = 'rackspace_3'
On Variation(0, 0)
rackspaceIndex = 4, rackspaceName = 'rackspace_4'
On Variation(0, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'
On Variation(0, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'
On Rackspace(-1, 0)
rackspaceIndex = 0, rackspaceName = 'rackspace_0'

I didn’t cherry-pick those results; that was just three consecutive loads of the gig file. As you can see, it appears that On Variation isn’t generally called for all five rackspaces, but for some subset. And in the first run you can see that it took more than a thousand attempts to get a valid rackspace index for one of them.

Here’s the entire gig file:
callback test.gig (707.4 KB)

Y’all are a bunch of great minds! I look forward to hearing your diagnoses.

Solomon

[David-san]

That’s what I suggested, but thinking about it it doesn’t make sense for the Global Rackspace. But it could be that the loading is sometimes so fast that the On Variation callback is not advertised from the Local Rackspace to the Global Rackspace because of any race conditions.

[dhj]

@Solomon

I look forward to hearing your diagnoses

I’m not going to get deep into this — what you’re trying to do is frankly way beyond the intent (and implementation) of GPScript. You’ve got loops that are going to totally fill the runtime scheduler buffers and you’re ignoring completely the effect of caching. There are things happening on different threads and the order in which things happen is non-deterministic and behavior is deliberately undefined. Some stuff doesn’t even get called while a gigfile is loading. Further, as enhancements are made to the system,

Basically, until GP stabilizes after everything is loaded, all bets are off!

[rank13]

@dhj

GigLoaded System callback

IMO you should definitely be making use of this. Myself and others normally use OSC as an easy mechanism to communicate between the Gig Script and Local Global rackspace scripts. You could also likely inject a midi message into the Local GP Port, and use the midi callbacks to pick this up in a global rackspace.

As you are getting more adventurous, you may also want to look into extensions, such as this one that @Frank1119 created:

An extension for storing global variables

The variables declared in gpscript are only visible in the script where they are declared, for example a rack-script, gig script, scriptlet or song-script. But sometimes it comes in handy to have some variables available everywhere. Therefore, I created this extension. It extends gpscript with some functions to created a ‘store’ with variables. These variables can be access by all scripts using these functions. Disclaimer: This is version 0.0.1, so use at your own risk. Feel free to clone it an…

[dhj]

@Solomon

I don’t currently have a Gig Script, but I should be able to communicate from the Gig Script to the Global Rackspace script using external widgets.

The GigScript doesn’t know anything about widgets.

What exactly are you trying to accomplish with all these callbacks?

[Frank1119]

@rank13

As you are getting more adventurous, you may also want to look into extensions, such as this one that @Frank1119 created

For 4.7, I have to recompile it. Tonight I can do that.

[Frank1119]

Tonight comes early today

GlobalVars-Ext.4.7.zip.sha256.txt (89 Bytes)
GlobalVars-Ext.4.7.zip (40.8 KB)

[Solomon]

@dhj

The GigScript doesn’t know anything about widgets.

What exactly are you trying to accomplish with all these callbacks?

During gig load I don’t need these callbacks to be called at all, but during normal operation my global Rackspace script needs to know when the user has switched to a different variation or rackspace so that it can read in a bunch of widget values from the new rackspace. I could use global parameters to achieve the same effect, but in v4.5 I didn’t think 128 widgets would be sufficient for my needs. Now in v4.7 I think 256 widgets might be plenty but it would require a major rewrite for me to start using global parameters at this point so for the time being I’m still using BindExternalWidget instead.