Agreed. I was going to make that same comment regarding the stream XML approach. “Garbage” collection seems like a tricky issue no matter how it’s done. (Ask the Windows registry about that.)
Perhaps a requirement like every extension must report an “identity” to GP to be loaded, and when GP writes anything on the extension’s behalf to the Gig file that identity is part of the XML. Then maybe on the “Options->Extensions” GP menu there’s a “Cleanup” option. That option could open a checkbox list of all the extension identifiers it sees in the Gigfile and the user can check the ones they want to delete.
Absolutely. If it were presented to users in the style of “Key” and “Value” pairs it should be hard for them to break anything. But that would put the burden on GP to implement it.
That’s clearly a risk for any extension that does one or more “set widget value” commands in response to a “widget value changed” callback. After some sporadic feedback loops in my early control surface efforts I decided “don’t do that.”
But for something like Radio Buttons, changing widgets in response to other widgets is the whole point. You can get into unexpected feedback loops even when only one script/extension is doing it, let alone two or more.
Two potential ideas for that are A) the standard “use at your own risk” (which sort of applies to all scripting and extensions anyway) or B) some kind of feedback detection in GP that will shut down an extension/script that’s “too active”. I’m not sure how to define a threshold or how to monitor it, but if it generates more than X callbacks in Y second it gets disabled.
I imagine the extension and scripting happens on separate threads from the audio, so some really crude monitoring of activity at the interface level between GP and the extensions might not impact audio performance at all.
I’ve been waiting for 4.7 to get out the door before starting up this discussion again as I’d dearly love the ability to store data in the gig file for my extensions.
Having re-read the thread and noting how @Vindes is storing data in text widgets. Given the desire to not cause issues with the integrity of the overall gig file I had a thought that might warrant some discussion.
What if we had a new widget whose only purpose was to store data, be that a blob, XML, text or whatever (…to be thrashed out), we could then programmatically read and write to and from these widgets and optionally programmatically create them if we need to ensure they exist should our extension require it. GP, I would assume, would only need to handle a new widget rather than a new sub-system to integrate into wider gig file handling.
My preference would be that it contains text, ideally editable in the Widget Properties panel into which we could store a serialized version of our chosen format (XML, JSON, raw text etc.) meaning we can use the format that makes most sense to each developer’s particular scenario. Someone using JUCE may want to leverage the XML capabilities whereas I would find JSON more useful given my usage of HTML & JavaScript in my extensions. Storing serialized data would keep options open to suit all developers’ needs - hopefully!
The only scenario this wouldn’t cover would be storing different data for Songs and Song Parts as the widgets are within rackspaces. So this will need some brain power to find a way to fit in with the storage widget idea.
I would vote for a more formal solution. There is a challenge with handling the variety of scenarios for where you might want to store data:
Global settings for your extension
Per-song data
Per-rackspace data
I’m also imagining being able to globally clear any stored extension data via the GP Options > Extension window. So if a user decides to disable/remove an extension, it’s straight forward for them to clear any data. Although this would only work for the active gig file….
Would/should this be formally linked to the Extension? e.g. internally the Extension identifier is linked to the stored data, so only it could query data that it had stored?
@rank13 has already suggested a menu item/button for a user to remove the data for an extension if they remove it. Could there also be an automatic check that would see if the gig file contains a dictionary for an extension that doesn’t exist upon opening or saving a gig file?
The problem with that is that you might use the extension sometimes so you don’t necessarily want to remove the data.
But I’m actually less concerned about removing all the data associated with a particular extension – the user can be provided with a list of what’s in the gig file.
By bloat, I was concerned about individual extensions putting more and more stuff, perhaps even temporarily stuff, and never getting rid of it.
I see what you mean. I would suggest setting limits or quotas, but the problem with that is someone always needs a bigger limit. Are there any technical size limitations for a gig file?
I’m not aware of any limits but I’d like to not discover limits unexpectedly
The other way to do this is to manage the dictionary but instead of embedding in the actual .gig file, we could have another file with the same name but a different extension and that would hold all the info. The only downside of course is that the user now has to manage two files
I am not too concerned about that, since plugins get to store data there too and even though the plugin developer landscape is huge and diverse, it works (mostly, at least). One difference between most plugin interfaces and the proposed dictionary API is that plugin instances get only one state, right? So that lowers the chance of “forgetting” a previously set key since any time a plugin is asked to save its state, it can just collect all of its current state, without needing to take care of what other bits of state (e.g. any other k/v pairs) it might have saved earlier.
So I’m proposing a very simple approach, which is: As an extension, you get to write (and read) one single blob of binary data which is stored in the Gig file like so:
int GP_StateStore(const uint8_t* data, int length); // called by an extension
int GP_StateTriggerRecall(); // called by an extension - you only get an error code, but no data
void OnStateRecall(const uint8_t* data, int length) { } // callback during which the extension can read the data buffer
Note that this design avoids the need for extensions to allocate their own buffer for recalls (since they might not know the needed size) as well as the need for a retain/release mechanism by using a callback for accessing the GP-allocated recall buffer.
This keeps the interface between extension and GP very small but still flexible, e.g. extensions are still able to overlay their own dictionary mechanism if they so desire. But since Gig Performer itself is not tied to the structure of the data, the probability of breaking changes between SDK ↔ extension and SDK ↔ GP interfaces is lowered.
Concerning the pile-up of data from unused extensions, I can think of a few approaches:
Have a dialog in Gig Performer where users can…
…inspect which extensions have stored data in their current Gig (and how much)
…delete data per extension
That would basically be the equivalent of a “manage website data” pane in a browser. As a user, you don’t need to know it’s there, but if you want to, you can inspect it.
Prompt users if it seems like their Gig is cluttered up with extension data, e.g. there is more than x kilobyte of data stored from this extension and the last n times the gig was saved, the extension was not active.
A plugin only exists in a rackspace. However an extension could be operating across any aspect of GP. For example, if the extension wanted to store data for a song, it could record the song uuid. However, the extension currently can’t identify that the song was duplicated.
However, if the extension data was stored with the song, it would also be duplicated.
But, particularly as a starting point, having a single blob per extension sounds like a good approach.
I would be a bit concerned to have a single blob to hold all the extension data. As @rank13 suggests, as users edit the gig file an extension’s code would need to hook into those processes and to maintain our blob of data so that it continues to match the whole gig file.
I think the dictionary idea seems the best option for most use cases as using song/rackspace/part and variation uuids as keys into the dictionary offers a very simple interface to the extension developer, simplifies the maintenance of the extension’s data and unifies the dictionary mechanism for all.
Going the route that is a little simpler and fool-proof for the extension developer would hopefully encourage more people to get into extending GP.
I agree with you that extension developers should have access to a convenient interface to store their data So I don’t think anyone should have to implement a custom binary-serializable hashmap in C, just to store per-song data in their extension. However, I do not think that Gig Performer itself should be involved in how an extension structures its data. In my opinion, this is the right place to use a third-party library, as for example JUCE’s ValueTree.
I feel David’s suggestion for Store() and Recall() functions to access a dictionary are much, much more obvious to use. You could use the rackspace/song UUID, optionally with a suffix, as the key and you can serialize whatever you want to be the value. Your code then only needs to be focussed on the data for that rackspace/variation and you don’t need to implement your own code to manage the larger structure, be that a JUCE ValueTree or something else.
For those developers who only need to store something very simple for each rackspace/variation (e.g. a colour code or a title) then they don’t even need to serialize anything, they can just store a simple string. The overall concept just seems much simpler, cleaner and easier to understand.
So I don’t think anyone should have to implement a custom binary-serializable hashmap in C, just to store per-song data in their extension. However, I do not think that Gig Performer itself should be involved in how an extension structures its data. In my opinion, this is the right place to use a third-party library, as for example