Storing/Retrieving Extension Data

EDIT: Moving to a new discussion topic.

Yes it would. But I don’t have a clear answer as to where the data should be stored. It will always be a bit of a hack to manage an external file or find some other workaround to store the data in the gig file.

Hmm, should you be able to query the default location for settings or should there be a standard way to define locations for individual extensions, perhaps a library specific folder under the GP Settings folder?

It might even make sense to expose dialogs for selecting files or folders?

2 Likes

I think my main concern is when the extension data is rackspace or song specific, then there is a greater chance of losing the link to the source rackspace/song e.g. they get renamed and re-ordered.

I don’t know if you’d want to extend the gig file data structure to allow extension specific attributes to be stored against specific rackspaces and songs? Too risky?

1 Like

Actually, that’s a great idea….requires it to be thought through carefully but I can imagine being able to invoke calls to extensions to request an XML representation of whatever you want saved/associated with specific rackspaces, or even with songs and song parts which would then be saved either directly with the gigfile or perhaps with an auxiliary file tied to the gigfile to avoid the gigfile getting too large,

4 Likes

I would second that request. I would love to be able to store structured JSON or XML data in the Gig file to store extension settings per song or rackspace and globally per gig file.

2 Likes

It would also make it simpler to save global rackspace data in reach local rackspace separately.

Currently I’m using external file, one per local rackspace it would be much more elegant to save this in the gig file itself.

@dhj in the short term, what do you think about adding:

  • A new status type for the OnGigStatusChanged callback for GPStatus_GigFileSaved. This way I could synchronise saving extension data when the user saves the gig file.
  • New functions to get the internal Song UUID and Rackspace UUID. I would feel much more confident managing data in an external file if I had the internal uuid, rather than relying on the song/rackspace names.
1 Like

I could but wouldn’t it be better if we used a mechanism where I could request an XML representation from the extension which I would then save and then I could feed that back to you when the gigfile is opened? It’s really easy to create that stuff with the XmlElement class in JUCE

Or perhaps an extension “state”, pretty much the same as we get for plugins? That way I would just get and send back a blob of data and it would be up to the extension to interpret it? Then we’re not dependent on JUCE and people could use either approach?

Sure

Ok, I’ve just figured out the conversions from file to XmlDocument to ValueTree for the latest extension I’m working on, so this sounds good.

If ‘state’ means binary data, then could that be an option, rather than the only method? For the type of data I want to store, I do like that it’s human readable, so XML seems better.

I can’t guarantee that the gigfie will always be human readable – at some point we may compress the whole thing to reduce the size - but we can certainly just use XML as the underlying approach.

But whatever you use, you need to pass a string representation of your XML — you can’t pass a pointer to an XML document.

So the functions would be something like

std::string getXML(); // Ask the extension to return XML text
void setXML(std::string xml); // Send xml string to the extension

Think about including attributes such as the version so you can deal with backwards compatibility, etc

2 Likes

Maybe two functions can be offered to a covert a binary array to a string and vice versa. Maybe floats array should be supported as many widget values are floats

I’m confused - what does this have to do with saving/restoring extension-specific data?
We are not talking about GP Script

The “extension state” option to get and set XML would work for my purposes.

@rank13 Be very careful with this — if you create a “document” this way, make sure that

  1. the first parameter of createDocument is an empty string (i.e, no DTD) and
  2. make sure that the third parameter (includeXmlHeader) is set to false

I’m suspecting that the function call into the extension to get the XML will have to have a bool argument to indicate whether the XML was successfully parsed

I thought it was GPScript related.

1 Like

This is the “Developing GP Extensions” sub category :slight_smile:

1 Like

I noticed indeed… Later this year i will check out more about extensions

1 Like

I like this idea of having storage available for extensions. Let me just throw out a couple thoughts as people consider what makes the most sense…

I’ve written a couple different control surface extensions, and there are a couple of broad types of data that they need to store:

  • global configuration data - e.g., which midi device to connect to, layout information (e.g. is your MCU an x-touch, an Icon M, a Presonus FaderPort), other user preference information

  • rackspace/variation specific data - e.g., for widgets grouped in banks, which banks should be active on the control surface when the variation/rackspace is engaged

  • widget specific data - e.g., what color should the knob appear on the control surface, what RGB on/off color for individual buttons, how should the controls be grouped for bank switching, etc.

The “global” stuff seems suited to this XML stream approach, but I’m not sure it works for the Rackspace and widget specific stuff.

As it is now, I store all of this information in text widgets. Nice things about this are that it’s easy for users to access, edit, and troubleshoot, you can hide the configuration widgets, it’s all automatically copied when you duplicate a rackspace, and it’s all automatically gone when you delete a rackspace.

The two downsides are that you have to create a lot of extra widgets, and I end up using things like widget fill color, outline color, and caption to cram information in that isn’t intuitively obvious. e.g., for RGB buttons I use fill color for “button on” color, outline color for the “off” color.

In my long-term dream for extension data storage I’d be able to link those directly to the “thing” they should relate to. e.g., for button on and off colors it would be great if I could attach “tags” to button widgets something like “SLMK3 off color: #454500”. I’d really love it if those would appear on the “Advanced” tab when editing widgets, or perhaps a new tab called “Tags”. It could just be an XML text box where users could edit things, or maybe just force it to “tag” and “value” pairs so users don’t accidentally butcher XML formats.

I imagine something similar could be done at the Rackspace or Song level if appropriate.

While it’s certainly possible to stick all this information into one global XML stream I think it would be cumbersome and error prone for the extensions to handle it that way. e.g., the extensions typically only “see” the current rackspace, so when it builds a widgetlist and configuration data for them it would have to store them in the XML using those widget names. But other rackspaces may use those same widget names, so the extension would need to store data using both the rackspace name and the widgetname. But now suppose you change the name of a rackspace while the extension isn’t active? When the extension is active again it’s looking for the data under the old rackspace name and it’s not there, so all your configuration data is effectively lost. I regularly share networks stored gigs across three machines.

There is also an unaddressed “need” right now to have a way for multiple extensions to share control access to the same widgets. e.g., suppose I wanted to use the Radio Buttons extension while using an MCU control surface extension? Both extensions require the widgets to be named a certain way (using the OSC/GPScript name) to be able to use them. Widgets can’t have two names, so only one can access them at a time.

If we were able to create and store “tags” on a per widget basis then multiple extensions could share the same widgets.

I like the idea of an XML stream for storing “global” data, but just want to note that it’s probably has some limitations. To the extent the ambitions for extensions are ambitious, it may be worth thinking about whether that should guide the initial steps.

3 Likes

One immediate issue that comes to mind with attaching tags to the internal gig file XML nodes is addressing what happens if the extension should go away — those nodes will end up with a lot of useless tags in them.

Interesting idea except we need to think carefully about how end users who are not developers/programmers/XML experts should manage this stuff

If we have rackspace/song specific storage, I wonder if it should be a ‘push’ from the extension, rather than a request that comes from GP? So the extension stores as required on an ongoing basis, rather than a single event.

I’m thinking about the scenario of GP only having a trigger like ‘rackspace deactivation’, and so I wonder if there’s a risk of impacting rackspace switching if there are actions to request/store from the extension? But maybe this is an unlikely impact?

This is an interesting dilemma. But there will be risks for multiple extensions controlling the same widget. Particularly if they involve setting the widget value. I’ve already seen issues with using the radio button extension with GP script that was also operating on the same widgets.