Hot Backup (Part 2: The Software Side)

While we discussed the decisions to go with a software-only hot backup here, there are some requirements and limitations that I’ve experienced.

Synchronizing the gig files
I want to handle only one Gig file for the main and the backup machine so the most recent version needs to be on both computers. While the were some problems with MainStage and cloud solutions (because it stores the settings in container files), it works perfectly with Gig Performer.
I installed the default sample libraries on both machines and song-specific samples live in the Gig Performer Folder that I completely moved to Dropbox.
I don’t use iCloud for this because you cannot select individual folders to synchronize, and also with Google Drive this only works with the streaming mode (I believe). The 3-machine limit of the free Dropbox version perfectly matches the one of Gig Performer and many plugins and it’s very fast.

Master Switch
If both machines are constantly running and you don’t switch them externally, I need to be able to turn the MIDI, audio and OSC output of both machines on and off within Gig Performer. This is pretty straightforward with muting audio mixers and MIDI outputs with a master switch in the global rackspace panel. For OSC, it’s a little bit more complicated, but a script is used to track the status of this master switch and it can determine whether the „send osc“ functions in the global rackspace should output OSC or not.

Tricks for old hardware
My main machine is a MacBook (M3 Pro) from 2024. The backup however is a rather old Mac mini (late 2012) that is controlled via screen sharing. This difference in power means that I need to make some compromises concerning the CPU load. The worst thing would be to limit my demands so everything runs well even on the old computer. This didn’t feel right, though, so I looked for ways to maximize the possibilities.
First, I adjusted the buffer sizes. 64 samples work well on the new MacBook and feel really good. On the old Mac mini 384 samples are just the maximum that seems playable in the rare case of an emergency. Since also the mic is routed through Gig Performer the latency is even doubled for the entire roundtrip.
But still this wasn’t enough to play the gig file without any glitches so I decided the backup should automatically use lighter chorus, reverb and dynamics plugins in the global rackspace. This lowers the load on the CPU in a way that also the backup machine plays the gig file as well as possible.

Automatic identification of the machines
The switching of plugins, however, requires Gig Performer to know which machine it’s running on. I discovered that there are some variables that are not stored in the gig file but within the instance that I can refer to from a script. So I chose the „OSC targets“ to give the different machines unique names.
Bildschirmfoto 2024-08-10 um 12.09.50
If the global rackspace script discovers the name of the backup machine in the OSC target list there, it can bypass the heavy global effects and unbypass the lighter ones by using the following callback:

If OSC_TargetNameExists("Backup-Ident") then

With this knowledge the master switch of one machine can automatically deactivate this switch on the other machine, too.

Setlist Changes
Gig Performer can now automatically sync which songs is active on multiple machines, but I implemented some advanced functionality when this hadn’t existed yet:
Before each gig I synchronize both machines via Dropbox and a hotspot from my phone. After that I turn WIFI off. Sometimes there are last-minute changes to the setlist after synchronization, but scripting helps me here:

The master machine always sends out the current song name. If this song name exists in the current setlist on the backup machine, it switches to it immediately. If it cannot be found there, the backup machine switches to the „all songs“ setlist and looks for the song there.

Here’s the code:

On OSCMessageReceived(m : OSCMessage) Matching "/CurrentSongName"
    var SongSwitch : boolean = false
    If Master == false then // If the machine is currently the backup
        ReceivedSongName = OSC_GetArgAsString(m,0) // Extract and stored the song name from the message
        If InSetlistView() then
            If GetSongIndex(ReceivedSongName) < 0 then // If the stored song name does't exist in the current setlist
                SwitchToSetlistByIndex(0) // switch to "all songs"
            End
            SongSwitch = SwitchToSongByIndex(GetSongIndex(ReceivedSongName),0) // then switch to the song with the stored name there
            End
        End
    End
End

So have I needed the backup yet? Well, there’s more to this question. More on that later… :metal:

5 Likes

Did I ever need a backup?
Coming back to the question from the beginning: Have I ever needed a hot backup? Well, yes, but not for reasons I designed the system for!

My plan was to have backup for everything that is software, but Gig Performer itself has never crashed so far during a gig. Not even the plugins behaved bad.

Once, however, my USB hub lost the communication to the MIDI interface which I couldn’t figure out at that time. I was glad I could instantly switch over to the backup system that uses network MIDI (and network audio) instead of USB.

Limitations
I think it’s important to know the limitations of this backup approach:

  • A hot backup running the same gig file does work well for the „no sound“ situations that I still know from MainStage: suddenly there is no sound without having done anything.

  • It also works, of course, should Gig Performer or plugins randomly crash.

  • In the past, I had one very loud noise created by a locked up sound card or driver (on Windows XP with Forte and VB3 version 1). In this case, I would have to pull out cables or hard reset the Computer since I do all the muting and unmuting of the main and backup system only within Gig Performer.

But the most common reason something went wrong in my personal experience with Gig Performer, however, was some of my scripts being programmed too quickly and not tested enough.
In these cases, of course, a hot backup doesn’t help since it’s running the exact same rackspace!

3 Likes