You very eloquently and accurately described the experience, yes, that’s pretty much how it works.
However, there’s an important difference when we’re talking about amp simulations. It’s aliasing. Aliasing is never pleasant or ok, and it’s an exclusively digital artifact, it doesn’t exist in the world of analog gear, be it amps or pedals or what not. And it’s not possible to get rid of it completely, you can only decrease it to levels where it is irrelevant, but the techniques to do it aren’t super easy, and they consume a lot of CPU (and in the case of ML modeling they are tricky to do, as far as I understand).
Aliasing is basically a kind of noise, and it’s not easy (or even possible) to point your finger to it and say, here, notice that thing at 4K, it doesn’t work like that. It’s just some unpleasant crap in the sound. When it’s in a mix that someone else did, it’s harder to notice, unless it’s at some insane levels. Cheap processors like Digitech RP, Boss GT, plugins like TH-U or Guitar Rig will have a ton of it, and there’s a reason - hardware manufacturers have hard CPU limits, high end DSP chips are expensive, plugin developers also need to worry about users with lower end computers, etc. Most ML based profilers have a lot of aliasing, too, for reasons I don’t quite understand (it’s some complex math), and there oversampling isn’t the solution - but here again, you need tons of CPU power to run more complex models. NDSP Cortex was very bad at least initially when I tried it, I hear they improved it though.
And that stuff is objective, i.e. aliasing can be measured, albeit not very easily, and it’s always bad. So there are bad and good simulations.
You can hide the unpleasant stuff in a well crafted mix, which is what a lot of people are doing when showcasing plugins in Youtube videos. Good old frequency masking works not only to make an instrument be heard well in a mix, it works the other way, too. So yes, you can sort of get a good result with almost anything. Most often people hide it by applying a lot of low pass (and a generous amount of high pass) filtering, so you get that muffled sound like a speaker is under a blanket or something. You may not notice a problem because low passed sound can fit in a mix well, but plugin makers know it (and certainly Mikko from ML sound knows it, he’s been capturing amps for years for a living), so when you hear that kind of recording technique from them you immediately know they’re hiding stuff. You can still produce a good sounding mix with that though.
However, there are two problems that aren’t just a matter of preference.
First, when you play yourself, your mind focuses a lot more on your sound, and picks even smallest nuances which you otherwise would miss, and we get to the situation you described so well. It is very annoying and unsettling.
Second, if there’s a lot of this unneeded crap in the sound and you filter it all out, then at some point you aren’t left with much, and here we get to the problem of using plugins live. When plugin developers make the factory presets, they target a typical bedroom player who will most likely listen to the sound in headphones and in isolation. So factory presets are typically very wideband, with wide stereo effects, very full. That of course never works in a live mix. So you start adjusting the sound, and there comes the issue - as you narrow down the spectrum to what you need, it starts sounding weird, doesn’t work in a lot of rooms, you either get lost or stick out, etc. To a degree, some adjustments are always needed with any sound, but somehow there’s a huge difference between amp sims in this regard. The video I posted above isn’t meant to show my sound crafting abilities - I have very little - but the fact that you can hear the guitar well, and that’s a first song in a venue where I never played before, we’re playing after another band, and the FOH engineer doesn’t know or care what it is we’re playing. And that’s the thing with good equipment - when I started using Axe-FX, it became a set once and forget thing (and the audience, even people who don’t know anything about sound, noticed the change immediately). It’s not the case with plugins at all for me. Something is off with most of them.
So yes, there are good and bad plugins, and no, it’s not all a matter of preference. There are real objective problems like aliasing, but even with more subjective stuff it’s sometimes possible to see where things go wrong - i.e. in the samples of mixes I posted above I can point to mistakes in how I did it, like I shouldn’t have used presets meant for live in a recording mix.