Yes, latency is calculated even for bypassed plugins.
Whenever a latency is changed then the internal audio graph is recalculated, and audio buffers memory are allocated according to the latency size.
So, in order to support bypass/un-bypass during runtime then the latency needs to be taken into account even during bypass.
Furthermore, changing the latency of mixed chains during runtime might cause audio artifacts which is also problematic.