EDIT: Nope, I lied. It seems that if I do anything with the toggle, it seems to have an immediate noticeable effect on input latency (regardless of setting) until I restart the game.
But one more thing: I've also noticed that osu! tends to ignore my AMD driver's "flip queue size" setting (aka "max pre-rendered frames" on Nvidia) unless I set the in-game frame limiter to Vsync and then force-disable Vsync through the GPU. The game seems to run at the default of 3, so I'm forced to use the Vsync workaround in order to use anything lower. This behaviour does not occur when using the DX renderer in the fallback build.
(This setting also affects input latency, so perhaps forcing a pre-render queue of 1 if possible would be better for everyone)
EDIT: Flip queue size also seems to work properly in compatibility mode.