I have an old pioneer vintage amplifier that all my audio connects through. The speakers are wired into the amp, and then my record player and my computer each hook up to the amp on different channels. I just flip a switch when I want to use one or the other.
My headset plugs directly into the amp so I can listen directly to whichever output I'm set to. I just have to remember to switch off the speakers when I have my headset on.
The drawback is that I can't have my computer audio and my record player with this aetup, but that's honestly never been an issue for me.
I thought about just using an amplifier but that's not really helping, because I also need my headset for talking with other people ^^