Definitely sounds like the noise.
It's a kinda welcome to one of my setups that produces, hdd, cpu, gfx noise all in one !
I'm bemused as to why the usb headphones are giving this, there are many roads to try and solve this. U can get it on regular phono headphones if the cable is too close to something strong like an iec cable, that wouldn't explain the mouse actions tho. A monitor/dvi/vga cable could do it I guess if it was an el-cheapo thin one (skimp on the shielding within).
Mostly tho, I tend to think that the sound device, the pci(e) card or onboard is the answer. The latency thing used to help with screeching creative soundblaster cards, sometimes, but as with myself, I'm looking at the gfxcard influencing the soundcard. I'm trying to think of some kind of shielded hood I can put over mine.