I have evidence this is related to CPU throttling. I disabled an option related to CPU power savings in BIOS so the CPU core frequency stays at max all the time. I've since played a long time, tried a restart and no framerate drops anymore.
The only downside is that the CPU runs slightly hotter (43 vs. 41C) at idle.