Depends on the workload basically, you need a >98% parallel workload to really make use of a GPU and even then if it's non-uniform or branchy like compilation or simulation a CPU would usually do it better. Modern CPUs have lots of specialised units for certain types of calculation that GPUs don't necessarily have and vice versa.
For 96-98% parallel workloads you can still get a good 30-60% speed up going from 32 cores to 64-cores. For 95% parallel going from 32c to 64c gives upto a 25% speed-up, going up to 4096/infinite cores you'd only get a 60% speed up from 32c but often with GPUs those cores are several times slower per "core" so it's not worthwhile when your workload is still this serial(5%).