GTX 460, by far, or get a new mobo and another 5770.
What are the rest of his specs? That would help the decision much more, because he may just need to do some CPU overclocking to get some beefy performance gains. If I can live with an 8800 GT at 1920x1200, I'm pretty sure there are other places to look for performance.