Yeah, unless your server is going to be actively virus checking, defraging loads of drivers, handling emails or website junk - just get the cheapest dual core you can find.
If it's just a box sitting there and ticking over, even just handling torrent downloads, it isn't going to do much work at all. The biggest worker may be it's network ports.
Even in business, they outlay thousands on servers with big arse xeon cpus that hardly do anything. I've got an old Q6700 in one here and it's totally wasted.
If you got an old cpu laying around, I'd try that first and see how you get on. An E2xxx for all it's cheapness, is a decent cpu for an inactive box.