It isn't a quadfire issue, its an eyefinity issue I'm pretty sure. Catalyst looks for all the screens connected to a pc, finds three on an eyefinity capable card, looks for its crossfire counterpart and then finds another one on the Nvidia card, tries to crossfire that with the non existent screen. It is a pretty rare occurence to have eyefinity, Physx and crossfirex.
I have had it working with either one 5970 and eyefinity and physx or two 5970s, physx but no eyefinity.