Not too sure it is the VRAM problem, There are games out there that look better and use less VRAM, GTA4 with a few mods looks as good as WD and doesn't use 3-4GB.
People must remember especially El Guvonator seeing as you don't really game a lot, Ubisoft are known for terrible optimizations or lack there of.
Take Assassins Creed 4 for example, The Physx patch for Nvidia cards is terrible, I've done testing while looking at a wall, A WALL, And turned Physx off and on, Off my FPS remained at 60, On they shot down to the 30's and 40's, Looking at a wall and it gets even worse once you venture out into the world not to mention the particle effects from smoke.
Ubisoft does not do optimizations at the level of other companies, If they did we would see WD use less VRAM, Maybe not a huge amount more but definitely a lot less.