It's pretty simple to explain why. Despite that amazing price-performance ratio of those cards, people still weren't buying AMD. Everyone was saying how hot and noisy it was, not how incredibly well it performed. Then the 970 and 980 came out and the world went crazy for them for having a whopping 0.1 kWh less electrical consumption (and later we find out no DirectX 12 hardware, no wonder it's more efficient). AMD dropped the price of the R9 290 down to £230 in retailers like Overclockers UK for the launch of the 970 and people couldn't care less, Nvidia was selling bucketloads of cards yet again at £350. AMD was the smartest choice all its history, but the average buyer is too dumb to realise that, or too unwilling to make a rational decision and needs to make a completely emotional decision of buying the top choice.
Now AMD isn't cheaper. People moan. AMD is still sold out in lots of places. Nvidia is still laughing all the way to the bank, and we gamers are left without competition in the consumer GPU area. Nvidia bled AMD dry of money for years and years, and still kept selling more even during fiascos like the GTX 5800 leaf blower, Bumpgate, the GTX 280 exorbitant price, the GTX 480 furnace, the very early discontinuation of drivers for older cards, and the 970 with 3.5 GB of VRAM in order to protect the price of the 980.
Will AMD ever be willing to compete on price against Nvidia? Possibly, but I doubt it. Nvidia has so much marketshare now that I don't think AMD can be competitive on price anymore, and will charge as much as Nvidia lets them. In the mean time, I'll continue to buy AMD when possible, and Vega is very much possible, especially if the current advantage in DX12 games carries on in future DX12 games (which I think it will, DX11 will be discontinued), and that means that even a Vega 54 is a better choice than a 1080.
People did buy the 290X over the 970/980 graphics cards though. If no one had any interest in the 290 or 290X, why did AMD release a successful revision of them? The 390 and 390X did reasonably well considering they consumed a huge amount of power. That suggests people who wanted the extra performance for higher resolutions and future DX12/Vulkan games, but without paying for Fiji or Maxwell, the 390 and 390X were there for them and were appreciated.
To me, the smartest choice isn't as categorical as you are suggesting. Some people look to out of the box performance, and Nvidia offered that with Maxwell and Pascal. Others prefer huge overclocking headroom, and again Maxwell and Pascal offered that in spades. Some people want really low TDP parts for their mini systems. Loads of people bought 970's for their living rooms so they would have quiet builds that didn't look out of place beside their 50" TVs. The Fury Nano was also a good card for that purpose, but it was too expensive for the majority, and quite a bit later in the game.
On the flip side, some people wanted better future proofing and higher resolution support. AMD offered that. They also offered better value for money at the midrange, and back in 2013 at the high-end too. Some enthusiasts also wanted to support AMD because they were usually at the forefront of technology, despite their limitations and drawbacks. I invested in HBM with Fiji despite its drawbacks because it was an exciting evolution of memory technology. The 290X had an extra gigabyte of VRAM than the 780Ti, and that helped hugely in the slew of 2014 and 2015 games which often drew 3.3GB of VRAM. But now with GDDR5X, the gain AMD had with HBM at higher resolutions is somewhat mitigated. In fact, the 1080Ti offers almost as much bandwidth as Vega.
My purchase of the GTX 1080 was both emotional and rational. I looked at what I needed in a GPU—low temperatures, good overclocking headroom, low noise, and good performance-per-dollar in the games I'm playing—and the superior GPU was the 1080. But when I bought my Fury two years ago, it was the more rational choice as it was clearly better than a 980 even in DX11 games, both back then and most certainly now. It was more expensive, hotter, and offered appalling overclocking, but it didn't matter as it was still a faster GPU. My emotional side encouraged me to buy Fury because I fancied a change, did not like Nvidia's practises, and wanted to support the underdog. Or more accurately, that was my 'wise' side.
The brain could be said to work in three ways: rational, emotional, and wise. Rational ignores emotions and makes decisions solely on objective analysis. Emotional is making decisions without thinking about repercussions. Wise decision making is a mixture of the two. It's where you allow your emotional side to play a part, but you don't let it take over rational thinking and cause mistakes. Both my purchase of my Fury and my subsequent purchase of the 1080 were what I consider to be wise choices—minus Freesync which ended up being a gamble that didn't quite play out. I buy graphics cards at least every two years, ideally every 18 months, and air cool them in a quiet environment. AMD is beginning to fail in that regard. I also really enjoy overclocking, and while Vega looks to be a lot of fun to overclock, it needs water. With Pascal water is not necessary, even for the 1080Ti. It helps, but it's not necessary.
As to your point regarding competition, even if it wasn't AMD's preferred choice to make a beast of a card that was to cater to both compute workloads, gamers, and miners, Vega 64 struggles to compete in the market I'm in, and now it's likely that Vega 56 is going to struggle to compete once the 1070Ti comes out. While I have always said that if people want competition in the market they have to actually support the competition—that's one of the biggest reasons why I bought a Fury. But that principle has limitations. I'm not just going to buy a weaker product just because we need competition. I'm a consumer and passionate about hardware, but I'm also a gamer and have particular requirements. If Vega came out with a 200W TDP, I would have bought it. But it's a 300W card so all its drawbacks become too much to swallow.
I personally can't see Vega 56 surpassing a 1080 in the majority of games for at least six months. And even if it does, I won't be playing those games for a couple of years because I wait for games to drop in price before buying. I'm currently playing two-year-old games (Mad Max and Arkham Knight). Two years from now I'll be playing the new Wolfenstein game which is likely to perform very well on Vega. Of course, by then Navi will be out and I could be on it. As will Volta which will have hardware support for DX12 and Vulkan. That's what I mean by the smartest choice not being so definitive. It's different for every consumer.
Sorry for the wall of text. I had a lot to say.
