I like the time graph of the FPS but as you say it doesn't tell the whole story. The problem is that people think that they know what standard deviation means, but don't really (a measure of variance/spread from the mean/average/normal as you mentioned). A normal distribution graph showing the deviations left and right to the average would be useful. A tall graph would indicate bad, a flatter smoother graph would indicate better and no deviances with a low mean would indicate ideal (am I right in thinking?). Would the average user understand it or be better enlightened by it?
Hmm, I've had a think about it but I reckon it would be too hard and long to calculate and display. I think you'd have to offset the expected average frametime from the actual frametime. That's easy if you are getting a solid 60fps throughout (you just use 16.67ms as the expected average and compare it to the actual frametime) but when you aren't getting 60FPS you need to lineup the frametimes against the right second (to get the right FPS and therefore the correct expected frametime for the period). That is impossible without a very specific bit of software! It suffers from the same issue as the percentile graph in my eyes:
This is the kind of thing you can get from a percentile graph, Excel heads on people, this may hurt:
What you are looking at is the amount of milliseconds it takes for a certain percentage of the render times to fall within. Let me put it another way:
1. Look at the 50% mark on the Red, Xfire line. It means that 50% of the frames are rendered in 9ms or less with the other 50% of the frames taking longer than 9ms to render.
2. Compare that to the green 50% mark and we see that it takes nearly 30ms to capture that 50% with a single card.
What does that mean? Well by itself not very much - it basically says that the xfire setup renders more frames in less time. It says nothing that FPS says more easily. It certainly doesn't add to the data which is trying to define the smoothness of the framerate. I mean; it should be pretty obvious that the XFire setup is rendering frames in less time.
It does say other stuff though, like efficiency of adding a second card. The greatest difference between the lines is between the 20-50% marks. In that 30% range adding a second card is having more benefit than at any other point. It's also a total generalisation, it doesn't tell you which sections of the bench benefitted the most (if you were interested in that).
In order to make it work in terms of smoothness you'd have to control the FPS. Have no fear folks M&P has done that for you as well because I know that you are all desperate for more excel graphs lol
So in this experiment I wanted to see whether using Vsync or a Framerate Limiter was better for smoother gameplay. This is actually a genuine question I've had for a while so a genuine test. Subjectively I prefer Vsync, no screen tearing is an obvious improvement but there was always something else which made it seem smoother. Could have been just me though. So Heaven 4.0 maxed out at the ready with FRAPS recording for 60s again. Here we go:
1. FPS stuff:
Funny old thing, very little difference. Not much use for me here then.
2. Let's go straight for it. Standard Deviation:
Oh right, Vsync is actually producing a much smoother flow of frames even at almost identical FPS. It's not a small difference either with the Limiter producing nearly a 20% greater average range of frame times.
3. Lets look closer. Given that we are getting a solid 60FPS at times during this experiment we should be seeing some pretty straight lines with every frame rendering in 16.666667ms (1 second / 60):
Ah, sometimes I am getting what I expected but for a lot of it the rendering times are all over the place. That's not really a surprise because different frames are more or less difficult to make so there should be some variation.
Although there is lots to look at here there is one bit that particularly interests me. The Cloud of blue on the left where frame times for the Limiter condition are all over the place yet the Vsync seems to only be surging upwards.
4. I'll pull out frames 240-300 which should be roughly 4-5 seconds into the bench. Looking at the FPS timeline I can see that this matches up with a period where the GPUs were working flat out but still not quite making 60fps. Similarly the second cloud of blue matches up with a second period where the GPUs can't maintain 60FPS. I see a pattern...anyway to the close up.
The FPS Limiter condition is what we saw during the first experiment. The GPUs are banging out the frames as fast as they can make them. Sadly that makes it look like the buffering is going to hell. It's spitting out one frame really quickly then not being half ready with the next and taking ages to get that one out by which point the next frame is nearly buffered and ready to go. Fast, slow, fast, slow etc.
The Vsync condition however is much more controlled. The refresh rate seems to be stopping the GPUs from spitting out the frames too quickly which is allowing more time for the next frame to be rendered so we are avoiding that uneven pendulum effect. In fact a large number of the frames are being rendered in a perfect 16.67ms which is exactly what I would like to be happening. The times that are above that butter magic figure will be the frames which are more difficult to render and which are dragging down the FPS.
5. Lastly, now that I have comparable FPS scores I can use the percentile formula to take a look.
So for both conditions I can see that roughly 90% of the frames are rendered in a near perfect 16.67ms, although Vsync does a better job all across that data range (remember that rendering too fast is nearly as bad as too slow). That last 9% of data is crucial in indicating here that when the GPUs need to bang it out, they bang it out more smoothly with Vsync on.
So all that analysis is possible to achieve with some effort and it adds some potentially interesting stuff but at the end of the day STDEV does it all much easier in my humble opinion.
For reference the earliest use of Frametimes I've seen in a review
was this one back in 2008 on Lostcircuits but they went back to FPS for simplicity.
That's all for now folks. Enjoy
