Posted by: felipe | February 9, 2011

Animation FPS tests

Patrick Walton has been demonstrating how useful some of the new animation APIs are with his framerate monitor add-on, which has already helped to understand some of our performance characteristics and to find various possible improvements for opportunistic wins in our rendering speed.

This got me thinking on how we could use this code to create tests to measure animation speeds and make sure that they don’t regress. One important goal that we have coming soon is to ensure that our UI is always responsive and animations are smooth, and that will certainly need some kind of testing framework.

So I set out on a little weekend project to include the framerate monitor in our mochitest framework. The main goal was that it should be useful to measure both:

  • stand-alone animations, like creating tests for various -moz-transition combinations and sequences;
  • animations in the UI itself, in a way that every time we add a new animated feature to the UI we should be able to write a test to trigger it and profile its FPS

With that in mind, I went to the simple approach of including the monitor as part of  the SimpleTest, such that it can be used from mochitest-plain, mochitest-browser-chrome, etc.

I wrote two tests to begin with and see how it behaves. One is a simple test that triggers a bunch of -moz-transition properties on various elements. The other one came from an interesting observation that Patrick saw some weeks ago: one of the untold benefits of the new and gorgeous about:home is that it’s also faster to resize. Of course, we should never have an entry page that is slow to resize, so the test opens a new browser window and resizes it during some seconds. Currently, for simplicity’s sake, both tests get the average FPS during the process and fails if it was less than 30.

The current patch can be found in this hg queue. Also, for those interested in this area, these problems are also being tackled from different angles too: for example, Ted is working on adding event loop instrumentation, and already has some patches for Linux and Win.


  1. Can FPS tests ever be reliable on Tinderbox, though? Tinderbox test machines are notorious for having unpredictable delays.

    • Yeah I don’t know, maybe not reliable enough for these tests to be useful. Although I’d expect some of the tests that we already run (like tp, txul) which depends on tricky OS races and conditions (filesystem IO, etc.) to be less reliable than pure rendering tests.

      In any case, Tinderbox probably would not offer all the different settings combinations that are needed for these to be useful (covering all the paths for d2d, d3d, gdi, opengl), so maybe this kind of test should live in a separate set that people can run locally (both when making theme, animation or layout changes). This would not provide a machine baseline but maybe it’s enough to catch some problems.

    • Note that we’ve moved away from running our tests on VMs, which were the worst for these kinds of issues, and we now run almost all of our tests on the same hardware that runs Talos. If we trust our perf numbers at all, we should be able to trust FPS tests as well.


%d bloggers like this: