Last week the QA community dedicated itself to testing a variety of graphic drivers during the Graphics Test Week. Over the three days, we had 29 people testing (most of them testing multiple days) resulting in updated information on 18 bugs. I took some time to get some metrics on the test days and thought I would share.
What's in a test day?
Each day had 22 test cases to verify for each of the drivers (Intel, Radeon, Nouveau). There were five categories of testcases: Basic, Desktop, Video, 3D and Experimental. Basic and Desktop got the most coverage - likely because that's where most people do their daily work.
3D was a great section. Testing 3D included playing cool games like Xonotic, 0 A.D., SuperTuxKart and the stellar astronomy app Stellarium. While there was some hiccups with Nouveau (for me at least), most of the tests went great (not to mention that you got to play some games).
Onto the numbers!
The Intel day had 187 tests ran, Radeon had 153 and Nouveau had 95. That's some pretty good coverage in a short amount of time. Even the Experimental testcases (Dual GPU, Steam, WebGL and Rendercheck) got decent amounts of coverage.
Gathering the results from the test days was done in the new TestDay App - which made inputting your results a whole lot easier. It also generated the results tables on each of the wiki pages. The app has been a great boon to running test days. However, gathering metrics about the test days turned out to be a bit harder.
The Test Day SOP (Standard Operating Procedure) has a nifty command line solution for finding all bugs referenced; but other metrics like number of test cases ran, testers, etc., is another story. Some of that data probably lives in the app, but I didn't find it. So all those numbers were gathered by hand - far from a wonderful solution.
Conclusions
All and all the test week went off really well. We had good participation as well as gathered some good data for bug reports. It will be a good if we can keep this trend up with future test days (spoiler: I think we will :) ). The community surrounding Fedora and Fedora QA is great and always looking for more people to jump on board. We'd love to have you - join in on an upcoming test day!
The next test days on the schedule are for Printing on Guy Fawkes Day and KDE Test Day on November 7th. I've been running the alpha as my main machine for the past couple weeks and haven't had any issues. So go grab a copy and install it somewhere - then tell us what you think!