F21 was a departure from past releases and gave the whole of the project a lot of new processes to create and improve. Instead of one single release we release 3 distinct products - Workstation, Server and Cloud. There was more to test and even more stuff to figure out for this release.
Fedora 21 is ready to be set loose on the world next Tueday (December 9th), and it's about time I published the contributor stats from throughout the release :) Sorry I've neglected to gather the stats throughout - but better late than never, right?
The stats here were gathered from Bodhi, Bugzilla and the Fedora Wiki using the "fedora-qa" stats scripts as well as relval. As we'll be going through all the stats for the entire release, this post is going to be rather long. Let's get to the statistics!
Highlights
- Throughout the F21 cycle, 2524 testcases were ran by roughly 30 testers.
- We had 5143 total Bodhi comments from over 175 testers
- 3470 new bug reports from over 500 people for F21
Those are some pretty impressive numbers - great work to all who were involved! From here on out we'll break down the stats from the Wiki (Release Validation), Bugzilla reports and Bodhi comments (updates testing).
Wiki Stats
This is a slightly trimmed list of people who submitted release validation testing results. The matrices changed a bit for F21 with some wonderful wiki-fu done by Adam. All the testcases are all on one single "Summary" page instead of several different pages like previous releases. Despite all our changes to the processes the stats are still very similar.
Alpha
Testers: 23
Comments: 463
Unique referenced bugs: 29
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| pwhalen | 124 | 1003122 1112387 1136990 892178 (4) |
| satellit | 82 | 1036306 1103496 1121409 1136570 1136581 1136985 1136994 1139015 1141414 (9) |
| michalv | 48 | |
| roshi | 38 | 1134524 (1) |
| adamwill | 26 | 1099299 1101341 1134524 1140867 1142512 1142516 1142517 1142520 1142521 1142522 892178 (11) |
| pschindl | 24 | 1143972 (1) |
| kparal | 23 | 1142862 1144040 (2) |
| jskladan | 22 | |
| nonamedotc | 15 | |
| wutao85 | 9 | |
| spstarr | 7 | |
| hguemar | 6 | |
| mattdm | 6 | |
| sgallagh | 6 | 1143988 1143990 (2) |
| lnie | 6 | |
| robatino | 5 | |
| jdulaney | 4 | |
| passthejoe | 3 | |
| jreznik | 3 | 1003122 (1) |
| wolnei | 2 | |
| b10n1k | 2 | |
| konradr | 1 | |
| jbwillia | 1 |
Over 450 results were submitted for Alpha, which is great. Keep in mind, this doesn't include all the rawhide validation testing we'd been doing until Alpha was branched. Pwhalen, Satellit and Michalv tore the matrix up with over 250 combined results submitted! Now let's move on to Beta (keep an eye out for a trend here).
Beta
Testers: 30
Reports: 889
Unique referenced bugs: 38
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| pwhalen | 135 | 892178 (1) |
| adamwill | 133 | 1142520 1142521 1142522 1145777 1145783 1148087 1150147 1155026 1155301 1155304 1155329 1156198 1156354 1156380 1156614 892178 (16) |
| satellit | 112 | 1036306 1103496 1136581 1141414 1145281 1148436 1149782 1158533 1169440 (9) |
| roshi | 105 | 1148087 1156198 1156603 (3) |
| kparal | 88 | 1038413 1155964 1156008 1156073 1156354 1156380 (6) |
| pschindl | 80 | 1158533 1158975 (2) |
| dmossor | 31 | 1135024 (1) |
| sgallagh | 27 | 1156378 1156593 (2) |
| yselkowitz | 26 | |
| jskladan | 25 | |
| jsedlak | 20 | |
| nonamedotc | 16 | |
| lnie | 15 | |
| robatino | 15 | |
| kushal | 12 | |
| kevin | 10 | |
| mattdm | 6 | |
| jreznik | 6 | 705086 (2) |
| optak | 6 | |
| tflink | 4 | |
| vehrlich | 4 | 1156499 (1) |
| eischmann | 3 | |
| szpak | 3 | |
| wolnei | 1 | |
| konradr | 1 | |
| pbrobinson | 1 | |
| lbrabec | 1 | 1099299 (1) |
| cmurf | 1 | |
| greylocks | 1 | |
| outsh | 1 |
Again, you can see Pwhalen doing some mad work throughout. Satellit comes in third after being edged out by Adamw by a mere 21 results. Between Alpha and Beta several new testers joined the cause and the reported results exploded from 463 to 889! Thanks to everyone for great testing during Beta!
Final
Testers: 29
Reports: 1172
Unique referenced bugs: 17
| Name | Reports submitted | Referenced bugs1 |
|---|---|---|
| pwhalen | 223 | 705086 892178 (2) |
| roshi | 148 | 1169085 (1) |
| satellit | 141 | 1135720 1136581 1169440 (3) |
| kparal | 86 | 1169332 1170153 1170499 1170623 (4) |
| amrithajij | 79 | |
| lnie | 71 | |
| mmarusic | 70 | 892178 (1) |
| jskladan | 61 | 1162215 (1) |
| dmossor | 54 | 1169085 892178 (2) |
| adamwill | 53 | 1142521 1142522 1170062 892178 (4) |
| pschindl | 49 | 1167959 (1) |
| mccann2 | 20 | |
| robatino | 20 | |
| kushal | 17 | |
| sgallagh | 16 | (1) |
| spmadden | 12 | 1169085 (1) |
| jsedlak | 10 | |
| outsh | 8 | |
| mattdm | 7 | |
| nonamedotc | 7 | |
| jreznik | 4 | 1170535 (1) |
| cmurf | 4 | |
| amritha | 3 | |
| tflink | 2 | |
| optak | 2 | |
| jdulaney | 2 | |
| sdharane | 1 | |
| lbrabec | 1 | |
| jstodola | 1 |
Again total reports shot up between Beta and Final. While the number of testers remained about the same, the amount of testing steadily increased as we got closer and closer to release. Very similar to how things went with F20. Did you spot the trend in those three tables? Pwhalen and Satellit dominated for an entire release. Pwhalen submitted 482 test results and Satellit submitted 335 results - about 30% of all testcases reported.
Awesome work! Looking forward to what kinds of numbers you post for F22 :p That's all we have for wiki stats, so lets move onto updates testing stats!
Bodhi stats
These numbers are gather from comments on updates waiting to be pushed to stable. Users leave karma on the update and once it gets enough karma from testers, it can go into the stable repositories. These numbers reflect the amount of karma each user left.
Alpha
Testers: 108
Comments: 550
| Name | Updates commented |
|---|---|
| Carlos Morel-Riquelme (n0oir) | 78 |
| Parag Nemade (pnemade) | 57 |
| Haïkel Guémar (hguemar) | 37 |
| Piotr Drąg (raven) | 26 |
| Igor Gnatenko (ignatenkobrain) | 26 |
| Fabio Valentini (fafatheone) | 26 |
| nonamedotc | 24 |
| Marek Dolezel (mdolezel) | 23 |
| Christopher Meng (cicku) | 19 |
| Adam Williamson (adamwill) | 19 |
| Chris Sandler (chr77) | 18 |
| Ankur Sinha (ankursinha) | 16 |
| Jiří Popelka (jpopelka) | 13 |
| taskotron bot user (taskotron) | 9 |
| Pravin Satpute (pravins) | 7 |
| Hans Müller (cairo) | 7 |
| Peter Robinson (pbrobinson) | 6 |
| Martti Kuosmanen (kuosmanen) | 5 |
| Menanteau Guy (menantea) | 5 |
| bitlord | 5 |
| menantea at linux.vnet.ibm.com | 4 |
| Endi Sukma Dewata (edewata) | 4 |
| Abhishek Koneru (kaskahn) | 4 |
| Elad Alfassa (elad) | 3 |
| Stef Walter (stefw) | 3 |
| Paul Whalen (pwhalen) | 3 |
| Marius Vollmer (mvo) | 3 |
| Mathieu Bridon (bochecha) | 3 |
| robatino | 3 |
| Fraser Tweedale (ftweedal) | 3 |
| Karsten Hopp (karsten) | 3 |
| nucleo | 3 |
| ...and also 76 other reporters who created less than 3 reports each, but 85 reports combined! | |
Carlos, Parag and Haïkel rocked the testing with a compined total of 172. That's 31% of all comments during alpha, great work! 550 comments and 108 testers might seem like a lot - but check out the stats for Beta.
Beta
Testers: 199
Comments: 2406
Beta is when Taskotron took over for AutoQA - which is able to run tasks on each package and leave comments on any issues it finds. A big thanks to the Fedora QA devel team for working so hard on this and getting it in place! When you take out the results from taskotron, we had an increase of 90 people leaving karma and 865 more comments left compared to Alpha.
Chris Sandler tested 222 packages, edging out Carlos by a mere 42. Parag dropped down to 4th and Hans Müller came out of nowhere in 3rd with 151. Great work!
Final
Testers: 185
Comments: 989
Even through to final, Carlos continued to rule Bodhi stats with 93 comments. Hans and Chris were neck and neck with 83 and 82 respectively. It's awesome to see so many people testing packages, helping get packages out to the rest of the Fedora community! Thanks to everyone who helped throughout F21!