Internet Explorer: Fewer Bugs Than Firefox?
So, which Web browser has the most software bugs: Google Chrome, Microsoft Internet Explorer or Mozilla Firefox? uTest, a fast-growing marketplace for software testing, uncovered some rather intriguing answers. Here’s the scoop, from The VAR Guy.
More than 1300 software testers from 68 countries recently participated in uTest‘s Battle of the Browsers competition. The uTest community found:
- 672 overall bugs
- 101 “showstopper” bugs, which are flaws that need immediate attention
So which Web browser was the buggiest? Apparently, Firefox 3.1 — though it’s important to note that Firefox had the most number of testers hunting bugs during the contest. In theory, the more testers you have the more bugs you’ll find. Here are some quick stats on each browser:
- Internet Explorer 8: 356 uTesters identified 168 bugs, including 9 percent that were classified as showstoppers.
- Firefox 3.1 Beta: 514 uTesters identified 207 bugs, including 24 percent that were classified as showstoppers.
- Google Chrome: 461 uTesters identified 297 bugs, including 12 percent that were classified as showstoppers.
Of course, the findings above aren’t exactly scientific results. But The VAR Guy was impressed to see Internet Explorer with comparatively few bugs. (Here’s uTest’s complete recap of the Browser of the Browser bug hunt.)
On Demand Bug Hunters
Still, there’s a bigger story or trend here. During the economic slowdown, many software companies will struggle to maintain their internal bug-testing staffs. On the flip side, novice external testers can either slow down a beta process or fail to properly document bugs for software companies.
Enter uTest, which could serve as an on-demand marketplace for software companies that need to polish their code but either (A) can’t afford a massive, internal code-testing team or (B) don’t have a loyal user base willing to test beta code. uTest potentially fills that void, marrying software companies with beta addicts who love hunting bugs.
The VAR Guy is updated multiple times daily. Don’t miss a single post. Subscribe to his newsletter, RSS and Twitter feed.
The really interesting thing is that even Google Chrome had more testers than Internet Explorer.
It’s not a Bug. It’s a feature! —- Fhew! A lot of Features.
The whole comparison does not make sense to me, as it is not a fair comparison in the first place. Firefox has source code for bughunters to scrutinise. IE should also disclose the source code for fair ground comparison.
Also, in terms of number of features, Firefox gt; Chrome gt; IE. Let all browsers have equal number of features, then we see who has more bugs. The fact that Chrome has more bugs than Firefox testify that despite that Firefox has more features (Add-ons support, private browsing, RSS, etc).
The announcement doesn’t say whether Firefox was tested with third-party extensions or not, so it’s hard to say whether the results are even meaningful.
Another interesting datum from the results is that NOBODY considered IE’s usability to be “excellent”, while Firefox and Chrome both were considered to have excellent usability by at least 10% of the testers.
Let IE (MS) open up their source code and then we can have a fair comparison. MS can’t be trusted to be open with their bugs.
Craig: Nice job pointing out the “usibility” data point. Very interesting, indeed.
The number of testers for each browser was no where near equal, so raw numbers need to be compared to a reference.
Ratios serve nicely for that purpose. When you compute the number of bugs per tester and the numbers of testers per bug, things look much different. When viewed this way, Chrome comes out being the buggiest, which makes sense given that it is a new product still in beta. Firefox comes out being the least buggy.
Chrome:
1 bug/1.55 testers
0.64 bugs/tester
Internet Explorer:
1 bug/2.3 testers
0.47 bugs/tester
Firefox:
1 bug/2.48 testers
0.40 bugs/tester
The picture looks a little different when you extract the percentage of showstopping bugs per tester:
Firefox: 9%
Chrome: 8%
Internet Explorer: 4%
However, I don’t see anywhere a definition of “showstopper”. What does this mean, and how is a bug classified as a showstopper?
Capella:
We do not need to count bugs in the source code to be fair. We do not have to count features because it is difficult what is counting features. I think it is the daily experience which counts. Just count obvious failures visible at the user interface: crashes and deficient functions. To my knowledge Firefox crashes more often and has more problems with printing web pages, as compared with IE.
Scott: Excellent point about the importance of drawing an apples-to-apples comparison. This is our first-ever Bug Battle, and I’m listening intently to the comments and questions that arise so we can make future results as interesting, accurate and useful as possible.
Also, we intend to host a new and different Bug Battle each quarter, and we’re going to be soliciting contest ideas for Q1 (eg: social networking sites vs. IM tools vs. operating systems). So we’re going to want to hear your ideas for the next uTest Bug Battle after the holidays.
-Matt
Matt Johnston
VP of Marketing, uTest
http://www.utest.com
mattj[at]utest.com
Matt: No need to test social networking sites. Everything a reader could ever want is already on TheVARguy.com — or coming soon to this site.
🙂
ObiWanKenobi:
Nope. I don’t agree that disregarding features is relevant.
More features lead to potential for more bugs. Besides, if you want to compare *YOUR* experience and *MY* experience, it is again subjective. To my knowledge, IE is slow, hangs often when visiting web pages with large number of table cells. So your poison and my poison are different, and that makes it all equal. Let’s come back to the actual parameters: Number of features, availability of source code, dependencies to other modules, etc. All these are measurable, and should be used as a basis to determine whether number of bugs found is relatively high or low. Also, Scott raised another important point, how many eyes are looking at the browser.
The point here is this, if you want to conduct a study, you conduct it scientifically where the methodology is fair and right, taking deviations into account. No point you have 1,000 people looking at Firefox and 5 people looking at IE. At the same time, the same 1,000 people looking at Firefox must be looking at IE to be fair. If not, be prepared to be bashed by reviewers.
who cares if IE has less bugs. MS is a despicable company whose sole purpose is to make Gates and Balmer rich.
Greg: How do you really feel about Microsoft?
How about bugs per platform? I know that IE only runs on Windows but what is the bug count for FireFox on OS X and Linux?
Is it possible that some of the FireFox bugs on Windows are introduced by undocumented APIs?
I’m going way back but does anyone remember WordPerfect for Windows 1.0?…or Netscape in the early days?
Microsoft are a lovely company. They go out of their way to give IT folks ample chance to prove their worth by constantly breaking compatibility between the Windows API and applications and all people do is complain.
It’s fair to say the IT support sector would be much smaller without Microsoft. Some companies might not even exist!
But back to the topic. How do we measure the amount of effort bug hunters are putting in to finding bugs? One of the problems I see with this browser bug hunt battler is that the sort of people involved in bug hunting for open source projects put in huge efforts to find bugs. So open source users might just be much better at finding bugs.
Aikiwolfie: You’ve caught The VAR Guy’s attention again. Yes, flawed Microsoft software has created thousands of IT jobs. Think about it: There are actually IT managers who specialize in patch management for Windows servers and desktops. That’s all they do all day is test the latest patches. And when their work is done, along comes a new patch.
John Orr: Undocumented Microsoft APIs? Say it ain’t so 😉
I too aggree with Cappella
Less number of features means less number of bugs. +
See the acid test marks : http://acid3.acidtests.org/
firefox : 71 out of 100
IE 7 : 12 out of 100
Opera : 85 out of 100
So Rendering capablity of IE is only 1/6 th of Firefox.
how these products can be compared?
Apart from the fact that they were testing the FF beta against the rest (not beta) is that the FF bugs will be fixed faster then the few IE bugs. Hahahaha !!!!
It always amazes me how “in the know” IT people can be so clueless and leave out Opera. I use all of the browsers on Mac and Windows. Each has a few things they do a little better. But Opera is hands down the very best browser made, no others come close. Firefox and IE are inferior and IE is used by me only for ActiveX and Microsoft forced uses. Opera has all platforms covered, it’s fast and flexible and tight. Get a clue.
KellyBundy: The VAR Guy has sung Opera’s praises from time to time. But uTest’s bug chase didn’t focus on that browser. Our resident blogger therefore didn’t mention Opera in the post above.
Unfortunately, I rarely keep up with the bugs or even notice them, without regard to how they each handle CSS differently. At work, I use Google Chrome mostly with a little Firefox 3 on the side. I almost never use IE except to test how a site’s CSS might render.
In terms of usability, I find them all fairly usable with Google Chrome being a bit too unincumbered by toolbars. I’m a toolbar junky and if you saw my FF, you’d agree!
To the credit of Kelly Bundy, I thoroughly enjoy using Opera. It seems to be the most stable (in terms of not crashing out on me) of any of the other top 3. Plus, it was MUCH easier to install on Debian – Etch than was Firefox 3. Another story – another time.
Great commentary!