2008-02-15

The art of writing a good bugreport

if at first you don't succeed... file a bugreport

Over the years I've seen quite a number of bug reports, and often they were inspired by a bug, and often enough, the bug was mine to fix.

In order to fix a bug, a developer needs a good bug report. The classic case of a bad bug report that we have used for a long time now in the eco team is the "kaboom" bug report.
The customer simply said: "I tried to do this and that and 'kaboom'!"... how were we supposed to interpret that? did the machine really explode?

Here are a few things to think about when you report a bug to someone else:

Reproducible:

Make sure you can reproduce the bug. Make sure you know the required starting point and the exact steps to make the bug appear. If you don't know these things, at least be honest about it, and preferably, find a good discussion forum to ventilate your problem and see if anyone else knows the steps to reproduce the issue before reporting it.
In some cases, it could be enough with just an exception message and a callstack to be able to fix the bug, but it is hard to know if the bug is really fixed if it can't be reproduced.

Steps:
If your bug is reproducible, then you know the steps to reproduce it. The developer responsible for fixing the bug hasn't seen the bug happen when he reads the report. He (or she) will need detailed steps in order to reproduce the bug.
Ideally, the steps looks something like
* New winform eco project
* add a class with an attribute of type X
* generate source
* Drop a button on the form, execute:
Exp: this
Act: that

Exp and Act
It is very important to explain both what actually happens (act9, and what you expected to happen (exp). Sometimes, maybe your expectation is wrong, and what really needs fixing is the documentation so that you know what to expect. In order for the developer to know that he has reproduced your bug he needs to know what happens in your environment.

Isolate the bug:

The fewer steps needed to expose the bug, the better. Try to eliminate as much noise as possible. If you are using third party components, try to reproduce the bug without those. If the bug doesn't seem to be related to persistence, try to reproduce the issue in a transient system.

Sample project:
In many cases, a small example project that exhibits the bug can be helpful, but sending your 100kloc project with a 3gb sqlserver database is usually not appreciated. Also, when you attach a testcase to your bugreport, please remove any compiled files unless you are certain they are needed to reproduce the bug. Zip up your project and then remove the following:
*.exe; *.dll; *.pdb; *.dc?il; *.identcache; bin-dir, obj-dir, any history/backup files.

Exceptions and Callstacks:
Callstacks are often very helpful for a developer to figure out what is wrong. The exact error message in the exception is also very important. Please include these if you get an exception.

Unit tests:
I have been thinking of writing this piece for many years, and I had always planned to add a contest at the end. The first person who would submit a previously unknown bug with an accompanying unittest that exposed the bug would win a beer. I had never seen this in any of the bug reports I have received in the past 10 years as a developer until Dmitriy Nagirnyak sent me a bugreport last october (http://magpie.sytes.net/mantis/view.php?id=237). Read and learn folks! It was a fairly complicated bug, but Dmitriy had done his homework and isolated the bug, prepared a testcase and explained all the details required to understand the bug. Instead of promising a beer to the winner (Dmitriy, whenever we meet the beer is on me), I promise a private build off the current stable branch of eco if you submit a new bug with a relevant isolated nunit-testcase as soon as the fix has been checked in.

So... if you find a bug, don't be afraid to report it, but please follow these guidelines when you do.

1 comment:

Dmitriy Nagirnyak said...

Jonas, Thanks for noticing me.
If we'll meet one day I'll buy a beer for you as well.. with pleasure :)

I hope I'll have a chance to travel to Sweden one day... at least to meet you :)