Once in a while you come across a bug that sticks to you. One of those bugs reoccurred to me during the Rapid Software Testing class (RST) given by James Bach. Among a lot of very useful lessons in testing James spoke about boundary testing. I learned that when a boundary is set often other boundaries are not far away. That lesson reminded me of a bug! James challenged me to name the technique I used that found the bug. So here’s the story on how a boundary test turned out to be an adventure, ultimately resulting in the discovery of the datatype positioning heuristic.
In the beginning there was a boundary test
I was testing a mobile application where I was testing the boundary of a list of items. The maximum items allowed was 20 items. That was all that was written as a spec.
I knew that we always tried to minimize service calls and message size. Having fast and lightweight messages keeps the end user happy on a mobile device, plus the calls in the back end cost less money. I wondered why the boundary was specifically 20 items and why file size didn’t seem to matter? The solution architect couldn’t give me a satisfying answer at the time so I started to explore around the boundary of 20 items.
Soon I discovered some strange error messages and inability to create new items. I was however far away from the maximum of 20 items. In fact, this behavior seemed to be very random. Sometimes it did not work after adding 17 items, or even 13 items.
Since I did play around the item boundary with different file sizes I thought I was on to something here. I found that arrays with larger items indeed seem to fail. After the solution architect looked at the bug he replied. “The middleware guys told me there is a file size limit of 300kb per array. This is to minimize network traffic. It doesn’t seem to be enough. I’ll have it made bigger to 500kb. This should be enough.” File size did matter and our quality attribute to use less traffic was indeed the cause. Hurray for the tester!
The discovery of a 2nd boundary
Or…well I had my doubts I did also find some test sets that failed even though they did not exceed the 300kb boundary. Was the solution architect jumping to conclusions? So my exploration continued.
I started to play around the 300kb boundary limit. Again I found the bug. It seemed that my test sets with diacritics seem to fail more often. But not all of them. So I zoomed in on diacritics and discovered that all non-diacritic test sets did not trigger the bug. Aha!
But why didn’t all my test sets with the same file size and diacritics fail?! Did file size not matter after all? I was using all the allowed diacritics in our character set and they where al displayed correctly. So different character sets in the chain seem not to be the issue. What was so different from the ones that failed and which ones didn’t?
The single diacritic that found a new boundary
Files with the € sign were more likely to fail, but not all. After some re-positioning of the € sign I spotted a pattern. All items following an item with 40 characters or more including the € sign gave me an error message. Did I again found a new boundary? I was now able to reproduce the bug after only creating 2 items. I found the pattern and could reproduce the bug, but what was causing this trouble? Clearly the amount of characters and € sign caused it, but why?
After investigation with the solution architect we found the problem. It turned out file size didn’t matter. The size of the database record however did! Again a new boundary was found. Records in the database only could store 40 bytes. Again a new boundary was found. Items received from the service where concatenated to one large string and then split by an algorithm. The algorithm split the string each 40 characters and inserted them in the database. And that’s where the bug was created! Because 40 characters is not necessarily 40 bytes. It turned out that there was only one character in our character set that was bigger than 1 byte. Yes, indeed the € is 3 bytes. Resulting into a record that was to big for the database to store. Mystery solved and a new heuristic was born: The datatype positioning heuristic.
Recommendations on how to use the datatype positioning heurstic:
1. Use diacritics that are bigger than 1 or 2 bytes (like the €). Use a unicode table website to find out how big characters are.
2. Re-position the diacritics in your message. You never know what boundary you’ll find in the chain.
3.Use API testing tools. I was mainly testing the UI of the mobile app, but Charles was a bliss to work with. Charles is a HTTP proxy tool I used as REST API tool. It made preparing test data and testing without the need of the UI so much easier and faster . Besides, I would have probably not found this bug, or at least not the root cause, through UI testing only.
4. Determination. If a solution is provided don’t immediately accept it, but understand the problem first.
Use the heuristic to your own benefit. I hope it can help you find a couple of bugs or at least deliver more information on the quality of your product. Do you also have heuristics to share? Or have a nice bug story of your own. Drop me a mail or write a blog about it yourself.
Thanks to James Bach to encourage me to name the things I do and big thanks to Rogier Saarloos on reconstructing this bug story with me.