Debugging as a Controlled Experiment

What possibly changed?

As with almost everything in life, there is the inherent need to know which factors influenced a specific outcome, good or bad. It is part of how we sometimes find out which foods and actions the body does not react well to. A similar concept applies to how we learn to relate well with friends and people from diverse backgrounds.

When there are too many varying factors, it becomes difficult to find out which particular one is causing a good or adverse effect. It could be one of them, a combination of them, all of them, or none of them. Do you catch my drift? It can be quite stressful when trying to figure out the cause of something among a thousand possible factors.

All we simply want to do is understand what is going on.

Being the second write-up of The Human Aspect of Debugging series, this focuses on some tips I have learnt and picked up when trying to nail down the cause of a code malfunction.

Victory. Defeat. Victory.

During the 18th century, when Britain and Spain were at war, a British commander, Commodore George Anson, led a squadron of eight ships on a mission. Their mission was to disrupt and capture the Spanish Empire's possessions, and it was fairly considered a success^[1].

In an unexpected turn of events, only 188 men out of the original 1,854 were recorded as having returned from the voyage. This was not due to losing men to an ambush by enemies. Instead, most of them were alleged to have died from a disease known as scurvy, now known to be caused by a lack of vitamin C^[2]. At the time, the concept of vitamins was unknown.

In an attempt to find out the cause of the strange fatal illness, James Lind, a Scottish doctor, performed one of the first recorded controlled experiments. He focused on the diet of British soldiers and divided a sample number of them into 6 groups. He then gave them the same diet but made a unique addition to each group's diet. Cider for one group, oranges for the next and so on. After months of observation, it became clear citrus fruits made all the difference.

All other things being equal

As observed in Lind's approach, the concept of a controlled experiment is aimed at minimising the effects of variables other than the independent variable to increase the reliability of the results.

A very helpful thing I've found to do when debugging is narrow down the surface area. This means that to discover the cause of a bug, one should try not to have too many unknowns. It leads to a larger area for the bug to hide.

This concept plays a major role in reproducing bugs. In James Lind's case, among many other factors, his aim was to try to pinpoint the deciding factor for preventing the disease. In the same vein, going back to the bug story shared in the previous section, I had to test a scenario where:

the Shopify site had only our app installed. Outcome: Widgets showed perfectly.
the other app was installed before my app. Outcome: Widgets did not show.
my app was installed before the other app. Outcome: Widgets showed perfectly.

That said, establishing a pattern that results in a bug does not guarantee a fix to the bug but is very essential. Once a bug is reproducible, the chances of fixing it increase enormously.

Instead of changing a lot of things at a go, it is helpful to make small changes while observing the behaviour. It is worth noting that, applying the concept of a controlled experiment means different things in different use cases.

Sometimes, bugs happen in the wild. That is, they can happen in environments which might be outside the reach of very helpful debugging tools. This can be a great source of frustration since there's not much to go on. In such cases, building on little clues and making little changes in the available environment and observing step-by-step helps to build up confidence and understanding of what the focus should be on.

Additionally, a few generic approaches I use are outlined below when it comes to debugging written code.

Commenting out parts of the code

When there is a lot that has changed about the code and there is no idea what could possibly be wrong, gradually commenting out bits and pieces of the code while testing the outcome can be vital. It helps to incrementally grow into understanding what might be the cause and what is surely not the cause.

Alternating between original and updated code

I have had experiences where a related existing feature just stopped working when new code changes were introduced. In such cases, it is only natural to focus on the new changes to spot what introduced the issue. However, the longer we spend investigating and not finding the reason, the more we are likely to get frustrated and impatient. It could be so elusive that we start believing it was already an existing bug. Surely, that is a possibility and should not be ruled out.

A handy way of confirming or eliminating such a possibility is to revert to the main version of the code and test out the same scenario.

The benefit of this is that, if the feature works well for the original code version for the same scenario, we are motivated to look at the new changes a bit more critically.

On the other hand, if the bug is found to have existed before the code updates. In a fair number of cases, bug investigations have eventually ended up revealing deep-rooted long existing issues.

And that's a wrap

In this write-up, we learn that, when debugging, making small incremental changes and observing behaviour could be helpful in figuring out what might be causing a bug. It also helps to improve understanding of what some sections of the code are really doing.

Thanks for reading! Any thoughts and further experience on this are highly welcome.

Wikimedia Foundation. (2022, November 2). George Anson's voyage around the world. Wikipedia. Retrieved December 1, 2022, from en.wikipedia.org/wiki/George_Anson%27s_voya..
James Lind - Wikipedia. (2022). Retrieved 5 July 2022, from en.wikipedia.org/wiki/James_Lind#Prevention..