Using test cases as a cost and risk control tool

Bugs going unseen until production end up costing even more time and money. Testing should be viewed as a cost and risk control tool upfront during project planning.

By Jason Sampers, Maverick Technologies June 28, 2016

We have all been on or seen software development projects that did not end up getting executed as planned for various reasons such as scope changes, excessive costs, or scheduling issues. Seeing how software projects are actually executed, especially when the aforementioned scenarios are in play, has highlighted how important it is to structure the project up-front.

It is unfortunate, but not uncommon, to see testing as being one of the first areas cut or cut back when a project is under cost or schedule pressure. Not only does that strategy not have the intended effect, it almost always does the opposite. Testing less or later in the process doesn’t change the quality of the solution. All it does is delay issue detection.

Finding the same issue early in the development process verses at the end, i.e. in unit testing vs production operation, is exponentially cheaper and is almost always requires less time to fix. To highlight this point, consider the scenario where a bug is found in a validated production environment:

  • The issue must be identified, reported, and potentially worked around. 
  • The issue must be reproduced, triaged, and assigned a priority.
  • The issue will likely be reproduced again by the developer and fixed.
  • The fix will need to be deployed and tested in a test environment.
  • The fix will need to be deployed and tested in a validation environment.
  • The fix will need to be deployed to the production environment.

The steps listed above don’t include approvals, code reviews, status reporting, change management, etc. However, this still illustrates that more time and resources are required to find and fix a bug that makes it to production instead of doing this during unit testing during initial development. While this is a worst-case scenario, it still illustrates the overall point: identifying bugs later in the process is expensive. And this is before the costs associated with potentially impacting the product, customer, or company brand are considered.

The farther from development a bug is found, the more expensive it is to fix. For this reason, testing should be viewed as a cost and risk control tool upfront during project planning. Adjusting the project plan to create test cases as part of, or in parallel with, the functional and technical design is very effective because:

  1. It ensures that proper test case creation is done up front. In doing so, it prevents well-intentioned decision makers from being able to reduce or cut test case creation from scope later in the project if/when the project is under pressure. We are structuring the project to protect us from ourselves.
  2. The process of creating test cases often emphasizes how the final solution will look to the end user. Working through the process of creating the test cases and reviewing them with stakeholders and end users prior to development, provides an opportunity for miss-communication and gaps to be identified before coding begins which reduces the risk of re-work.
  3. The test cases provide developers additional context that augments and clarifies the designs. This helps ensure that developers clearly understand what the solution should look like which reduces the likelihood that bugs will be introduced in the first place. Avoiding bugs entirely is a huge time and cost savings. Especially when compared to the minimal time investment of creating test cases; test cases that would have been likely created later in the project anyways.
  4. Developers now have a robust test script to follow when performing their unit testing that should closely match the final validation test scripts. The cheapest place to find and fix a bug is during unit testing so giving developers additional tools that increase the up-front detection rate is incredibly valuable.

While making such a small shift to require test case creation prior to development may seem trivial on the surface, in practice it can have a profound positive impact on the project.

What are some other testing related tips and tricks that you have found to help detect bugs earlier in the project and reduce overall costs?

This post was written by Jason Sampers. Jason is an application architect at Maverick Technologies, a leading automation solutions provider offering industrial automation, strategic manufacturing, and enterprise integration services for the process industries. Maverick delivers expertise and consulting in a wide variety of areas including industrial automation controls, distributed control systems, manufacturing execution systems, operational strategy, business process optimization, and more.

Maverick Technologies is a CSIA member as of 6/28/2016.