Release Pipeline Risk Assessment

Agile frameworks have changed software delivery dramatically. Nowadays, more companies are adopting well-defined frameworks like Scrum or Kanban, and, as a result, clients expect to release new features faster without sacrificing quality or security. A release pipeline involves the implementation of continuous integration and continuous delivery practices. With stakeholders increasingly needing features to be deployed faster, more frequently, and in a controlled manner, the risk assessment of a pipeline has become a “must” in tech companies today.

Release Pipeline Scenario

To understand the importance of risk assessment, let’s look at a realistic scenario. Imagine a feature for a website store finder that was built by a development team, tested in the QA server, and even tested by the client in User Acceptance Testing (UAT).

 

———-

You might also like our blog
How Can UAT Testing Drive Your Software Results?

———-

 

Since all testing stages went well, the team got a green light for deployment. But then, during smoke tests performed by QA, it was discovered that the finder did not work properly; the server returned an HTTP 500 error on form submission. When the technical lead then tried to rollback the deployment, the pipeline for rolling back failed, giving errors while trying to build the last code tag released.

 

A development team had been asked to build a store finder inside the main page of a website that supported campaigns for prepaid mobile phones. Basically, the client wanted users to be able to easily find their nearest shop and purchase new devices. The requirement involved a database connection, some backend code, and of course frontend code. The team decided to have a different database per each environment. As stated, after the build, testing was performed, and all parties were “go.”

 

After the release branch was created and merged to master, the deployment to production was performed the next day at 1PM by the technical lead. This is when the issues with the finder and rollback pipeline popped up during the smoke tests.

 

Obviously everyone was in a panic, and the client was not happy. The end result of this unfortunate scenario was that the team had to hard reset the master branch and perform another deployment to get the application back to normal. All in all, the issue was noticeable to end users for 30 minutes.

Release Pipeline Risk Analysis

Many of the problems described above could have been avoided by analyzing some common risks involved in the release pipeline. 

 . These range from company culture and poor communication to improper testing and versioning strategies. Suffice it to say, every aspect of a release—both technical and non-technical—comes into play and needs to be carefully analyzed and taken into consideration. Here below, we cover some of the more typical risks teams face today.

Building Without Tests

All branches have to be validated via continuous integration. A pipeline without continuous integration is not really a release pipeline . QA and developers should always work together in order to guarantee that every artifact is correctly tested, whether via automated or manual checks.

Complex Pipelines

Complex configurations are harder to maintain. DevOps teams should apply the KISS principle as much as possible so every team member can easily understand how the whole process works and cover for each other when members are on vacation, sick, etc. Having only one person with knowledge of the pipeline makes the project far less reliable as it will be too dependent on that one individual.

 

———-

You might also like to read
8 KPIs for Stress-Free Testing

———-

Not Testing in a Pre-Production Environment

Developer changes should never go to production without going to a pre-production environment first. This environment usually has the same configuration as in production, so the development team and clients can get an idea of how the feature will perform once it gets in production. DevOps should make sure that there are always both a staging and UAT environment. It is extremely important that clients check features before going live so they can validate if developed features are aligned with business necessities.

Using The Wrong Tools

Architects and DevOps should always use pipeline technologies that are closer to the software stack. Obviously, if a product is deployed on a Linux server machine, then a Windows BAT file should not be used in the configuration. For mitigating this risk, release managers, IT managers, and architects need to fully understand a project and select the tools that suit it best.

Human Intervention

IT managers should focus on automating their release pipelines. The more you remove human intervention, the more predictable releases will be. Here, DevOps discipline is the key—work on building pipelines with quality, resilience, and speed.

 

Lack of Consensus

Governance guidelines usually define the way a team should deliver new features—what would be the best time for deploying, who is responsible for the deployment, etc. Nowadays it’s very odd to have only one person in charge of triggering the deployment; but it can happen that one team member goes ahead and starts the pipeline on his own. This is very dangerous, because one person’s capacity may not be enough to perform a rollback in case of failure. In order to mitigate such a risk, team members should receive training, correct permissions must be set in the pipeline’s configuration, and, most importantly, managers need to achieve proper communication inside the team.

Releasing the Wrong Build

Teams have to be aware of the branching and versioning strategies being used in order to configure the pipeline correctly. Imagine having deployed a build that was supposed to have fixed a bug, and the client then realizes that the bug is still in production. Developers must ensure that everyone is following conventions and need to check the correctness of the pipeline frequently. Having policies in the governance documentation can help avoid this scenario.

Including Experimental Features

Due to a lack of vigilance and a weak pipeline configuration, new experimental features can get into production, revealing a client’s intentions and thus possibly leading to financial losses for it as well. Managers and developers need to be aware of the pipeline’s configuration and versioning strategy, so they know which branches are going to be deployed and which ones are not. Another strategy is to limit the branches that developers can operate on, for instance, only allowing the master branch to be deployed to the production environment.

Downtimes

What should be the state of the application while a new build is being deployed? Will the client be happy if the application goes down while the release team is triggering a new deployment? Obviously the answer to that second question is a resounding “no.” Dev and DevOps teams need to adopt a strategy to ensure the application experiences zero downtime. And this should be defined in the governance documentation as well.

Delivering in a Rush

A team can easily come under immense pressure to release a feature or a bug fix fast—meaning that every member is more liable to make mistakes. Delivering in a rush is always risky, but a reality of business today. That’s why teams need a checklist for releasing new code in order to feel confident in the process and avoid human error.

Lengthy Roll Backs

When a feature is released with a bug, it has to be removed from production as soon as possible. The longer the bug lives in production, the more clients lose trust in the company. Release managers should always push for the testing of rollback procedures and their performance to make them as swift and as smooth as possible.

Late Lockdowns

Once the pipeline is validated, not everyone should be able to modify it. Limited access permissions need to be instituted as soon as a pipeline works, and DevOps needs to check and test access permissions regularly.

 

———-

You might also like our blog
Beyond Test Automation: Leave Nothing to Chance

———-

 

Identifying and Managing Risks Successfully

Software releases have evolved towards agile frameworks. And yet, with evolution comes new risks. But how can companies properly analyze, manage, mitigate, and resolve the risks related to a pipeline release? Today teams handling an end-to-end development and delivery process need a third-party tool to detect risks early on and manage them throughout the pipeline. They need the ability to analyze the impact of new features and changes to software on their company’s business processes in order to highlight where risks exist, while eliminating unstable code and removing many of the risks discussed above.

 

Third-party tools such as Panaya’s Release Dynamix [RDx] provide the means to assess, detect, and manage risk, yielding a stable, risk-free release pipeline and successful deployment process. This includes visualization of the scope of releases across multiple project waves including valuable impact analysis and code quality review data, which RDx uses to automatically generate test plans as well. RDx’s risk-based testing is a powerful tool for addressing the risks it discovers and ensures that these risks have test coverage. Additionally, RDx presents risk-related data for all stakeholders to clearly see, including current risk levels, development status, code quality, test planning and execution, defects closure, and overall requirements completion.

 

Implementing continuous integration and continuous delivery practices into release pipeline practices can be a challenge, but with the right tools, companies can achieve fast, secure, and top-quality pipelines they need to succeed.

 

Continuous Delivery for Enterprise Applications

I Want Risk-Free Change

×