Stories from the Trenches - How Sirius Achieved Automated Change Control
So you are migrating to the cloud.
Your team is ready to embrace the increased agility, flexibility, and scalability; benefits the industry has promised native cloud, DevOps, and agile will provide.
How do you plan to manage all of these accelerated changes?
Unfortunately, once transitioned to the cloud, many enterprises are discovering that their old traditional ITIL-based change management processes do not transition well to the dynamic nature of the cloud.
The standard change approval board and ITIL processes are unable to evaluate and review all of the key planned changes in a timely manner, as the changes are coming in huge volumes and moving fast, each change being relatively small.
More than that, the traditional change framework is not equipped well enough to accurately assess the risk of these changes as it relies on manual methods. Neither can it ensure that the changes are executed as planned.
The traditional ITIL processes simply can not keep up with the constant, rapid changes required in the cloud.
While slowing things down is the last thing you want to do in your cloud environment, you still need change controls, as even the most minor changes can lead to detrimental stability, compliance, and security of your IT environments.
So how can IT maintain the same control as your previous ITIL processes while keeping up with the speed of change in the cloud?
To solve this problem, Sirius Computer Solutions brought in Evolven.
With Evolven, Sirius Computer Solutions has been more successful by achieving both speed and control across their cloud environments.
“We wanted automated detection of changes, and not just the things that we knew about, but the things that we might not know changed,” said Sean Kaminski, in a recent webinar, “My Cloud Environment is Always Changing!”.
With Evolven, Sirius was able to achieve their key goals to address change in their cloud practice, including:
- Know what's changing from development to stage, to production, tracing changes across the lifecycle
- Receive automated reports on any drift or inconsistencies in their cloud environments
- Reduce recovery times and help customers determine the root-cause through detailed change information and analytics
- Integrate with ServiceNow to decide which changes need to be processed through the change board
Stories from the Trenches
During the recent webinar, “My Cloud Environment is Always Changing”, Sean Kaminski shared some use cases for when Evolven helped to quickly investigate incidents which would otherwise take a long time to address.
Customer Deployments through CI/CD Pipeline Kept Failing
In the first case, the customer deployments through CI/CD pipeline based on Jenkins kept failing. The customer and Sirius experts couldn’t figure out what was causing the failures.
Evolven in Action
Using Evolven, Sirius was able to see that early in the update process, ECS instances were scaled from 1 to 2. However, the Autoscaler had a max of two instances.
Therefore, when the rolling update was attempted, there was no “headroom” for the new instances to spin up.
With Evolven, Sirius was able to pinpoint what was happening and significantly reduce MTTR by quickly identifying the cause of the failure.
Customer Reported Errors in their Web Application
The second instance shared by Sean was a significant production issue.
A customer had reported errors in their web application for some users.
The last thing your customers want to see when entering your website is an error message!
The initial investigation outside of Evolven stated that no changes had been made to the environment.
How many times have we heard that before?
Evolven in Action
When Sirius’ support staff went to look at the Evolven console for that environment, they saw that Evolven had detected an actual change and marked it as High Risk.
Looking further, it was determined that the NGINX configuration was modified and the “max_conns” parameter was changed from 0 to 50.
As soon as Sirius provided this information to their customer, their customer was able to quickly resolve this issue.
Driving Cloud Ops with Evolven
According to Sean, when incidents were reported before Evolven, engineers had to review logs and manually check configurations logging into the production environment to determine a root cause.
Now with Evolven, Sirius can easily correlate an actual change to the incident the change was causing. Evolven makes it easy to identify the historical configuration value before a change, when the change occurred, and who made the change.
This information used when troubleshooting incidents, helps to significantly reduce the recovery time for Sirius’ client issues.
When it comes to our ever-changing cloud environments, Evolven makes it easy to track changes, proactively respond to risks, and maintain control.
The Evolven Change Analytics engine, based on patented machine learning algorithms, tracks all planned and unplanned changes, and detects the potential risk of changes early on, enabling IT Ops, DevOps, SRE and CloudOps teams to prevent performance, compliance, and security issues.
To learn more, set up a time to see Evolven in action here or follow @Evolven Software on LinkedIn to stay updated on the latest news in cloud, IT Operations, DevOps, and more.