Manufacturing Happiness? How a Customer Improved Stability and Radically Cut Unplanned Changes
In my travels and business encounters I get to meet a lot of different IT professionals and see how many different data centers function. So I am continually surprised when I see IT operations struggling with stability and maintaining performance. This state of affairs alone doesn't hit me so hard, but that for many in I&O, they have come to just accept this situation as a norm and simply bear with this daily struggle, without even seeing how to improve or enhance the state of their data center infrastructure management.
Financial Services Customer Struggling with Agile Application Deployments
I saw this initially at one of our customers that provides key financial services. They have a diverse customer base depending on their services, including some major corporations. Our customer is expected to rapidly address very dynamic market requirements, making the agility of development and operational processes a critical success factor for their services.
As we know today though, agility comes at a price, where some of the changes implemented on the fly might impact stability. So as releases and updates became more frequent, they had to make a lot of configuration changes to the environment, without a firm configuration management effort in place.
Downtime and Instability: A Necessary Evil?
Despite experiencing periodic stability issues, they accepted this phenomenon as a necessary evil, and maintained this approach not having the time to consider the consequences of ongoing configuration changes.
For example, look at this issue that came up came after a major upgrade to their environment. Users suddenly were reporting that they could see other user private financial data. This was sporadic and could not be reproduced, making it difficult to pin down the cause and even harder to solve. After days of investigating the incident, they discovered that one of the patch files was not installed properly on one particular server, while all the other servers were working fine.
IT Operational Stability is not Enough: Proactive Change Management & Configuration Management
When we met the IT Director, he was focused on maintaining operational stability, and reducing performance dips. So, I introduced him to a whole new approach to managing IT, by presenting the idea of collecting configuration changes, analyzing them for critical impact, and making the team aware of the corollary effects of configuration changes. This approach had never occurred to them before.
This customer started to use the Evolven Configuration Analytics software to track changes introduced during overnight maintenance. The operations team implemented Evolven to ensure that changes are deployed correctly. The plan was every morning they would verify that all the changes moved to production during the night were consistently applied on all the target servers. Also all the new changes had to be replicated to a disaster recovery environment, which needed to be validated as well for consistency. Just as for many environments, they run multiple load balanced servers to support the daily workload.
Enhanced Visibility Rapidly Reduces Daily Changes
When the Evolven application, began to e-mail change management reports every morning across the organization, the operations team was surprised to discover the amount of daily changes had rapidly decreased. Following a quick investigation into the issue, they found that developers, infrastructure administrators and other application stack stakeholders simply avoided unnecessary changes since with Evolven's configuration monitoring capabilities they would become visible and raise questions.
Over the course of a few months, this customer reported that the amount of unexpected changes had dropped by 90%! This result came even though no formal process, workflow, change committee had been added to their management approach. The only enhancement to their mode of operations was that a daily change report was distributed.
Influencing IT Behavior: Evolven Configuration (Psycho) Analytics
We saw that with the technical capabilities of Evolven Configuration Analytics, they could become aware of change inconsistency. This was augmented by the transparency applied to organization behavior. All this proved to be a huge bonus to our customer, and an unintended benefit to our product offering.
Now I'd like our Evolven engineers to start a crash course in IT psychology. Who knows? Maybe the next version release of Evolven Configuration Analytics will also offer a way to make you happier.
Find out More
Learn more about how Evolven's groundbreaking Configuration Analytics solution takes on the dynamics and complexity of the modern data center and cloud, in a way that was never available before.