Entering the Change Twilight Zone and How to Overcome It
Change happens. Change keeps the world on the move. In IT infrastructure and operations, change is important, enabling continuous improvement and proper functioning of IT services. In complex dynamic ecosystems, such as IT infrastructure, change happens a lot. On any given day applications and operating systems are being upgraded, patches are being installed, automated processes are running that alter files and system environments and configurations are manually being updated. Sometimes these activities are performed correctly and sometimes they're not. When they're not, the cause may only be identified when a business impact or even failure occurs.
While IT operations teams strive for 100% service availability, change can disrupt availability and performance. To avoid problems and downtime, I&O teams need effective change management to know what is happening in their environments, and managing these changes means managing much of the potential risk that change can bring with it.
The problem is that key parts of the IT operations suffer from limited visibility, laying the foundation for problems to follow.
Recently we had the pleasure of conducting a webinar with Steve Brasen, Research Director for the Enterprise Management Association (EMA). During the webinar, Steve explored some of the barriers to success of change management, and he introduced the idea of The Black Hole of Change, the critical areas where processes typically break down and where critical information is often lost due to lack of visibility. Steve states that: "Without establishing some level of control over IT changes, businesses are kept in the dark about the status and reliability of their IT."
We describe this area as the Change Twilight Zone. Just as Rod Serling said in his introduction to the original TV show The Twilight Zone, "It is the middle ground between light and shadow," so is the nature of the Change Twilight Zone, a hazy area that falls outside the visibility of IT Ops.
Change Slips into the Shadows
When looking at four key critical processes of IT service management, as defined by the ITIL framework, you can see that there is a wide opportunity in all of them for change events to be lost, and for the business to suffer.
- Change Management: Change implementation loop is left open, leaving possibility of failed changes
- Configuration Management: Unauthorized changes are detected too late – critical incidents
- Incident Management: Manual, inefficient incident investigation process – long MTTR
- Continual Service Improvement: Limited traceability of actions – compliance risk
No Visibility into the Change Twilight Zone
Vital information for IT operations drops into the shadows or what we have termed: The Change Twilight Zone, a hazy frontier outside the visibility of IT administration. You have probably found crucial parts of processes drifting into this zone.
- When a planned change is introduced, how do you know if it was successful and that it will not impact the environment?
- When systems are built into a baseline configuration in configuration management, how will you know if, when, and to what degree it drifts from that standard?
- After an incident is resolved, through Incident Management , how do you know what was fixed and can you be sure that the problem won't occur again?
- When ensuring service KPIs are being met with Continual Service Management, how can you be sure the personnel are actually performing required activities?
In the Dark About Change
Without setting a reasonable amount of control to changes critical processes, businesses are left in the dark about the actual standing and availabilityof the support infrastructure. Why? This is because today's dynamic IT ecosystems are extremely complex. The cause of a problem on one system may be something that was reconfigured or completely different on an unrelated system.
ITIL offers tools to support IT departments for handling change, and ITIL's Change Management process is meant to protect the IT service provider from undesired consequences of a change.
As defined by ITIL, the main objective of Change Management is to carry out changes beneficial to IT and the business, while minimizing possible disruption to the business. However in this process, each particular change requires formal authorization, from a change authority. As suggested in the recent Forrester Research report "Assessing Complexity in IT Operations" "an overly complex change process that calls for a lengthy chain of review and approval may deter employees from using it and consequently push the number of unauthorized calls in the name of expediency and effectiveness."
Changes Happen Faster
Against this background, you also have to keep in mind the rapid frequency in which environment changes occur. The more changes that occur, the more likely those changes will affect the environment. All of this results in systems moving from an optimally configured environment to being environments that have an increased risk of failure. The degree to which a system has diverged from a standardized or optimal state is often referred to as its configuration drift. The greater the configuration drift, the more likely there will be a business impacting environmental failure and the more difficult it is to identify the cause.
So the question is how will you manage one-off changes or changes that do not follow policy like this? Also, how can you discover and identify changes that have occurred in a network that may? It's simple to correct a change in one system. However, how can you validate your systems' configurations, and then update or correct any ad-hoc changes that were made? The problem is complex, and difficult to resolve.
Identifying these changes in a timely manner before it has impacted the application or soon after, reduces the risks around business continuity.."
Manually Monitoring Change is Impossible
It is simply impossible to monitor and evaluate every single configuration point in a support stack using purely manual processes. You would need a workforce that is the population of something like Cleveland (p. 478,403) just to perform the necessary system review.
That's just not practical. Existing IT support staff already have enough on their plates, with chasing down known problems, and really don't have time to carefully evaluate every single change that occurs on every single supported system.
Shedding Light on the Change Twilight Zone
So how can you actually remove the veil cast by the Change Twilight Zone over critical IT processes? Evolven uses breakthrough analytics to identify those specific changes and differences that are introduced across the environment and actually matter to the user, and could impact performance and stability of critical business applications. So, for IT this means that you can finally remove the shadow of the Change Twilight Zone, and confidently close processes and answer important questions.
- Validate the changes in your application deployments and software deployments, and know that they were implemented correctly, avoiding downtime.
- Monitor and proactively detect any unauthorized changes and configuration drift, preventing downtime.
- Investigate incidents by analyzing if the changes or re-configurations are the root cause of the incidents - cutting mean-time-to- resolution (MTTR).
- Use the information collected to audit any changes made - improving your ability to meet compliance requirements.