Looking for Log4j in All the Wrong Places
Deep in the weeds of application architecture, a somewhat obscure, but highly pervasive utility known as Log4j was being secretly subverted by the exploit Log4Shell. Suddenly Log4j, something of concern to Java developers was being discussed on major media and social networks.
The Log4Shell exploit forced all of us to start asking questions about the effectiveness of our monitoring tools. What was assumed to be safe was not. Today’s observability tools collect telemetry consisting of logs, metrics, and traces for the purpose of monitoring and debugging the performance of large-scale cloud-based applications. However, observability as defined today is not sufficient. The Log4Shell exploit doesn’t show up in the logs, metrics, and traces of observability.
Observability must be extended to connect to our business requirements and include a new dimension, an awareness of change and its impact across the entire IT environment. Only by embracing change, detecting the unknown unknowns, and providing the ability to search and detect the details of the most granular configurations across the IT environment can we ensure customer experience, minimize risk, and protect against exploits like Log4Shell and Spring4Shell.
The Log4j Exploit
Apache Log4j is a Java-based logging utility used by most Java developers. Log4j is everywhere. It has been downloaded millions of times and is in production at some of the largest businesses on the Internet including Apple, Google, and Microsoft.
Log4Shell, (the name of the exploit) abuses a feature in Log4j version 2, that lets an assailant initiate custom code to perform malicious actions on a targeted system. These actions include stealing private information, taking over the besieged system, and depositing malware on remote users interacting with the under-attack server. The Apache Software Foundation gave the vulnerability a 10 out of 10 score, due to its potential for widespread exploitation and the ease with which malicious attackers can exploit it. It is as bad as it gets. So, why isn’t it detected when using observability tooling?
Observability and The Missing Dimension
Readers of the seminal book about the multiple dimensions of perception, Flatland know that if you live in a 2-D world you can’t perceive the third dimension of height. Today’s observability as typically defined describes a 3-D world. But the actual world we live in consists of 4 dimensions. So, what is left out? The missing dimension describes how CI’s change over time, the potential risk these changes incur, and the additional context change adds to the other three dimensions. Without detection of how IT assets are currently configured now and how they change over time, we risk missing something as critical as the Log4Shell exploit.
The four-dimensional Change Observability solution from Evolven adds an aspect that observes and analyzes changes, who made them, when they occurred, if they were authorized, and whether they are the cause of a problem now or risk being one in the future. The awareness of future harm enables IT Ops and DevOps to transcend observability’s limited usage in reacting to problems and instead focus on preventing problems such as exposure to Log4Shell.
What is Commonly Attempted to Address Log4Shell
Many businesses are posting information about what they are doing to address the Log4shell issue to show users they are “working on it”. That work entails mostly manual labor searching endlessly for the vulnerable libraries. This is a very time-consuming and almost never-ending activity. It gets worse as new code is deployed as part of the CICD process adopted by many businesses that may inadvertently push new code into production along with the vulnerable libraries. With the current methodologies, it might take years for all instances of this exploit to be successfully patched.
Other enterprises have implemented commercial solutions for the detection of this and other vulnerabilities; however, for a number of different reasons the effectiveness of these solutions was limited. This included: the inability to scan frequently sufficiently, usage of caching which generated false positives, lack of flexibility in specifying file names, and slow vendor response in targeting new vulnerabilities.
Evolven’s Automated Vulnerability Awareness
Evolven can detect vulnerable instances of Log4j in four different ways:
- An entire disk scan can be run to find not only in-use versions of Log4j but also ones that might be used at a future time and thus prevent tomorrow’s problems. See figure 1. A built-in capability to control how much resources are used by the scan prevents potential performance impacts.
Figure 1: Sample Inventory of Log4j instances discovered
- Custom collections can be set up where Evolven only searches configuration items (CIs) contained in a user-defined collection for Log4j (or any other CI of interest). These can be name or path matches or more sophisticated approaches matching for example checksums or classes. See figure 2.
Figure 2: Sample Custom Collection
- Evolven can detect the running applications and trace the location of the Log4j libraries that are currently in use.
- When Evolven is used in pre-production, integrated with deployment automation tools such as Jenkins it can scan the release content before it is deployed to prevent new releases of vulnerable libraries from being pushed out and reinfecting production.
Evolven allows users to create a set of rules defined in a policy specifying the desired state of an environment configuration, in this case, one that has the non-vulnerable, patched version of Log4j (see figure 3).
Figure 3: Sample Policy
In addition to automated detection of risks and anomalies, you can provide Evolven-specific environment checks that need to be executed and observed with high priority. Any detected mismatch of an environment state against a policy rule is reported to users as a non-compliant change or non-compliant configuration. Risk and compliance personnel can use policies to ensure regulatory compliance or compliance to internal organizational standards, avoid common issues with a well-known root cause, and drive operational best practices. Evolven automates the checking process that otherwise would be manual.
Evolven uses AI ML analytics to auto-reconcile actual changes of Log4j detected by Evolven with approved change requests and deployments. Using analytics-based correlation and clustering, Evolven also identifies unauthorized changes that led to the deployment of Log4j (definitive and potential) that need human review and escalation, providing a review dashboard for click-through access to all details.
The change and configuration data captured by Evolven is available for easy ad-hoc search or programmatic access and utilization. These queries enable the user to understand the actual IT environment better and uncover unexpected inconsistencies or potential trouble spots such as Log4j and prevent impact.
To facilitate resolution of the Log4j issue, service desk tickets can be created by Evolven advising subject matter experts of the location of undesirable CIs such as Log4j that must be updated. In addition, integration with tools such as ServiceNow, PagerDuty, and Jenkins can be used to drive automated remediation through the workflows within your existing toolchains.
Evolven’s four-dimensional Change Observability solution makes systems safer. It closes the gap between application security tools utilized by SecOps of Infosec and observability, reducing the time it takes to address a vulnerability. It immediately detects any existing afflicted systems using policies to define compliant environments. This includes detecting the Spring4Shell variant as well as vulnerable Log4j libraries. Certainly, there will be more exploits sent to prey on vulnerable libraries and other assets. The ability to quickly find new vulnerabilities in complex enterprise environments is a challenge that Evolven can effectively help with. Evolven is instantly aware of the library locations for all running applications, detecting running versions of log4j. It can also search the IT environment without impacting performance (utilizing a user-defined threshold setting for max CPU usage) to report on policy mismatches as non-compliant.
Using topology and AI machine learning, Evolven’s root-cause analysis and early detection of change can instantly identify all applications that have been or could be impacted by this exploit and rank them by risk to facilitate the prioritization of remediation efforts by the security team to prevent impact.
With its integration with ITSM and other automation tools to initiate the push of compliant versions of Log4j into production, Evolven raises the bar for observability, closing the loop from detection to resolution.
For more information, contact Evolven to speak with one of our experts.