tl;dr Run our new tool by adding -javaagent:log4j-jndi-be-gone-1.0.0-standalone.jar to all of your JVM Java stuff to stop log4j from loading classes remotely over LDAP. This will prevent malicious inputs from triggering the “Log4Shell” vulnerability and gaining remote code execution on your systems. .
In this post, we first offer some context on the vulnerability, the released fixes (and their shortcomings), and finally our mitigation (or you can skip directly to our mitigation tool here).
Update (12/16/21): This post has been updated in line with new information regarding incomplete patches for CVE-2021-44228; CVE-2021-45046; weaknesses in the official mitigations; and the log4j release with updated fixes, 2.16.0.
Context: log4shell
Hello internet, it’s been a rough week. As you have probably learned, basically every Java app in the world uses a library called “log4j” to handle logging, and that any string passed into those logging calls will evaluate magic ${jndi:ldap://...} sequences to remotely load (malicious) Java class files over the internet (CVE-2021-44228, “Log4Shell”). Right now, while the SREs are trying to apply the not-quite-a-fix official fix and/or implement egress filtering without knocking their employers off the internet, most people are either blaming log4j for even having this JNDI stuff in the first place and/or blaming the issue on a lack of support for the project that would have helped to prevent such a dangerous behavior from being so accessible. In reality, the JNDI stuff is regrettably more of an “enterprise” feature than one that developers would just randomly put in if left to their own devices. Enterprise Java is all about antipatterns that invoke code in roundabout ways to the point of obfuscation, and supporting ever more dynamic ways to integrate weird protocols like RMI to load and invoke remote code dynamically in weird ways. Even the log4j format “Interpolator” wraps a bunch of handlers, including the JNDI handler, in reflection wrappers. So, if anything, more “(financial) support” for the project would probably just lead to more of these kinds of things happening as demand for one-off formatters for new systems grows among larger users. Welcome to Enterprise Java Land, where they’ve already added log4j variable expansion for Docker and Kubernetes. Alas, the real problem is that log4j 2.x (the version basically everyone uses) is designed in such a way that all string arguments after the main format string for the logging call are also treated as format strings. Basically all log4j calls are equivalent to if the following C:
printf("%sn", "clobbering some bytes %n");were implemented as the very unsafe code below:
char *buf;
asprintf( buf, "%sn", "clobbering some bytes %n");
printf(buf);Basically, log4j never got the memo about format string vulnerabilities and now it’s (probably) too late. It was only a matter of time until someone realized they exposed a magic format string directive that led to code execution (and even without the classloading part, it is still a means of leaking expanded variables out through other JNDI-compatible services, like DNS), and I think it may only be a matter of time until another dangerous format string handler gets introduced into log4j. Meanwhile, even without JNDI, if someone has access to your log4j output (wherever you send it), and can cause their input to end up in a log4j call (pretty much a given based on the current havoc playing out) they can systematically dump all sorts of process and system state into it including sensitive application secrets and credentials. Had log4j not implemented their formatting this way, then the JNDI issue would only impact applications that concatenated user input into the format string (a non-zero amount, but much less than 100%).
The “Fixes”
The main fix is to update to the just released log4j 2.15.0 2.16.0. Prior to that the release of 2.15.0, the official mitigation from the log4j maintainers was:
“In releases >=2.10, this behavior can be mitigated by setting either the system property
Apache Log4jlog4j2.formatMsgNoLookupsor the environment variableLOG4J_FORMAT_MSG_NO_LOOKUPStotrue. For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove theJndiLookupclass from the classpath:zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.”
So to be clear, the fix given for older versions of log4j (2.0-beta9 until 2.10.0) is to find and purge the JNDI handling class from all of your JARs, which are probably all-in-one fat JARs because no one uses classpaths anymore, all to prevent it from being loaded.
Then there is the fix for more recent versions of log4j (2.10.0 to 2.14.1), which is to set a magic Java system property (log4j2.formatMsgNoLookups) via the command line or weird XML shenanigans, or to add a magic environment variable (LOG4J_FORMAT_MSG_NO_LOOKUPS) to your Java processes (likely by reconfiguring your [systemd] service definitions). And you should be aware that both of these simply disable all ${} handling. Lastly, the real prior fix is was to ‘just update it’ to the new version (2.15.0) that defaults to the functionality being turned off in an overly complicated patch that… removes the formatMsgNoLookups handling, but sets the default to true? They apparently changed the handling so that lookup handlers are opt-in, enabled by XML-based configuration, a change so cumbersome I question how long it will last.
Update (12/16/21): As it turns out, there was an edge case left in 2.15.0 enabling denial-of-service attacks (CVE-2021-45046) — but also RCE? (see below) — against log4j message processing. Due to the removal of the formatMsgNoLookups/ handling, which otherwise would have prevented any lookup based logic from triggering, 2.15.0’s denial-of-service vulnerability cannot be mitigated with that setting (as it holds no meaning in that version).LOG4J_FORMAT_MSG_NO_LOOKUPS
Additionally, the formatMsgNoLookups/-based mitigation does not appear to provide protection in cases where the layout pattern is customized to include “ContextMap” lookups (e.g. via LOG4J_FORMAT_MSG_NO_LOOKUPS"${ctx:...}") and application/library code enables users to configure org.apache.logging.log4j.ThreadContext values (or custom org.apache.logging.log4j.core.ContextDataInjector and possibly org.apache.logging.log4j.core.util.ContextDataProvider implementations), such as via calls to ThreadContext.put().
Lastly, one of the more esoteric official recommendations that has since been removed from the log4j website, was the following: “Users since Log4j 2.7 may specify %m{nolookups} in the PatternLayout configuration to prevent lookups in log event messages.” Additional research on the official mitigations done by LunaSec has shown that “%m{nolookups} will only apply to the content being logged in the call to logger.info() and will not apply to the resolving of ${ctx:apiversion} which will contain our payload” (emphasis ours).
This is interesting because the explanation for the “new” vulnerability, CVE-2021-45046, is the following, which describes only a denial-of-service issue:
It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations. This could allows attackers with control over Thread Context Map (MDC) input data when the logging configuration uses a Pattern Layout with either a Context Lookup (for example, $${ctx:loginId}) or a Thread Context Map pattern (%X, %mdc, or %MDC) to craft malicious input data using a JNDI Lookup pattern resulting in a denial of service (DOS) attack. Log4j 2.15.0 restricts JNDI LDAP lookups to localhost by default.
Apache Log4j
The denial-of-service listed appears to be the potential for infinite loops whereby an controlled ThreadContext value self-references itself. Based on the stack trace provided by LunaSec, it appears that at least simple self-references are detected by Log4j, which raises a java.lang.IllegalStateException a type of RuntimeException that would generally raise until a catch-all exception handler. While this is technically a denial-of-service, had the exact conditions for it not also enabled code execution as outlined above, it would have only presented a risk to code that assumed there would be no uncaught exceptions.
None of these solutions are ideal because they all presume that everyone has a 100% handle on their dependency chains and that you won’t end up with a mix of log4j versions strewn across the subcomponents of apps segmented across multiple classloaders (looking at you, WAR files). The reality is that the node_modules-style of dependency embedding is not completely alien to the Java ecosystem (and Java has a host of left-pad type issues due to how its build systems resolve dependencies across multiple repositories). In addition to this, many apps and proprietary libraries and app servers possibly vendor-in log4j in weird ways that end up preferring their own versions over any shared version that might actually be updated by your ops team. All told, tracking down and updating every instance of a library like this in your application stack is a nontrivial process and potentially error prone as any one rogue log4j copy can potentially keep a service vulnerable.
A tool to mitigate Log4Shell by disabling log4j JNDI
To try to make the situation a little bit more manageable in the meantime, we are releasing log4j-jndi-be-gone, a dead simple Java agent that disables the log4j JNDI handler outright. log4j-jndi-be-gone uses the Byte Buddy bytecode manipulation library to modify the at-issue log4j class’s method code and short circuit the JNDI interpolation handler. It works by effectively hooking the at-issue JndiLookup class’ lookup() method that Log4Shell exploits to load remote code, and forces it to stop early without actually loading the Log4Shell payload URL. It also supports Java 6 through 17, covering older versions of log4j that support Java 6 (2.0-2.3) and 7 (2.4-2.12.1), and works on read-only filesystems (once installed or mounted) such as in read-only containers.
Update (12/16/21): Due to the way it works, log4j-jndi-be-gone will prevent any JNDI lookups, including Thread Context Map-based ones that still impact log4j 2.15.0, but it does not prevent the limited “denial-of-service” attack against that version under such configurations.
The benefit of this Java agent is that a single command line flag can negate the vulnerability regardless of which version of log4j is in use, so long as it isn’t obfuscated (e.g. with proguard), in which case you may not be in a good position to update it anyway. log4j-jndi-be-gone is not a replacement for the -Dlog4j2.formatMsgNoLookups=true system property in supported versions, but helps to deal with those older versions that don’t support it.
Using it is pretty simple, just add -javaagent:path/to/log4j-jndi-be-gone-1.0.0-standalone.jar to your Java commands. In addition to disabling the JNDI handling, it also prints a message indicating that a log4j JNDI attempt was made with a simple sanitization applied to the URL string to prevent it from becoming a propagation vector. It also “resolves” any JNDI format strings to "(log4j jndi disabled)" making the attempts a bit more grep-able.
$ java -javaagent:log4j-jndi-be-gone-1.0.0.jar -jar myapp.jarlog4j-jndi-be-gone is available from our GitHub repo, https://github.com/nccgroup/log4j-jndi-be-gone. You can grab a pre-compiled log4j-jndi-be-gone agent JAR from the releases page, or build one yourself with ./gradlew, assuming you have a recent version of Java installed.
