A major bug in a widely-used piece of open source software called Log4j has thrown the IT world into pandemonium. The hole was not even made public a month ago (as of this writing), and yet it’s already been classified by Internet security analysts as among the biggest vulnerabilities in cybersecurity history.
By some estimates, for instance, some 93 percent of enterprise cloud computing environments around the world are affected. According to sources quoted in the Financial Times, as of Dec. 14, more than 1.2 million cyberattacks (at a rate of as much as 100 attacks per minute) had been observed—with no likely end in sight for, according to these sources at least, “months to come.”
As the industry scrambles to plug the gaps, Google has upgraded one of its security tools to help open source developers hunt down the vulnerability and others like it.
The panic around this new bug, which has been dubbed Log4Shell, is primarily down to its sheer pervasiveness. The tool it targets is used in a huge number of applications and the list of affected services is a who’s who of leading tech companies. The nature of the vulnerability also makes it relatively simple for attackers to run code that lets them take complete control of targeted devices.
“Fuzzing” bombards programs with random inputs to force errors that reveal security vulnerabilities. Google has now updated its fuzzing tool to track down Log4Shell—the name for the vulnerability in Apache’s ubiquitous Log4j code.
When the bug was disclosed on December 9, it was given a severity score of 10 out of 10 by the Apache Software Foundation (ASF), the non-profit whose volunteers develop Log4j. While ASF has released patches that remedy the flaw, it could take months or even years to find and fix every instance. The incident has reignited debates around how reliant today’s critical computing infrastructure is on open source code, which is typically maintained by small teams of under-resourced developers working for free in their spare time.
While’s there’s no silver bullet, Google’s open source security team thinks one potential solution is to provide open source developers with better tools to hunt for bugs. One promising approach is called “fuzzing”, which bombards programs with random or intentionally invalid inputs to force errors that reveal stability issues or security vulnerabilities. Google has provided a free continuous fuzzing service called OSS-Fuzz to major open source projects since 2016. And now the company has collaborated with security firm Code Intelligence to update the tool so that it can hunt down Log4Shell and other vulnerabilities that rely on the same mode of attack.
“We're trying to expand the tools capabilities to find similar sorts of vulnerabilities so that more developers can secure their own code bases,” says Google’s Jonathan Metzman. “The developer doesn't really need to think about how the fuzzer is detecting them, it just does it for them.”
Given the complexity of modern software, developers don’t have time to build every module from scratch and so they often rely on open-source software components like Log4j. As the Java software was designed, Log4j keeps records of activity within applications which can help keep track of errors and performance problems. As an example, clicking on a dead link or typing in a wrong URL, which typically produces a 404 error message, are among the activities that Log4j keeps track of for a web domain’s system administrators. Log4j thus performs a critical function for many types of software, which is the reason for the tool’s ubiquity.
Building a fix is only the first step in getting the crisis under control; now developers and system administrators across the industry have to scour their code for instances of the bug.
But last month, engineers at Chinese tech firm Alibaba discovered they could get Log4j to log a message containing a string of malicious code that triggers a connection to an external server under their control. Once this connection is established the attacker can remotely run whatever code they want on the targeted system.
The Alibaba researchers notified ASF as soon as they found the bug and gave them time to create an update that deals with the vulnerability before disclosing it. Since then, two more harder to exploit vulnerabilities in Log4j have also been uncovered and patched. But building a fix is only the first step in getting the crisis under control; now developers and system administrators across the industry have to scour their code for instances of the bug.
“The way software is built today is very much layer upon layer upon layer,” says Gary Gregory, a member of the ASF project management committee responsible for Log4j. “So developers, applications, companies may not even realize whether or not they're using certain software.”
Even once they’ve found the bug, some companies have stringent processes governing how they can make updates, which may delay their ability to resolve the problem, says Gregory. There are also likely to be companies relying on older software that is no longer supported or whose vendors are now defunct.
And while the updates put out by ASF have completely removed the functionality that allows the Log4j tool to connect to an external server, Gregory points out that it’s a generic capability baked deep into Java. “We've just ripped that out,” he says. “But I'm betting people will look at other software programs for the same type of vulnerability.”
This functionality is exactly what Google’s updated fuzzer looks for, though, which means that as well as detecting Log4shell it should also be able to find other bugs that use the same mode of attack. The solution is not a replacement for more formal security testing, but OSS-Fuzz has already discovered more than 7,000 vulnerabilities since its launch.
Metzman says the group has plans to further expand the kinds of bugs OSS-Fuzz can detect as well, and believes the approach can be a powerful tool for under-resourced open-source teams. “The randomness of fuzzing explores a lot of program states,” he says. “It’s very good at reaching these kind of obscure states that are deep into the program and finding vulnerabilities in there.”
It's hard to say whether this kind of tool could help catch the next major vulnerability, says Metzman. But he points out that researchers have shown that fuzzing could have detected the Heartbleed bug that set the internet on fire in 2014. If we want catch the next major vulnerability though, we need to provide more support to open source developers across the board, says John Hammond from security firm Huntress Labs. His company also released a tool to test for Log4Shell in the aftermath of the disclosure, but he thinks an more even important goal is to increase education and awareness of security issues in the open source community. With any luck, this crisis will provide the impetus, he says.
"Maybe it will shine a spotlight on the fact that we need a little bit more love for the open source community," he adds. "Because a lot of our modern world and technology certainly stands on their shoulders."
Edd Gent is a freelance science and technology writer based in Bengaluru, India. His writing focuses on emerging technologies across computing, engineering, energy and bioscience. He's on Twitter at @EddytheGent and email at edd dot gent at outlook dot com. His PGP fingerprint is ABB8 6BB3 3E69 C4A7 EC91 611B 5C12 193D 5DFC C01B. His public key is here. DM for Signal info.
