Protect Your AWS Environment Beyond Patching Log4j

Check out crucial strategic lessons overlooked by enterprises dealing with the recently reported Log4j vulnerability.

Unless you’ve been under a rock or otherwise off the cybersecurity grid, you probably heard of the serious vulnerability in the Apache Log4j library. In a nutshell, for versions Log4j2 2.0-beta9 through 2.12.1, and 2.13.0 through 2.15.0, this vulnerability creates a situation in which, according to the NVD, “an attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled.”

It’s important to understand this vulnerability because the Log4j library is so ubiquitous the event has been referred to as a realization of the August 2020 xkcd comic:

Source: https://xkcd.com/2347/

Widespread use of the Log4j library has made software vulnerable on a massive scale – with subsequent massive impact on a wide range of vendors.

The technical aspects of the vulnerability have been thoroughly analyzed so we won’t take the time to do so here. For an excellent technical review of the vulnerability, check out the Log4j post by our colleagues at Mitiga as well as their useful list of related technical resources.

We do feel the need to discuss the general, and probably more strategic, lessons to learn from this incident. Even though a very serious cybersecurity incident, the Log4j vulnerability is not the only one that will be discovered and likely not even the only one placing infrastructure around the world at grave risk as we speak.

The approach many may take to this incident – “I need to upgrade to Log4j 2.17.0 and everything will be fine” – can be visually described as:

From "Gunshow" by K.C. Green

You may have patched up the issue at hand but are ignoring other threats that put your infrastructure at risk. If you recognize that such an event will probably happen again, you can use it to improve your cybersecurity strategy and be better positioned next time one creeps up on the industry.

Log4j and your AWS security strategy

Other than the AWS services affected by this finding (which you can keep track of through AWS bulletins like this one), the Log4j issue puts a spotlight on other aspects of AWS security strategy that are on your side of the Shared Responsibility Model. Among these are the permanent credentials and third-party access to your AWS resources.

Permanent AWS credentials: What are they good for?

AWS credentials can be stored in environment variables and/or local files on computing resources and workloads. These storage locations have not eluded criminals who design malware. Christophe Tafani-Dereeper, in his analysis a few years ago of cloud security breaches and vulnerabilities, noted that “static credentials remain the major initial access vector” for breaching cloud environments.

As described by Greg Linares, the weaponization of this specific vulnerability for the purpose of harvesting AWS credentials from such locations didn’t take long:

Here are the AWS variables I am seeing being targeted in the wild by #log4j exploits at this time (even without rce):

AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_SHARED_CREDENTIALS_FILE
AWS_WEB_IDENTITY_TOKEN_FILE
AWS_PROFILE
AWS_CONFIG_FILE
AWS_ACCESS_KEY_ID

— Greg Linares (Laughing Mantis) (@Laughing_Mantis) December 11, 2021

And this use of it was apparently rather popular:

Most common attempted env stolen have been: 
AWS credentials
Docker creds
Hadoob installs

Ransomware payloads: 4%
Cryptominer payloads: 22%
Info stealer payloads: 61%
Unknown payloads: 13%

JDNI gadget payloads: 23%

Spray N Pray Headers: 70%
Single Attempt: 21%
Other: 9%

— Greg Linares (Laughing Mantis) (@Laughing_Mantis) December 15, 2021

Identity impersonation is especially dangerous when harvested credentials are permanent, such as IAM user access keys. For this reason, the Log4j vulnerability discovery is a timely reminder for security practitioners to ask themselves: “What are permanent credentials even good for?” The answer: typically very little.

Of course, it’s convenient to use IAM users and access keys since setting up an effective replacement can be a bit troublesome. That “bit” makes it hard enough for most organizations to stop using permanent credentials as they can’t see the benefit of removing them. We argue, though, that the potential fallout from being hacked due to a vulnerability such as permanent credentials far outweighs the slight inconvenience of replacing them.

This aspect of credentials management is also worth your time because many good solutions exist today for just about every use of IAM users – and specifically access keys. Here are a few:

• Are you using access keys for CLI? Then move to using AWS SSO and, if not possible, consider adopting saml2aws and using identities managed elsewhere to generate temporary credentials for use as needed. If you must, use aws-vault to store credentials in the OS’s secure keystore.
• Are you allowing users to access the AWS console with IAM users? Switch to federating identities from your identity provider or adopt AWS SSO for easy multi-account access management.
• Do you provide third party organizations with permanent IAM user access keys to access resources in your environment? Stop everything you’re doing and switch to a service that you can give access to using a third party role with a properly set up external ID – and then disable those permanent access keys.

Aidan Steele recently posted his thoughts on exactly this topic:

I think there's no need for AWS IAM users today. So I made a PoC to "prove" it. First here are the general use-cases for creds

* AWS SSO works great for humans 👍

* Roles work fine inside AWS 👍

* Federation works fine from other clouds 👍

* Raspberry Pi in your closet ❓
1/4

— Aidan W Steele (@__steele) November 5, 2021

Aidan goes on to explain that even for the obscure case of running a Raspberry Pi in a closet, he can suggest a good replacement (and provides open source to assist in implementing it).

Bottom line: It may take some work to fully eliminate use of IAM user access keys or IAM users in your environment. But if you have the slightest fear of AWS credentials being harvested – and the Log4j incident is solid proof – eliminating permanent ones is a good investment of your time and effort. On a more general note, avoid using any kind of static credentials as much as possible.

Third-party access: Do you have it under control?

Supply chain security is an extremely important part of your overall security posture – and doesn’t always get center stage. Typically, only a handful of legal and technical controls are available for ensuring that access delegated to third parties is not compromised or mismanaged. So it is crucial that you properly manage and monitor such access since it could be a clear attack vector through which a malicious actor can gain unfettered access to your environment.

As mentioned, there is a proper way to give external organizations access to your AWS environment. That said, even if you grant access legitimately and correctly, once the third party is hacked, the access to your environment can be abused. The Log4j library event showed that such threats are no longer theoretical. It’s very likely that some vendors that have legitimate access to your cloud resources were exposed to the reported vulnerability – potentially allowing attackers to use their permissions to your environment. How do you keep things under control?

First, keep track of vendor release notices relating to their current status regarding this vulnerability. If you can’t find such a notice, don’t be embarrassed: contact each vendor and ask point blank if they were exposed and how they are actively mitigating the exposure. Second, the entire event should be an alarming reminder for you to place all third-party access under close scrutiny and continuous monitoring. The obvious next question to ask yourself is whether you are aware of all third parties that can currently access your AWS environment. Do you know exactly what actions they are permitted to perform? Is that access really necessary for them to perform the services they provide? In fact, are they even active – or is the access a relic of an old project that no one took the time to remove?

Where does Tenable Cloud Security fit in?

As a vendor of a comprehensive cloud security solution, we work extremely hard to provide proper controls for such strategic issues. Using our SaaS technology we constantly and continuously review your environment’s configurations, permissions and logs - allowing you to easily gain insights on the threats arising from such configurations. Let’s see how.

Detection of static AWS credentials

Leveraging the Tenable Cloud Security platform’s unique view into identity intelligence, you gain immediate visibility into the existence of static credentials in the form of IAM user access keys:

More specifically, the platform delivers immediate results on which identities have been inactive for a significant period of time. Such findings are likely a “quick-win” to discard.

Permissions rightsizing

Having access to each identity’s permissions and the logs that indicate what actions the identity is actually performing, and on which resource, allows the platform to automatically generate a per-identity least-privilege policy recommendation to perform relevant business functions.

Proactively rightsizing the permissions of the identities with access to your cloud infrastructure is extremely effective in reducing the blast radius from any type of compromise. This is also true with regard to exposure to the Log4j vulnerability.

For example, let’s say one of your workloads runs on an EC2. While you know that this EC2 requires access to S3 buckets, you don't quite know which and for what purpose. As a result, you decide to attach the “AmazonS3FullAccess” or even “S3ReadOnlyAccess” policy to the IAM role that the EC2 assumes. Albeit a very bad practice, we all know this happens more often than we care to admit.

If it runs software with a vulnerable version of the Log4j library, the EC2 could have vast access to all S3 buckets within the environment – with potentially catastrophic implications for the availability of mission critical data and/or access to confidential or sensitive information. Kat Traxler did a great job of articulating how Log4j can be used to exfiltrate credentials from EC2 instances - allowing attackers to impersonate the EC2 and use its permissions in the attacked environment. By proactively limiting the EC2 permitted actions to the bare business functions, the potential fallout from such a breach would be minimal.

Of course, proactively limiting permissions continuously and at scale is easier said than done. Tenable Cloud Security fully supports this process by automatically generating right sized policies and allowing AWS environment administrators to apply them instead of over permissive ones.

Anomalous behavior alerts

Analyzing CloudTrail logs allows Tenable Cloud Security to detect malicious anomalous behavior in your environment.

In a related exercise, we used the Pacu exploitation framework to demonstrate various elements of the cyber “kill chain'' that an adversary would use to wage a campaign on a victim’s environment. The exercise identified multiple behaviors that a threat actor would be inclined (if not compelled) to perform in the attacked environment and how the accurate log analysis allows the person managing the breached environment to detect malicious activity and nip the campaign in the bud.

For example, an attacker could be inclined to perform privilege escalation on itself by attaching an overly permissive policy on a role that it is allowed to assume. If this operation has not occurred before, Tenable Cloud Security will flag it as anomalous and consequently immediately raise an alert.

Keeping third parties under control

Right-sizing policies and keeping tabs on identities is crucial. As described, doing so in the case of third party identities is ultra-crucial because, if these external entities are compromised, the repercussions can be far reaching – and beyond your immediate control.

Typically, permissions provided to third parties are as per their request. Unfortunately, administrators rarely question the vendor request and actual need. For example, as part of our work with many vendors, we received a CloudFormation template that provisioned a role with an inline policy that allowed the vendor to use “iam:AttachRolePolicy” on “*”. This specific permission may seem harmless, but it actually allows the subject to privilege escalate their permissions to admin. During our analysis of actual usage by the third party, we were astounded to discover that the permission was excessive and not necessary for the regular business function the third party served – so it could have been removed.

Tenable Cloud Security also helps you easily find long forgotten roles allowing third party access that haven’t been used for a significant period of time.

 

Summary and a resolution

The Log4j issue was a serious, high-profile incident that required immediate attention by affected parties. We encourage you to use any tool at your disposal to detect and remediate exposure to the Log4j vulnerability by removing unnecessary use of the vulnerable components or upgrading to a safe version. However, keep in mind that such actions are merely tactical. Let Log4j, which is by no means a one-off event, remind you of the bigger picture. As Haroon Meer ironically puts it:

Good news:
Log4j is the only library you use that’s been trivially vulnerable for about a decade.

— haroon meer (@haroonmeer) December 20, 2021

Other severe and ubiquitous vulnerabilities are surely lurking in the shadows – and will surface one day. Let Log4j spur you to strategic action: take a close, deep look at your security initiatives and evaluate how – and, perhaps more importantly, how effectively – you are managing and controlling your AWS environment security, and take resolute steps to make it better.