Linguistic Lumberjack: Attacking Cloud Services via Logging Endpoints (Fluent Bit - CVE-2024-4323)

Tenable Research has discovered a critical memory corruption vulnerability dubbed Linguistic Lumberjack in Fluent Bit, a core component in the monitoring infrastructure of many cloud services.

Key takeaways

• Fluent Bit is a logging utility heavily used by all major cloud providers.
• Tenable Research discovered a critical vulnerability dubbed Linguistic Lumberjack (CVE-2024-4323) within Fluent Bit’s built-in HTTP server that could potentially allow for denial of service, information disclosure, or remote code execution.
• The vulnerability was introduced in version 2.0.7 and exists thru 3.0.3. It is fixed in the main source branch and is expected in release 3.0.4.
• The issue can be resolved by …
  • … upgrading to the latest version of Fluent Bit.
  • … appropriately limiting access to the vulnerable endpoint.

An overview of Fluent Bit and of the Linguistic Lumberjack vulnerability

Fluent Bit is a lightweight, open-source data collector and processor that can handle large volumes of log data from various sources. It was designed to be highly scalable and easy to use, making it an ideal choice for collecting and processing logs in cloud-based environments. The project boasts upwards of 3 billion downloads as of 2022 and continues to see more than 10 million deployments each day. It is used heavily in almost every major cloud provider’s infrastructure.

Fluent Bit [...] is used heavily in almost every major cloud provider’s infrastructure.

Tenable Research has discovered a critical memory corruption vulnerability in Fluent Bit’s built-in HTTP server that could potentially result in denial of service (DoS), information leakage, or remote code execution (RCE). The issue was reported to the project’s maintainers on April 30, 2024. Fixes were committed to the project’s main branch on May 15 and are expected in the release of version 3.0.4. The issue has been assigned CVE-2024-4323.

^{Image generated via ChatGPT 4o / DALL-E by Nick Miles}

Technical details

While investigating a security flaw in a cloud service (details of which are still pending public disclosure), Tenable researchers discovered that they were able to access a variety of metrics and logging endpoints internal to the cloud service itself. Among these endpoints were a number of Fluent Bit instances. Access to these endpoints alone could result in cross-tenant information leakage, but after testing Fluent Bit in a separate, isolated environment, the researchers happened upon the memory corruption issue detailed here.

Fluent Bit’s monitoring API is intended to allow administrators or other users to query and monitor information internal to the service itself. For example, HTTP endpoints exist to indicate service uptime, plugin metrics, health checks, etc.

Among these endpoints are /api/v1/traces and /api/v1/trace, which allow end-users to enable, disable, or retrieve information about configured traces. Regardless of whether or not any traces are configured, it is still possible for any user with access to this API endpoint to query it.

^{Example request from https://docs.fluentbit.io/manual/administration/troubleshooting}

During the parsing of incoming requests for the /api/v1/traces endpoint, the data types of input names are not properly validated before being parsed. They are incorrectly assumed to be valid MSGPACK_OBJECT_STRs. 

By passing non-string values, such as integers, in the “inputs” array of a request, it is possible to cause a variety of memory corruption issues. During the flb_sds_create_len() function that assigns the input_name variable, the values passed when an integer is supplied happen to be a pointer to the beginning of the inputs array and the raw integer value as the “size” of the value.

^{Snippets of Bug Locations}

For some examples, passing … 

• … large integer values (or a negative value) can cause a crash due to a “wild copy” in a later call to memcpy() when it attempts to write to protect memory.
• … negative values between 1 and 16 can cause heap overwrites of adjacent memory. These will later result in a similar “wild copy” situation due to conversions between int, size_t, and uint data types.
• … integer values not large enough to crash can cause disclosure of adjacent memory to the client making the request.
• … a value of “-17” will cause a crash due to a null pointer dereference after a failed malloc() of zero later in the code.
• … smaller and more targeted integer values can trigger a variety of stack corruption and other memory corruption issues, such as corrupted chunks and broken links in the heap management mechanisms.

In their lab environment, the researchers were able to reliably exploit this issue to crash the service and cause a denial of service scenario. They were also able to retrieve chunks of adjacent memory, which are returned in the HTTP responses. While this is generally unlikely to reveal anything other than previous metrics requests, the researchers were able to occasionally retrieve partial secrets during their testing, indicating that this issue could potentially leak sensitive information.

As for the remote code execution possibilities of this issue, exploitation is dependent on a variety of environmental factors such as host architecture and operating system. While heap buffer overflows such as this are known to be exploitable, creating a reliable exploit is not only difficult, but incredibly time intensive. The researchers believe that the most immediate and primary risks are those pertaining to the ease with which DoS and information leaks can be accomplished.

This issue was ultimately fixed by properly validating the data types of values in the “inputs” array sent to the “traces” endpoint. Further details can be found in the following commit: https://github.com/fluent/fluent-bit/commit/9311b43a258352797af40749ab31a63c32acfd04

Proof of concept: Denial of service

Request to traces endpoint:

# python3 -c 'print("{\"output\":\"stdout\", \"params\": {\"format\": \"json\"},\"inputs\":[\"" + "A"*8 + "\"," + str(0xffffffff) + ", \"" + "B"*500 + "\"]}")' > test
# curl -v http://<target>:2020/api/v1/traces/ -H "Content-Type: application/json" -H "Expect: " --data "@test"

Crashed instance:

The above debugger output indicates a heap-buffer-overflow, but ultimately the crash is caused by an attempt to write to a protected memory region.

Mitigations

As previously mentioned, a fix for this issue is in the project primary branch. Linux packages can be found here. 

At the time of this writing, a general announcement has not been made on the Fluent Bit website and a formal release has not been generated by the maintainers despite the information regarding the vulnerability being committed to the public repository.

If deployed in your own infrastructure and environments, it is recommended to upgrade to the latest version as soon as possible. If upgrading is not possible, it is recommended to review any applicable configurations in your environment that allow access to Fluent Bit’s monitoring API to ensure that only authorized users and services are able to query it. If unused, be sure to disable this endpoint.

If you rely on cloud services that are known to make use of Fluent Bit, we recommend reaching out to your cloud provider to ensure that updates or mitigations are deployed in a timely manner. With regards to usage by major cloud providers, Tenable notified Microsoft, Amazon, and Google of this issue via their respective vulnerability disclosure mechanisms on May 15, 2024 so that they could begin their internal triage processes.

Conclusion

Regardless of whether your company makes use of cloud-based services or hosts its own infrastructure, it’s almost a guarantee that there’s some variety of logging and monitoring utilities in use. While these utilities are known to contain lots of juicy information for attackers, it’s important to realize that information leakage isn’t the only thing to be concerned with. It’s essential for organizations to update these utilities regularly, adopt adequate defense-in-depth measures, and utilize the principle of least privilege to ensure these tools cannot be misused by attackers.

More information can be found in the Tenable Research Alert TRA-2024-17.