Event Analysis Training – More SSH Worm Analysis

I recently observed a SSH worm in progress at one of the research sites running our suite of products. I was looking into a spike of SSH events that had been alerted on by the Log Correlation Engine’s stats daemon. Filtering on the remote IP address (that came from the 240.0.0.0/8 Class A address space) that was causing the anomalies, displayed this screen:

This particular demo site is monitored by Snort, the Tenable Network Monitor (that logs TCP sessions) and a variety of host logs from a few servers. In the above graph, the remote IP in question generated a few ping attempts around 2:00 PM and again about 20 hours later, a series of sessions on port 22 swept the network. There are several items of interest in this graph:

• The bulk of the SSH scans are captured in the TNM-TCP_Session_Short event. Most of these are TCP connections on port 22 that last less than a few seconds.
•  For the few systems we are getting log files from, there is a stray firewall event (from a Linux server running IP Tables) as well as several SSH logs for closed connections and failed password attempts.
•  There are 1951 Crowd Surge events (previously blogged about here) back to the host performing the scanning. That means for whatever reason, not only has this host started to scan the network, but some of the hosts scanned have connected back to it. Note that these events do not correspond to specific servers. The actual number was about 20 compromised hosts.
•  There were three “TNM-Long_TCP_Session_Many_Hours” events, indicating that the host performing the scan had three connections that lasted longer than an hour. This could indicate command and control software, a stalled TCP session or, in some cases, network equipment that has proxied the connection and is trying to keep the session open.

I felt it was worth zooming into the activity graph and displaying the last six hours instead of the last 24 hours to get a better sense of the distribution of events over time. The results are shown below:

Clearly, as seen by the Command Session SSH logs, there is a quick increase and drop off of SSH sessions. This is followed immediately by a set of continuous “Crowd Surge” events at a consistent rate. It is as if the hosts that have been compromised have consistently and methodically attempted to connect back to the scanning host.

Quick analysis of the three hosts that have been involved with the “long” TCP sessions with our remote scanner are shown below:

Host A:

Host B:

Host C:

These filters were all done with just the IP address of the target host on the defended network and not the remote host that was doing the SSH scanning. One thing that stood out was the Snort ICMP event that occurred late at night. If you refer back to the original graphs, there was no activity occurring during the night in question.

The specific log shows a new IP address, in this case from the 94.0.0.0/8 Class A address space.

Filtering on this new IP address for the last 24 hours shows the following graph:

This IP address performed a network scan during the evening. This seems very coordinated since it took place between the initial ping from the remote scanning in the 240.0.0.0/8 Class A and the subsequent SSH sweep.

This could indicate a manual effort or an automated botnet looking for new hosts to expand to.

What is also interesting in this scan is that for many of these hosts, this Snort ICMP event occurred on them for the very first time. “Never Before Seen events”, previously blogged about here, can indicate a wide variety of things.

In essence, each of these hosts has not had this activity occur in the past. It is important to understand why. It is unlikely this is the first time a network of this size was ping swept. I would expect that there was some sort of firewall rule change that has opened up ICMP packets to the network.

Conclusion

If this were my network, I would perform a closer investigation of the three hosts involved with the long TCP connections. A vulnerability scan (from a credentialed Nessus scan or passive vulnerability data) was not available but, if it was, we could determine what version of SSH was running to see if we are indeed vulnerable or if this could be a zero-day exploit.

Second, I would also watch the network traffic of each host that communicated back to the scanner from the 240.0.0.0/8 network. The list of IP addresses involved with the Crowd Surge events can be created as a static asset list in the Security Center and then audited for any common vulnerabilities or traffic patterns.

This blog entry was one of many that have been written in our “Event Analysis Training” series of blog entries. If you are running a SIM, an IDS or performing some sort of NBAD monitoring, these blogs offer many tips for working with different types of logs and suspicious activity.