Amazon VPC Traffic Mirroring

Importance of Network Visibility

The first means to collect security-relevant information at Cloud SIEM Enterprise (CSE) was our Network Sensor. It was built to analyze network traffic and provide visibility beyond traditional SIEM's down to the network-level. Beyond organizing packets into flows, the sensor supports more advanced features such as decoding of common protocols, file carving, SSL certificate validation, OS fingerprinting, clustered deployment and more.

Since first introducing the Network Sensor, we have released a Log Sensor, Windows Sensor, as well as support for Cloud-to-Cloud ingestion. While logs can illuminate what users and processes do on a higher-level, completely relying on log-driven frameworks for security monitoring limits the ability to detect and understand security incidents. Much like a forensic science expert would need access to the complete crime scene in order to tell you exactly what happened, a security analyst would want to see data throughout the OSI model to get a clear picture of the implications of a threat or ongoing attack.

When contrasting network data and log data, the major difference that comes to mind is freedom of implementation. A log is defined by a developer based on expected (and in some cases unexpected) use of an application or service. There is no widely accepted standard that tells us what format to use or what to log. In contrast, network communication protocols like TCP/IP are well-understood and universally adopted.

Why is this important? A useful thought exercise for developers, engineers and security analysts is answering the question, "How will my applications react to an attack?" Going from crash to exploit is a common path in many of the vulnerabilities being discovered today. What would your favorite application do if a vulnerability was triggered that caused it to crash? Generate a log? A stack trace? Anything at all? One thing that is certain, if the attack is happening over the network, there will be additional traces of the attack on the network layer.

[Learn More: AWS Monitoring]

Amazon VPC Traffic Mirroring Beta

Earlier this month CSE participated in the Amazon Virtual Private Cloud (Amazon VPC) traffic mirroring beta. The release of this service is a testament to the value of network traffic visibility in the cloud, and the goal is to arm customers with additional tools to secure their infrastructure.

Test Environment

To test out this new service, a dedicated VPC was created in the beta-region with a source and destination instance. These instances were given two interfaces each - one for management via SSH (eth0 and ens5) and another dedicated to traffic mirroring (eth1 and ens6).

After installing the traffic mirroring service, the following steps were taken to start mirroring traffic from source to destination instance:

1. Create traffic mirroring target (ens6)
2. Create mirroring session 1 (from eth0 to target)
3. Create mirroring session 2 (from eth1 to target)

Once set up, the two active sessions were ready to be used.

ubuntu@ip-10-88-2-116:~$ aws ec2 describe-traffic-mirror-sessions
{
"TrafficMirrorSessions": [
{
"Tags": [],
"NetworkInterfaceId": "eni-03709e2270eaf2fe4",
"TrafficMirrorTargetId": "tmt-0cce54d15fca32bf0",
"SessionNumber": 1,
"VirtualNetworkId": 16139010,
"TrafficMirrorSessionId": "tms-07201225a9063878b"
},
{
"Tags": [],
"NetworkInterfaceId": "eni-05b6da61cd252e478",
"TrafficMirrorTargetId": "tmt-0cce54d15fca32bf0",
"SessionNumber": 2,
"VirtualNetworkId": 11166435,
"TrafficMirrorSessionId": "tms-055cd3ba74a159b88"
}
]
}

VXLAN Decapsulation

To verify mirrored traffic being received on the destination instance, we launch tcpdump listening on the target interface (ens6), while ssh'd in through the management interface (ens5). Connecting to the source instance management interface (eth0) mirrors traffic to the destination instance on port 4789.

VXLAN decapsulation is a new feature added to the Network Sensor that allows it to process the inner packets as normal network traffic. With VXLAN decapsulation enabled, we can see this traffic resulting in a SSH record.

ubuntu@ip-10-88-0-100:~$ cat /opt/trident/sensor/output/manager/ssh.log| jq{"ts": 1559444407.168741,"uid": "Cus5nNtAg5XWQXsF6","id.orig_h": "1.2.3.4","id.orig_p": 49882,"id.resp_h": "10.88.2.116","id.resp_p": 22,"version": 2,"auth_attempts": 0,"direction": "INBOUND","client": "SSH-2.0-Go","server": "SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8","cipher_alg": "aes128-gcm@openssh.com","mac_alg": "hmac-sha2-256-etm@openssh.com","compression_alg": "none","kex_alg": "curve25519-sha256@libssh.org","host_key_alg": "ecdsa-sha2-nistp256","host_key": "e9:c3:79:ec:0a:d4:22:03:bd:1f:69:ce:4b:15:92:06"}

File Carving and Portscan Detection

With protocol decoding validated, next up is making sure file carving is working as expected on mirrored traffic. To do this, we pick a recent malware sample with malicious executables and documents and use it to simulate a user compromise. The files contained in the malware sample are downloaded through the source instance (eth0) via an AWS S3 bucket using the script below. Source instance (10.88.2.116) in the upper terminal window and destination instance (10.88.0.100) in the lower part.

Capstats, one of the tools bundled with the Network Sensor, is used to monitor incoming traffic on the destination instance receiving interface (ens6). We can confirm that no nic_drops are recorded. Time to look at what the sensor uploaded to the CSE portal.

The Entity page shows us all Signals associated with the source instance (10.88.2.116). Since this is the only activity associated with this Entity, we can see a timeline of all signals at the top. The first validation we want to do is to match up the signal times with the script logs in our terminal. From the first detection to the last, the timestamps line up with the output of the script and we can see a time-to-detection that's within minutes of the recorded activity.

The first signal group we see is named "Script/CLI UserAgent string." These Signals were triggered from our use of wget. We can drill into one of these signals to get more details.

This specific Signal was generated by the very first use of wget to download the malware sample (password protected zip). This screen capture is showing about 1/3 of the Signal fields available, and gives us information about the file’s origin and some of its properties.

The next group of Signals is "Potential Malicious Document Executed." These Signals triggered due to a document (identified by mime-type) being downloaded followed shortly by an Executable or DLL.

Next, we see various YARA matches, including one that's triggered on the password protected zip. Although we cannot successfully extract the zip and look at each mime-type, this rule analyzes filenames contained inside the archive and looks for the .exe extension.

Lastly, we have a Port Scan signal that was triggered from our nmap scan of 10.88.0.100.

Due to the Severity and combination of Signals, a High Severity CSE Insight was generated that's currently unassigned and awaiting further investigation.

With protocol decoding, file carving and portscan detection validated on mirrored traffic, we look at facilitating deployment in AWS.

Facilitating Production Deployments

The natural choice to facilitate deployment is through a CloudFormation template. The template deploys an auto scaling group (ASG) with instances built on a CSE Network Sensor AMI behind a Network Load Balancer (NLB). Mirrored traffic is pointed at the NLB interface, which in turn distributes the traffic across the instances in the ASG.

This deployment strategy, in combination with AWS PrivateLink and Amazon VPC peering, will be a powerful addition to existing CSE ingestion capabilities. We look forward to helping our customers take full advantage of the Amazon VPC traffic mirroring to get increased visibility into their cloud infrastructures.

You can find an additional, in-depth blog post on Amazon’s website.