In traditional data center environments, security teams usually leverage their standard security tools and agents to capture network-level logs. Capturing these logs gives teams visibility into network architecture and traffic flow. However, when we migrate applications to AWS, these standard practices change. The same toolsets might not be as efficient as they were in the traditional environment. While security fundamentals are the same, the approach changes. Security teams need to explore new options and adopt new tools to ensure adequate security coverage.
AWS Virtual Private Cloud (VPC) Flow Logs can help security teams who are looking for tools to combat cloud security challenges. In this post, we will discuss how to leverage VPC flow logs to solve security challenges.
How VPC Flow Logs Help Security Teams
One of the major security challenges for security teams is getting visibility into application traffic flows moving in and out of the virtual infrastructure. Another security issue is validating whether or not the application’s build covers critical security and compliance aspects. Unfortunately, security is often an afterthought during the product design and development stages, which leaves organizations vulnerable to security mishaps that can result in disastrous consequences.
To help data security teams deal with these security concerns, AWS released the VPC Flow Logs feature in June 2015. This feature can be used by security teams “to troubleshoot connectivity and security issues, and to make sure that network access rules are working as expected.” VPC Flow Logs collects and stores information about network traffic that is coming in and going out of network interfaces in the VPC. Trend and pattern analysis, troubleshooting, and alarms for a specific network traffic are just a few of the use cases where VPC Flow Logs can be helpful.
VPC Flow Logs and Structure
AWS allows you to set up VPC Flow Logs for VPCs, subnets, and network interfaces. Once a flow log is established for a VPC or subnet, it records all of the traffic that enters and exits the network interfaces that are part of that VPC or subnet. You can use the CloudWatch Logs service to store the logs for every network interface that has its own, unique stream. In addition, you have the capability to choose the type of traffic you want to store (i.e., accepted traffic, rejected traffic, or all traffic). VPC Flow Logs are not restricted to the AWS EC2 service; you can also register network logs for other services, such as AWS RDS, ELB, ElastiCache, etc.
The VPC Flow Logs format looks like this:
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
Below is a description of each field from the AWS documentation.
|version||The VPC flow logs version.|
|account-id||The AWS account ID for the flow log.|
|interface-id||The ID of the network interface for which the log stream applies.|
|srcaddr||The source IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.|
|dstaddr||The destination IPv4 or IPv6 address. The IPv4 address of the network interface is always its private IPv4 address.|
|srcport||The source port of the traffic.|
|dstport||The destination port of the traffic.|
|protocol||The IANA protocol number of the traffic. For more information, go to Assigned Internet Protocol Numbers.|
|packets||The number of packets transferred during the capture window.|
|bytes||The number of bytes transferred during the capture window.|
|start||The time, in Unix seconds, of the start of the capture window.|
|end||The time, in Unix seconds, of the end of the capture window.|
|action||The action associated with the traffic:|
|log-status||The logging status of the flow log:|
Image 1. VPC Flow Logs field descriptions
Let’s look at some sample logs:
2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK
2 123456789010 eni-abc123de 172.31.9.69 172.31.9.12 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK
These log examples should raise multiple queries for security teams, including:
- Why is the application at 172.31.16.169 trying to SSH into 172.31.16.21?
- What is the nature of this application?
- Why is port 22 open on this instance?
- Why did the application at 172.31.9.69 attempt an RDP operation to 172.31.9.12 even though the ports are blocked?
- Are the REJECT events common for this EC2 instance destination?
- Is the 172.31.9.69 EC2 machine prone to performing such actions across other EC2 instances?
SIEM tools can be used to ingest these logs, which can then be used to show correlations between various events. Having this information can bring up new scenarios, such as new connection patterns, for security teams to review and uncover some previously unknown security threats to the organization.
The Security Value of VPC Flow Logs
Let’s say an enterprise organization is migrating their workloads to AWS, and the security team needs to support the investigation of a number of incidents. For example:
- The audit logs on the application servers say that a management server is sending a large number of requests on port 22.
- The web server is performing slowly during a specific time of day and there is reason to suspect that it has been compromised.
- There are a large number of connection requests during off-business hours.
It is crucial for the security team to address these situations as soon as possible. With every passing hour—or in some cases, with every passing minute—the security threat remains unresolved, the potential for damage to the organization’s business increases. With this in mind, your organization needs to have a plan in place and the appropriate tools at its disposal in order to find and fix problems quickly.
VPC Flow Logs can help security teams resolve incidents and service requests efficiently. Security teams can leverage this feature to set up compliance standards and continuous monitoring, which will allow for easy detection of suspicious events. VPC Flow Logs offer many security benefits, including:
- Validation of network design and configuration: Organizations can monitor incoming and outgoing traffic from each network interface and validate the network traffic flow. For example, the web server should be communicating with the application server, and the application server should be communicating with the database server. VPC Flow Logs can help the security and application teams validate their understanding of application security. There shouldn’t be any direct connection from the web server to the database server. By analyzing the network traffic and mapping the incoming and outgoing connections, the security and network teams can put the adequate security group rules and NACL rules in place, which will help harden security posture.
- Protection from malicious traffic: By analyzing network traffic arriving to web-facing servers or services (such as ELB), the security team can perform the reputation and location checks of the source IP addresses and determine the nature of the requests. Let’s say you have a case like the one that was mentioned in the previous list, where the application is receiving a high number of requests during off-business hours. Here, the security team can analyze the IPs and their locations during off-hours and conclude whether the requests are genuine or not. If the requests are coming from a region or country where the organization doesn’t have any business, or if they are coming from blacklisted IPs, the security team can block the IP range or put up necessary web application firewall rules to mitigate the attacks.
- Encryption reports: It is somewhat of a mandate these days to ensure that all of the internal traffic in a VPC is also encrypted to protect from any misuse or attacks. VPC Flow Logs can help audit this by analyzing the network traffic logs and triggering an alarm using CloudWatch Events for any communication on unencrypted ports. The risk factor increases when the connection is extended to the data center devices, and that’s when continuous monitoring of VPC Flow Logs is required.
- Network traffic and usage reports: Security teams can use VPC Flow Logs to generate network traffic and usage reports to examine new threat patterns, usage patterns for critical assets, usage of non-compliant or risky protocols, week-by-week optimizations, etc. These reports are very helpful to auditors during compliance audits.
VPC Flow Logs are very useful for providing security teams with visibility into application running in AWS. There are many other advantages of VPC Flow Logs that were not covered in this post, so we suggest that you take a look at the references below for more information.
While we highly recommend using VPC Flow Logs, you should be aware that this feature only works on Layer 4 of TCP/IP stack and that it is restricted to network-level monitoring. More sophisticated attacks are carried out on Layer 7, so it’s important that your organization invests in the right combination of tools; doing so will give you the complete security picture of your environment that is running in AWS.