Properly managing your log data in your cloud environments is paramount for providing insight into the health and security of your systems and applications. From preventing downtime, to early detection of a security or compliance incident, or debugging processes in your CI/CD pipeline, having a holistic view in to your organizations log data can set your teams up for success. Let’s demonstrate by going through a scenario from a fictitious organization that is starting their migration to the cloud.
Let’s assume that this organization’s security operations program is fairly mature (they have a vulnerability management solution, edge devices for their edge and content filtering, etc.) and that they’re pulling all relevant logs into Splunk for ingestion. They’re using Splunk for their SIEM, and they have a handful of playbooks created and operationalized in Splunk Phantom (their SOAR solution) to handle events. The Cloud team has been tasked with coordinating their migration efforts with the security team to ensure visibility into their AWS organization and to ensure all subsequent logs and telemetry data makes its way to their Splunk deployment.
The organization has staffed a small team specifically tasked with designing the architecture for their AWS organization, as well as ensuring the security best practices are met in conjunction with their existing security team. The cloud team has worked with the security team, and they are now running one AWS organization with 3 accounts – dev, test, and production. Let’s take a closer look at how they can effectively and efficiently ingest their data and operationalize it with their current processes.
For the security team to monitor actions taken by a user, role, or an AWS service in these AWS accounts, they will need to collect the AWS CloudTrail events in Splunk. The organization has chosen to make use of the Kinesis Firehose to send the CloudTrail events directly into Splunk for near real-time monitoring. The following steps are performed to get the AWS CloudTrail logs into Splunk:
- Configure Splunk Heavy Forwarders behind an Elastic Load Balancer in the production AWS account. These heavy forwarders will be used to receive the logs from AWS, perform the necessary parsing, and forward the events to the indexers.
- Install the Splunk Add-on for Amazon Kinesis Firehose on each of the Heavy Forwarders and Search Heads so that Splunk can properly parse the CloudTrail events.
- Configure the HTTP Event Collector on the Heavy Forwarders to receive the data from the Kinesis Firehose and to send the CloudTrail logs into the “aws” index.
- Configure a trail in CloudTrail to send logs to CloudWatch Logs.
- Configure the Amazon Kinesis Firehose stream to send CloudTrail data to the Elastic Load Balancer address.
- Configure CloudWatch Logs to send logs to the Kinesis Firehose stream.
Now that the security team has the CloudTrail logs in Splunk, they will have the same visibility into the actions taken across all of their AWS accounts in Splunk. See the below sample event for an unauthorized attempt to create a new user in the AWS account.
With insight into all actions taken in their AWS accounts, the security team can now begin to create proactive alerts for any suspicious activity within their account. In Splunk, the security team configures an alert to identify suspicious provisioning activities within their cloud accounts. Using the source IP address in the CloudTrail events, the security team uses the geolocation functionality in Splunk to flag provisioning activities from an unexpected location. The security team can now proactively detect any rogue AWS resources in their account and quickly identify the compromised credentials. Additionally, Splunk Phantom playbooks can be created to respond to the suspicious activity by blocking the source IP at the edge, resetting the credentials of the user in question in AWS IAM, snapshotting and terminating the suspicious VM, or any combination of these.
We’ve demonstrated at a high level that visibility and awareness into activities in your cloud environment via managing and monitoring your log data enables flexibility of response to suspicious activities. Be sure to watch for our whitepaper coming out soon that will dig further into extending cloud monitoring and response capabilities in Splunk.
***
This blog was written by Nick DiPasquale, Senior Solutions Architect and Brandt Varni, Data & Analytics Practice Manager at Set Solutions.