Cisco is a partner of the Amazon Security Lake, supporting the Open Cybersecurity Schema Framework 

At AWS re:Invent 2022, Cisco was proud to be a launch partner for Amazon Security Lake, a new AWS service that automatically centralizes an organization's security data from cloud, on-premises, andcustom sources into a purpose-built data lake stored in a customer's account. With support for the Open Cybersecurity Schema Framework (OCSF) standard, the service can normalize and combine security data from AWS and a broad range of enterprise security data sources. Amazon Security Lake helps you analyze security data, so you can get a more complete understanding of your security posture across the entire organization.

As part of the Cisco Secure Technical Alliance, I had the opportunity to build the Cisco Secure Firewall

integration into Amazon Security Lake for the public preview. With the general availability of Amazon Security Lake, I updated the support of OSCF and validated the integration.

If you've never worked with Secure Firewall or eNcore, here is a summary:

Secure Firewall serves as an organization's centralized source of security information. It uses advanced threat detection to flag and act on malicious ingress, egress, and east-west traffic while its logging capabilities store information on events, threats, and anomalies. By integrating Secure Firewall with Amazon Security Lake, through Secure Firewall Management Center, organizations will be able to store firewall logs in a structured and scalable manner.

What is the eNcore Client

The eNcore client provides a way to tap into message-oriented protocol to stream events and host profile information from the Cisco Secure Firewall Management Center. The eNcore client can request event and host profile data from a Management Center, and intrusion event data only from a managed device.  The eNcore application initiates the data stream by submitting request messages, which specify the data to be sent, and then controls the message flow from the Management Center or managed device after streaming begins.

With eNcore you can access to full list of firewall event types and medata records, including packet records, security intelligence events, enhanced intrusion data, legacy events and more.  In total over 1000+ records types are supported by eStreamer, going back to inception of the Secure Firewall.  More details can be found in the full eStreamer specification.

eNcore runs on Python 3.6+ and supports Firepower Management Center version 6.0 and above, for more details on the eNcore client please see our operations guide.

What's New with the General Availability?

With the Amazon Security Lake release, I enhanced the Cloud Formation deployment script for the eNcore client to automate more features and make the install process easier. Additionally, a user interface has been added for the eNcore client to manage and monitor firewall logs in and out of the Amazon Security Lake . The Network Activity OCSF schema mappings have been fine-tuned to match fields to the proper class structure definition and support has been added for additional firewall event types, including malware and intrusion events.

The Goal: Provide Adaptable Framework to Evolve with OCSF 

Normalization:

The OCSF standard aims to provide a common representation of nested data structures of security data across all sources, vendors and applications. You can find an interactive schema that allows you to drill down into the OCSF class structures and data definitions.

Cisco released an updated version of the eNcore client that can stream firewall logs to multiple destinations. The update provides support for converting the logs into OCSF format. The Firewall data is represented in the Network Activity events class and the logs are mapped to the various attributes and data types under that class.

This integration builds a portable framework in the eNcore client that helps decode Secure Firewall data, translates it into key value pair data sets based on Python classes that mirror the OCSF framework providing transformations that adapt Secure Firewall logs to Network Activity events.  In short, eNcore is the glue that maps raw Cisco Secure Firewall events into a concise consumable format for the Amazon Security Data Lake.

Validating OCSF Compliance

OCSF compliance was validated using tools provided by the OCSF schema such as the OCSF swagger API.

This API will help determine if data fits the OCSF schema and its object hierarchy. It is accessible under the OCSF server project and is continutely updated to support new data types and constructs, as of this writing the eNcore client supports the development version (v0.0.0) of the OCSF schema. Events from secure firewall are modeled against the Network Activity class structure, by executing the /api/classes/NETWORK_ACTIVITY URI we can validate output in real time to determine if the output structure matches the latest OCSF standard.

The Design

The eNcore client provides a way to tap into message-oriented protocol to stream events and host profile information from the Cisco Secure Firewall Management Center. The eNcore client can request event and host profile data from a Management Center, and intrusion event data only from a managed device. The eNcore application initiates the data stream by submitting request messages, which specify the data to be sent, and then controls the message flow from the Management Center or managed device after streaming begins.

These messages are mapped to OCSF Network Activity events using a series of transformations embedded in the eNcore code base, acting as bothauthorand mapperpersonas in the OCSF schema workflow. Once validated with an internal OCSF schema, the messages are then written to two sources: first, a local JSON formatted file in a configurable directory path, and second, compressed parquet files partitioned by event hour in the S3 Amazon Security Lake source bucket. The S3 directories containing the formatted log are crawled hourly and the results are stored in an Amazon Security Lake database. From there we can get a visual of the schema definitions extracted by the AWS Glue Crawler, identify fieldnames, data types, and other metadata associated with your Network Activity events. Event logs can also be queried using Amazon Athena to visualize log data.

Get Started

To utilize the eNcore client with Amazon Security Lake, first go to the Cisco public GitHub repository for Firepower eNcore, OCSF branch.

Download and run the cloud formation script eNcoreCloudFormation.yaml.

The Cloud Formation script will prompt for additional fields needed in the creation process, they are as follows:

Cidr Block:  IP Address range for the provisioned client, defaults to the range shown below

Instance Type:  The ec2 instance size, defaults to t4.large

KeyName  A pem key file that will permit access to the instance

AmazonSecurityLakeBucketForCiscoURI:The S3 location of your Data Lake S3 container.

FMC IP:IP or Domain Name of the Cisco Secure Firewall Management Portal

After the Cloud Formation setup is complete, it can take anywhere from 3-5 minutes to provision resources in your environment. The cloud formation console provides a detailed view of all the resources generated from the cloud formation script, as shown below.

Once the ec2 instance for the eNcore client is ready, we need to allow list the client IP address in our Secure Firewall Server and generate a certificate file for secure endpoint communication.

Steps:

1. In the Secure Firewall Dashboard, navigate to Search->eStreamer, to find the allow list of Client IP Addresses that are permitted to receive data.
2. Click Add and supply the Client IP Address that was provisioned for our ec2 instance.
3. You will also be asked to supply a password, click Save to create a secure certificate file for your new ec2 instance.

4. Download the Secure Certificate you just created and copy it to the /eNcore directory in your ec2 instance. Or upload using the eNcore GUI which is detailed in the next section.

eNcore GUI

Now that we have the certificate, we can use the eNcore GUI to upload to the certificate, this is the new piece that we've added since the public preview back in December 2022. Users can now control and configuration connectivity to the Firepower Management Console (FMC) in a central location, versus installing and running complex command line scripts. Although system administrators and power users are more than welcome to still use that method.

To access the eNcore GUI navigate to <Your EC2 Instance IP Address> -in this case http://52[.]207.21.3:8184. In this example we run a secure SSL tunnel with port forwarding using the AWS pem file to redirect traffic from our ec2 instance to our local host, depending your organizations network security posture you may be able to access the eNcore GUI directly without a SSL tunnel.  Port information can be substituted with any free port on local system, for more details on how to route ec2 instances to your localhost please see the AWS documentation.

ssh -i eNcore-ubuntu.pem -N -L 8141:ec2-52-207-21-3.compute-1.amazonaws.com:3000 [email protected]

Click on the Configuration section to see an outline of the steps needed to execute the eNcore streaming process. Since we used the AWS Cloud Formation Script, the first two steps have already been completed as shown in the picture above.  Next, we can upload the certificate file and provide the password in the field. This will create a key and cert file that will be used to secure communication between the FMC and the EC2 instance with the eNcore client.

Now that we have our communication established, we can send data to Amazon Security Lake.  Click on SEIM Integrations ? AWS Data Lake link to see the active connections. You will see a list populated with the FMC we specified in our cloud formation script. Click the Start button to initiate data streaming.

This will begin the data relay and ingestion process. We can then navigate to the S3 Amazon Security Lake bucket we configured earlier to see OCSF compliant logs formatted in gzip parquet files in a time-based directory structure.

We can verify this by heading back to our AWS Data Lake repository to view the results.  As we can see in the screen below we have new folders that conform to the partitioning required by the Amazon Security Data Lake.  The data we configured previous in the Cloud Formation script creates partitioning that enable the AWS Crawler to efficiently consume and process event data and tie to back to our custom data source we defined earlier, CISCOFIREWALL.

Event data is placed into S3 buckets by event time, will rotate file creation based on the size with a maximium file size of 256MB.   The files are named according the time which the last event was processed providing a first hand look at how far long the eNcore client is in the data streaming process.

Amazon Security Lake then runs a crawler task every hour, to parse and consume the logs files in the target s3 directory, after which we can view the results in Athena Query.  With Amazon Athena we can visual analytics in variety of different tool including Amazon Grafana and Quicksight, in the future we plan to build visualizations to showcase Firewall in the AWS Security Lake.

More information on how to configure and tune the eNcore eStreamer client can be found on our official website. This includes details on how to filter certain event types to focus your data retention policy, and guidelines for performance and other detailed configuration settings.

You can check out the Amazon User Guide for more information. I encourage you to take a look at OCSF yourself and see how it might help the community in the quest for normalization.

If you'd like to get a head start on the process, check out this 5-minute tutorial:

We'd love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!

Cisco Secure Social Channels

Instagram
Facebook
Twitter
LinkedIn