In the May Store release, HLA version 36.0.19 released support for a new data input for Amazon Data Firehose makes it simple for customers to stream CloudWatch logs in their AWS accounts to HLA: no MID server needed.
This means that logs from popular AWS managed services like Amazon EC2, AWS Lambda, Amazon Managed Kubernetes (EKS), or Amazon API Gateway can be streamed in real time to HLA with minimal effort. This post walks through what that looks like.
Additional resources and product documentation are linked to in KB2117152.
Step 1: Getting approval from your cloud and security team
The new integration was designed from the start require minimal security privileges: to stream logs via Amazon Data Firehouse, no IAM credentials are needed. AWS accounts are in full control of which logs they stream to HLA over an encrypted HTTP connection using CloudWatch subscription filters.
We’ve written KB1957114 that details the overall architecture and specific requirements to share with cloud and security teams for them to review specific requirements and understand what data will be sent to ServiceNow HLA.
Step 2: Configuring your instance
As part of the new setup experience, we’ve bundled a CloudFormation template to make this easy. The template will create a new Data Firehose Stream that is configured to send data securely to HLA in the appropriate regional data center.
To get started, search for “firehose” in integrations Launchpad and select “Amazon Data Firehose (MID-less)”:
(If you get a warning message that says prerequisites are not met, follow the instructions to either enable HLA Scale for your instance via a support case or create a new instance KeyStore.)
After naming the integration, click next and click “Download the CloudFormation template” and save it to your computer.
Copy the three values under “Advanced Configuration Details” – we’ll be using this and the template in the next section to configure AWS.
Click the “Activate” button your instance will be ready to receive data.
Step 3: Create a new Firehose stream in your AWS account
At a high-level, what you need to do in your AWS account is configure CloudWatch log groups to forward their data to a new Amazon Data Firehose that streams logs to a regional ServiceNow data center:
Let’s use the the CloudFormation template you downloaded from your instance earlier during the data input setup to configure this in your AWS account. Create a new stack in CloudFormation and choose “upload file”:
Next, when prompted, paste in the three values provided in Integration Launchpad to the CloudFormation template:
After you complete the setup, you should have a new Firehose Stream that’s configured to send data to HLA after a few minutes.
If you get errors, check with your cloud team that your account has the correct permissions to create the required managed services. The full list of permissions required is detailed in KB1957114.
Step 4: Forward logs to HLA
Log forwarding from Amazon CloudWatch is configured via Subscription Filters. Subscription filters tell CloudWatch to forward specific logs to external sources. In our case, we’ll be creating a new subscription filter to send logs to the Firehose Stream that was automatically created in Step 3. In CloudWatch, for a log group, click “subscription filters”:
Next, create a new Data Firehose filter and choose the Firehose Stream and role that was created by the CloudFormation template:
Important: start conservatively with the logs you send to HLA. Choose a specific server or low-volume service to validate the end-to-end flow before streaming large numbers of logs. Sending large volumes of logs will have an impact on your AWS bill.
Click create and new logs will start streaming to the Firehose Stream connected to HLA. If you open up the Firehose Stream that was created in CloudFormation, a good metric to keep an eye on is “delivery success” – after a few minutes it should look something like this:
If you see any errors, refer to KB1957226 for troubleshooting suggestions.
Step 5: See anomalies in HLA
That’s it, after a few minutes you should see logs from Firehose streaming into HLA:
Follow the regular HLA setup and configuration process at this point to detect anomalies and configure your alerts.
As a best practice, it’s a good idea to create separate Data Input per service type in AWS: for example, don’t mix EC2 and EKS logs but create separate Firehose objects and streams for each.
Recap
In this post, we:
- Create a new MID-less Firehose data input
- Used CloudFormation to create a new Data Firehose stream in am AWS account
- Created a new subscription filter to forward logs from a log group (in this case logs from an AWS Lambda function) to the new Firehose
- Saw data successfully streaming directly from AWS to HLA.
More features and functionality are planned in 2025, including support for more standards (like OpenTelemetry) and additional cloud providers. Reach out to your account team if you’d like to learn more or if you have any questions.
