Schedule and manage the jobs that download AWS billing data

Billing Download jobs download, organize, and store billing data for your payer account on the schedule that you specify. The system analyzes the data to generate reports and to make recommendations for changes in your cloud operations that can lead to cost savings.

始める前に

Role required: Cloud Integrations Admin [sn_cld_intg_core.cloud_integrations_admin]

Ensure that your system has 8 GB Clotho memory and 4 GB mid memory.

Enable cost allocation for each Kubernetes cluster before you run an AWS Billing download job to view the Kubernetes spend. For more information, see Enable cost allocation in AWS for Kubernetes cluster.

このタスクについて

During billing download, all the resources are pulled into the system. AWS Redshift resources are placed in the [cmdb_ci_cloud_database] table. After the 4.0 upgrade, make sure that the billing download is executed with reimport true.

注:

From Cloud Cost Management version 8.1 onwards, the default time for billing download jobs has changed from 12:00 to 01:00 (UTC). Also, schedule or run Cloud Cost Management jobs only during off-business hours and when there's no other heavy operations or jobs running on the ServiceNow instance.

If you have installed the Cloud Cost Management Infra Stack application, then toward the end of a job, significant database update, and insert operations are performed based on your Cloud CIs and spend data volume. Thus, any other job that impacts database performance shouldn't be run during this time alongside Cloud Cost Management jobs.

• Billing Download jobs can't be in the Global scope.
• If the Cloud Provisioning and Governance application is installed on your instance: Both Cloud Provisioning and Governance and Cloud Cost Management download billing data. The two download jobs are separate processes and they don't interfere with each other.
• You can create only one Billing Download job for each account (enrollment).
• Configure AWS Billing download for the main account otherwise the spend doesn't generate.
• To ensure accurate reporting and recommendations for some providers, make sure that the Discovery application runs before the scheduled execution.
• Each successful execution of a Billing Download job triggers the Budget Forecast, Business hours, Reservation/saving plans, Rightsizing, and Unused resources jobs to analyze spend and usage data and to update the actionable recommendations in reports.
• Each successful execution of a Billing Download job updates tagged costs. Recent updates that you make to tag category definitions (for example, adding a tag name to a category) might not be reflected in cost reports. You should map tag values to tag categories after the first run because populating tag values needs at least one successful run. You can apply the latest tag category definitions to cost data without running a Billing Download job, by selecting Cloud Cost Management Workspace > Operations > Cost usage tags > Tag categories and then selecting Re-Apply Categories.
• When the scheduled time arrives, job execution happens in multiple stages (for example, connect to the provider, download the data, perform the post-import sort, and so on). The system logs status and results on the Price Sheet Executions page for each stage.
• Cost categories are updated with new cost tag values each time billing data is downloaded. (You define cost tags in the provider portal to associate usage data with a particular business entity. For example, the Cost Center category might include the tags development, testing, and QA. The User category might include the names of your users.)
• Billing data is downloaded in reverse chronological order by month. For example, if the range is from March to June, data for June is downloaded first.
• To ensure meaningful results for the first billing data download, the app downloads data for at least 30 days. For example, if today is June 10 and you specify June for download, the system downloads data for both May and June to ensure at least 30 days of data for analysis.
• You can download data only for the most recent twelve months.

For AWS.

• You can create only one Billing Download job for each service account.
• Only costs of the unblended type are supported.
• The Billing Download job updates the billing node data table [sn_cld_intg_aws_cost_usage] with the CIs in the CMDB that correspond to each resource ID.
• The CI placement process associates downloaded cost and usage data with the appropriate CIs in the CMDB. See Add an AWS CI class type to ensure accurate billing data for the details.

手順

1. Review the settings to verify that the settings on the report meet the requirements for Cloud Cost Management when you have an existing AWS Cost and Usage Report (CUR).
2. On the AWS Management Console, define an AWS CUR as follows:
   注:

   You must configure Legacy CUR because the Cloud Cost Management application supports the legacy version only.
   1. Follow the instructions in the Creating Cost and Usage Reports topic and use the following settings:
      • Select the Include resource IDs check box.
      • Select the Automatically refresh your Cost and Usage Report when check box.
      • Report path prefix: optional
      • Set Time granularity to either Daily or Hourly as needed. The Hourly setting generates a larger, more granular data set.
      • Set Report versioning to Create new report version.
      • Set Compression type to GZIP or ZIP. Cloud Cost Management doesn’t support Parquet.
   2. Track the following values because you enter them into a Cloud Cost Management form in a moment.
      • Report name
      • Report prefix
      • Name of the S3 bucket that is the data storage area for the daily detailed billing records on the AWS account.
3. Navigate to Cloud Cost Management Workspace > Operations > Administration > Billing download jobs.
4. Select the AWS tab.
5. Select New.
6. On the form, fill in the fields:

   表 : 1. AWS Billing Download Job

   Field	Description
   Name	Meaningful name for the scheduled execution of the Billing Download job.
   Last successful execution	The timestamp of the most recent successful execution.
   Notify users/groups	The users or groups to be notified by email about the status of the job execution (for example, download failure). The system doesn’t send a notification for success. Users or groups with the Cloud Integrations Admin [sn_cld_intg_core.cloud_integrations_admin] role are well suited to handle these issues. To update the email template, navigate to System Notification > Email > Notifications and open the Notify on billing job execution error template. For information on configuring the email, see Create an email template.
   Active	Option for activating the job. Only active jobs are executed.
   Run	Frequency with which to execute the Billing Download job. 注: For AWS, Billing Download jobs collect data once per day, even if AWS collects hourly usage data. Depending on the value that you select, additional fields appear. This field is automatically set to Daily.
   Time	Time of day to execute the job.
   Latest execution status	The status of the most recently executed job.
   Latest execution details	Execution details of the most recently executed job.
   AWS Settings
   Service account	The AWS service account.
   Bucket	Amazon S3 bucket that is the data storage area for the detailed hourly billing records on the AWS account.
   Report name	The report-name value that appears in your AWS Cost and Usage Report.
   Report prefix	The report-prefix value that appears in your AWS Cost and Usage Report.

7. Select Save.
8. On the Billing download jobs page, select the created job.
9. Select Test connection.

   The Test Connection workflow uses the settings that you configured to attempt to access the provider account. The system displays a progress pop-up and a success/error message that suggests actions to fix the configured settings.

10. Select Execute now to execute the job after the connection succeeds.
11. In the Download billing data dialog box, fill in the fields.

    Field	Description
    Start month	Starting month for downloading the billing data. 注: Data is downloaded for the specified months. If fewer than 30 days of data for analysis is received, data for the preceding month is also downloaded.
    End month	Ending month for downloading the billing data.
    Re-import data	Option for overwriting data from an earlier download attempt.

12. Select Download.
    An AWS billing download job goes through the following states during the job execution:

    1. Ready: The job is in the queue. For example, when a job is created to download billing data for a few months, the jobs for all months except the latest month are in Ready state.
    2. Requested: The job is fetched and processed immediately. For example, the job that's created to download billing data for a month is in the Requested state. Also, when a job is created to download billing data for more than a month, the latest month job is also in the Requested state.
       注:

       If you have installed the Cloud Cost Management Infra Stack application, it takes some time for the job to change from Ready to Requested state for AWS billing download job with Assume role authentication. This is because it takes time for the files to be downloaded to Cloud Cost Management MID Server. For details, see Processing AWS billing jobs with Assume role authentication.
    3. Reserved: Job is picked by the Kubernetes cluster for processing. The Kubernetes cluster also sends the Job Id to Cloud Cost Management Glide to show which Kubernetes job is processing the billing job.
       重要:

       This state is applicable only if you have installed the Cloud Cost Management Infra Stack application.
    4. In Progress: Billing data is being processed.
    5. Sink Begin: Kubernetes cluster sends some part of the processed billing data to the tables such as Spend Report Daily Aggregate Cost and Spend Report Monthly Aggregate Cost on Cloud Cost Management Glide. The records in these tables are in the Generating state. The job remains in the Sink Begin state until the entire billing data is sent to Cloud Cost Management Glide.

       In the Sink Begin state, the CI placement also starts and the Status of the CI Placement Stage changes from Ready to In Progress.

       重要:

       This state is applicable only if you have installed the Cloud Cost Management Infra Stack application.
    6. Sink Complete: All the processed billing data is sent to Cloud Cost Management Glide from the Kubernetes cluster.
       重要:

       This state is applicable only if you have installed the Cloud Cost Management Infra Stack application.
    7. Finished: Status after Sink Complete when all the billing files are sent to the Glide database.
       注:

       The following changes happen when the job is marked Finished:

       • The state of the records in the Spend Report Daily Aggregate Cost and Spend Report Monthly Aggregate Cost tables change from Generating to Active. Also, existing records are marked inactive.
       • The billing account for which the billing file was downloaded is automatically set as a Master account in the Service Accounts table.
    8. Success: Job is completed successfully.

タスクの結果

The following events happen when the job executes:

• While downloading the data, Cloud Cost Management updates the billing node data table [sn_cld_intg_<provider>_cost_usage] with the CIs in the CMDB that correspond to each resource ID. If a CI doesn’t exist, the system generates a placeholder CI. On subsequent discovery, the system reconciles the placeholder CI.
• Cloud Cost Management generates a log entry for each stage of the execution on the Billing Download Executions page.