Amazon S3 metadata collector

  • Release version: Australia
  • Updated June 18, 2026
  • 1 minute to read

    • The Amazon S3 metadata collector harvests read-only metadata from an AWS S3 account.

    The collector harvests metadata from AWS S3, including buckets, objects, and their associated metadata.

    Authentication supported

    The collector supports the following authentication methods. For details, see Credentials and authentication in the AWS documentation.

    • Authentication details supplied through an AWS credentials file
    • Access Key ID and AWS Secret Access Key

    Metadata cataloged

    Table 1. Amazon S3 metadata cataloged by object type
    Object Information cataloged
    Bucket
    • ARN (Amazon Resource Name)
    • Region
    • Name
    • Version state
    • Creation date
    • ACL (access control list) owner ID
    • ACL grantee ID
    • ACL grant permission
    Object
    • Key
    • ARN (Amazon Resource Name)
    • Region
    • Size
    • Last modified date
    • ACL (access control list) owner ID
    • ACL grantee ID
    • ACL grant permission
    • Metadata

    Relationships between objects

    By default, the harvested metadata includes catalog pages for the following resource types. Each catalog page has a relationship to the other related resource types. If the metadata presentation for this data source has been customized, you might see other resource pages and relationships.

    Table 2. Amazon S3 resource page relationships
    Resource page Relationship
    S3 Bucket S3 Object
    S3 Object S3 Bucket

    Limits for S3 buckets

    The collector has a default limit of 10,000 objects per bucket. If a bucket exceeds this limit, the collector skips the bucket, harvests no metadata for it, and logs a warning message.

    To increase the limit beyond the default, set the --max-resources parameter in your collector command. The maximum value for this parameter is 10,000,000 (ten million). If the total contents across all buckets and objects exceed this limit, the collector stops cataloging additional buckets and logs a warning message.