Domain separation and Stream Connect

  • Release version: Washingtondc
  • Updated February 1, 2024
  • 2 minutes to read
    • Summarize
    Summarized using AI
    This content was generated using new OpenAI-powered functionality. Results are provided on an as is basis and are not guaranteed to be accurate or complete.

    Summary of Domain Separation and Stream Connect for Apache Kafka

    Domain separation is a feature supported in Stream Connect for Apache Kafka, allowing users to organize data and processes into distinct domains. This separation enables strict control over data visibility and access among users, ensuring that specific administrative tasks are confined to designated groupings.

    Show full answer Show less

    Key Features

    • Domain-Aware Application Properties: The application properties are tailored to accommodate domain separation as necessary.
    • Message Replication Tables: Records in several Stream Connect Message Replication tables are separated by domain, including:
      • Message Replications
      • Channel Replications
      • Message Replication Statistics
      • Kafka Topic Replications
    • Topic Namespaces: Configures domain access to Kafka topics via linked namespaces, ensuring users only interact with accessible topics as determined by access control lists.
    • Producers and Consumers: Both are domain separated, allowing producers to send data only to visible topics and ensuring that runtime data tables are also domain-specific.

    Key Outcomes

    By effectively implementing domain separation in Stream Connect for Apache Kafka, customers can achieve improved data governance and operational efficiency. This capability allows for the creation of domain-specific Kafka streams, ensuring that all data is accurately categorized and processed within the correct domain. Customers can expect enhanced control over their integrations and data management practices, aligning them with their organizational needs.

    Domain separation is supported for Stream Connect. Domain separation enables you to separate data, processes, and administrative tasks into logical groupings called domains. You can control several aspects of this separation, including which users can see and access data.

    Support level: Standard

    • Includes all aspects of Basic level support.
    • Application properties are domain-aware as needed.
    • Business logic: The service provider (SP) creates or modifies processes per customer. The use cases reflect proper use of the application by multiple SP customers in a single instance.
    • The instance owner must configure the minimum viable product (MVP) business logic and data parameters per tenant as expected for the specific application.

    Sample use case: An admin must be able to make comments required when a record closes for one tenant, but not for another.

    For more information on support levels, see Application support for domain separation.

    Message replication

    Records in the following Stream Connect Message Replication tables are domain separated.
    • Message Replications [sys_sc_message_replication]
    • Channel Replications [sys_sc_channel_replication]
    • Message Replication Statistics [sys_sc_channel_replication_metric]
    • Kafka Topic Replications [sys_kafka_topic_replication]

    The domain for a message replication record is specified in the record's sys_domain field.

    For channel replication and replication metrics records, the domain is determined by the domain_master attribute. The domain_master attribute uses a reference field's domain to determine the domain for the current record. A channel replication record gets its domain from the referenced message replication record. A replication metrics record gets its domain from the referenced channel replication record.

    A Kafka topic replication record has the same domain as its associated channel replication record because the Channel Replications table is a parent of the Kafka Topic Replications table.

    Topic namespaces

    You can use topic namespaces to configure which domains can access a Kafka topic on a domain-separated instance. In ServiceNow, Kafka topics are linked to a namespace. Namespaces can be linked to ServiceNow domains. When a namespace is assigned to a specific domain, all the topics created with that namespace are also assigned to the namespace's domain. Users can only see and interact with the topics they have access to, based on domain visibility and access control lists (ACLs).

    For more information, see Managing namespaces and topics in Hermes.

    Producers and consumers

    Stream Connect producers and consumers are also domain separated. For producers, you can produce data only to topics that are visible to the domain.

    For consumers, all the tables used to keep runtime data are domain separated. These tables include the following.
    • Kafka Streams [sys_kafka_stream]
    • Kafka Subscriptions [sys_kafka_subscription]
    • Kafka Subscription Partition Groups [sys_kafka_partition_group]
    • Kafka Consumer Statistics [sys_kafka_consumer_statistics]
    • Kafka Unprocessed Messages [sys_kafka_unprocessed_messages]

    Domain users can create domain-specific Kafka streams with the topics that are visible to the domain. When activating a Kafka stream, the subscription and partition groups are created within the same domain. At runtime, the domain of the partition group is set by the consumer thread so that all the data is imported to the correct domain. For more information on producers, consumers, and Kafka streams, see Using Stream Connect for Apache Kafka.