Configure a Transform Map consumer

Import and process data from your Kafka environment using your existing transform map configurations.

Before you begin

• Role required: integration_hub_admin
• This consumer requires a Stream Connect subscription. For more information, see https://www.servicenow.com/products/automation-engine.html.
• The following plugins are required.
  • The ServiceNow IntegrationHub Kafka Consumer [com.glide.hub.kafka_consumer] plugin.
  • The ServiceNow IntegrationHub ETL Consumer - Kafka [com.glide.hub.etl_consumer.kafka] plugin.
• This consumer requires a transform map.

Formatting Kafka message payloads

Use simple JSON payloads for Kafka messages. Each Kafka message should have just one, flat JSON object. Complex JSON payloads, such as those with lists and nested objects, are not supported.

Example of simple, valid JSON input, using a flat map of key-value pairs.

{"key1": "value1", "key2": "value2"}

Examples of complex, invalid JSON input. The first uses a list. The second uses nested objects.

[{"key1": "value1", "key2": "value2"}, {"key1": "value3", "key2": "value4"}]

{"key1": "value1", "key2": {"key3": "value3", "key4": "value4"}}

About this task

To configure a consumer, you need to create two records.

1. The consumer record, which specifies how to import and process data.
2. A record for the Kafka stream, which defines the stream of data to your consumer.

This task covers the consumer creation. For instructions on creating a Kafka stream, see Create a Kafka stream.

Procedure

1. Navigate to All > IntegrationHub > Consumers > Transform Map Consumer.
2. Select New.
3. In the form, fill in the fields.

   Table 1. Kafka Transform Map Consumer

   Field	Description
   Name	Name of the Transform Map consumer.
   Transform Map	Name of the transform map to use to process data.
   Delivery guarantee	If there's a node failure, option to specify the delivery guarantee for incoming messages. Select one of the following. No lost but duplicates: Ensures that all messages are delivered at least once. Some messages might be delivered more than once. Once or not at all: Ensures that a message isn’t delivered more than once. Some messages might not be delivered at all.
   Column mapping	Option to specify whether the message's JSON key maps the data to the column's name or label in the import set table. Select one of the following. Label Column name
   Synchronize Inserts	Option to guarantee there's only one record with unique coalesce field values by synchronizing record inserts.
   Application	Application scope for the Transform Map Consumer.

4. Select Save.

What to do next

Create a Kafka stream for this consumer. After the stream is activated, you can start receiving messages from your Kafka environment.