Activate multimodal captioning for attachments from an indexed source

Generate searchable captions for images, tables, charts, and complex layouts in attachments from records in an indexed source table.

Before you begin

An administrator must have activated the plugin on your instance. For details on this procedure, see Activate the Platform Multimodal Service plugin.

Role required: ais_admin

About this task

When indexing attachments for search, AI Search includes the attachment's text but ignores its images by default. For tables, charts, and complex layouts, indexing extracts text but doesn't necessarily preserve the context of that text.

To improve search recall, you can configure the to automatically generate descriptive captions for images, tables, charts, and complex layouts in attachments found on records from an indexed source table. With the service activated, you can find indexed source content attachments with these entity types by searching for terms that match the generated captions.

As an example, an attachment might include an image that yields the generated caption woman in office. Searches for woman or office can match the caption terms. Such searches return the attachment as a search result even if its text doesn't otherwise contain either of those terms.

Note:

Multimodal captioning runs as a separate process after text indexing is completed for the attachment in question. Records with attachments that contain images, tables, charts, or complex layouts won't show generated captions until multimodal captioning completes.

Note:

Activating multimodal captioning for attachments found in an indexed source has no effect if the index_attachments attribute is set to false for that indexed source. Make sure this attribute is set to true if you want to index attachments and generate captions for their content.

Procedure

1. Navigate to All > AI Search > AI Search Index > Indexed Sources.
2. Open the indexed source that you want to activate multimodal captioning for.
   Note:

   Multimodal captioning is currently only supported for records from the Knowledge Table indexed source (for the Knowledge [kb_knowledge] table).
3. In the Advanced Configuration related list, check for an existing index_mms_attachments attribute.
   • If an index_mms_attachments attribute already exists with value false, change its value to true. No further steps are needed.
   • If an index_mms_attachments attribute exists with value true, multimodal captioning is already enabled for the indexed source. No further steps are needed.
   • If no index_mms_attachments attribute exists, continue with the following steps.
4. In the Advanced Configuration related list, select New.
5. On the Indexed Source Attribute form, fill in the following fields.

   Field	Value
   Attribute	index_mms_attachments
   Value	true

6. Select Submit.

Result

The new index_mms_attachments indexed source attribute appears in the Advanced Configuration related list.

What to do next

To make the new indexed source attribute take effect, perform a full table reindex for the indexed source. For details on this procedure, see Perform a full table index or reindex for a single AI Search indexed source.