Handling Embedded Hyperlinks in ServiceNow Knowledge Articles with a Custom AI Agent

Karteek KVLN · 5 hours ago

Knowledge articles aren't just text. Ours had SharePoint links, intranet portals, Viva Engage pages, external docs — all embedded as raw <a href> tags in the article HTML. When we built the agent, we initially ignored them. Bad idea. Users were getting half-answers because the agent had no idea what was behind those links.

So we built a proper link-handling pipeline. Here's what we ran into and how we solved each problem.

Extracting links from article HTML

The first step was getting the links out of the article's text field. Simple regex over the HTML, classifying each URL as either an internal KB article, a ServiceNow instance link, or an external URL.

function extractLinksFromHtml(html) {
  var links = [];
  var pattern = /href=["']([^"']+)["'][^>]*>([\s\S]*?)<\/a>/gi;
  var match;

  while ((match = pattern.exec(html)) !== null) {
    var rawUrl   = match[1].trim();
    var linkText = match[2].replace(/<[^>]+>/g, '').trim() || 'Link';

    links.push({
      url:           rawUrl,
      link_text:     linkText,
      link_type:     classifyLink(rawUrl),   // 'kb_article' | 'servicenow_internal' | 'external'
      markdown_link: '[' + linkText + '](' + rawUrl + ')'
    });
  }
  return links;
}

That link_type classification is what drives everything downstream — whether we try to fetch the content, resolve it as a KB sys_id, or skip it entirely.

Problem 1: SSO-gated domains were wasting fetch calls and breaking the output

Every link went through our Fetch External Link Content tool. For intranet URLs sitting behind SSO — SharePoint, internal portals — the tool was firing an HTTP request, getting a 403, and returning a plain-text fallback with no URL in it.

The rendered output looked like this:

🔐 This link points to an authenticated resource and cannot be accessed automatically.
🔐 This link points to an authenticated resource and cannot be accessed automatically.
🔐 This link points to an authenticated resource and cannot be accessed automatically.

Five times. Identical. No URL, no document title, nothing clickable. The user had zero context about what those links were.

We fixed it with a domain pre-check at the top of the fetch tool — before any network call happens. If the hostname matches a known auth-required domain, we skip immediately and return a pre-formatted markdown hyperlink.

var AUTH_REQUIRED_DOMAINS = [
  'share.example-intranet.net',
  'example.sharepoint.com',
  'my.example.com'
];

function isAuthRequired(hostname) {
  return AUTH_REQUIRED_DOMAINS.some(function(d) {
    return hostname === d || hostname.endsWith('.' + d);
  });
}

var parsedHost = url.replace(/^https?:\/\//, '').split('/')[0].toLowerCase();

if (isAuthRequired(parsedHost)) {
  return {
    fetch_status:     'skipped',
    reason:           'auth_required',
    fallback_message: '🔐 [' + (linkText || parsedHost) + '](' + url + ') — Requires authentication. Click to open directly.',
    source_url:       url,
    fetched_content:  ''
  };
}

Key thing: the URL is embedded directly in fallback_message as a markdown hyperlink. Each link now renders distinctly with its actual label. The agent instruction for this case is equally important — it says render fallback_message verbatim, do not paraphrase. We learned early on that if you tell the LLM to "indicate the link requires auth," it will do that differently every time.

Problem 2: JS-rendered pages returning HTTP 200 with no usable content

Some URLs — Viva Engage community pages especially — return a valid 200 response. But the body is a SPA shell: a handful of <script> tags and an empty <div id="app">. No readable content whatsoever.

The fetch tool was returning 800 tokens of script noise. The LLM was trying to summarise it. The summaries were nonsense.

We added a detection function that runs on the response body before extraction:

function isJsGatedContent(responseBody) {
  var lower = (responseBody || '').toLowerCase();

  var scriptCount = (lower.match(/<script/g) || []).length;
  var textLength  = lower.replace(/<[^>]+>/g, '').replace(/\s+/g, '').length;

  // Many script tags, almost no readable text = SPA shell
  if (scriptCount >= 3 && textLength < 200) return true;

  var signatures = [
    'you need to enable javascript',
    '<div id="app"></div>',
    '<div id="root"></div>',
    'window.__initial_state__'
  ];

  return signatures.some(function(s) { return lower.indexOf(s) !== -1; });
}

if (isJsGatedContent(responseBody)) {
  return {
    fetch_status:     'skipped',
    reason:           'js_required',
    fallback_message: '🔗 [' + (linkText || 'View page') + '](' + url + ') — Open directly in browser.',
    source_url:       url,
    fetched_content:  ''
  };
}

Same pattern as the auth case — skip early, return a clean hyperlink, don't waste the LLM's context on garbage.

Problem 3: Unbounded fetching blew up execution time

After fixing the auth pre-check, we ran the agent and saw execution time jump from 27 seconds to 215 seconds.

What happened: the auth pre-check was short-circuiting a lot of URLs in the original run. Once those were handled without a network call, the remaining external links actually got fetched — three articles, six to eight links each, that's up to 24 HTTP calls. Every response landed back in the ReAct loop's context. Context ballooned. The parser started generating RETRY/ConversionError nodes. Retries added more context. Classic spiral.

The fix: a hard cap of 6 fetch calls total, tracked across all articles.

// In agent instructions (Step 7):
// LINK FETCH CAP: Track a running count of Fetch External Link Content calls
// across ALL articles combined. Stop at 6. For beyond-cap links, render
// as 🔗 [text](url) without fetching.

// Enforced in the tool as well:
var MAX_FETCH_CALLS = 6;
var fetchCallCount  = 0;

function shouldFetch(url) {
  if (fetchCallCount >= MAX_FETCH_CALLS) return false;
  if (isAuthRequired(url))              return false;
  fetchCallCount++;
  return true;
}

Beyond-cap links still appear in the response as hyperlinks — users can click through. They just don't get a content summary for those. That's a fine tradeoff.

Run	Changes	Time	Result
Baseline	None	27s	403 errors, broken output
Run 2	Auth pre-check	215s	Fetch loop, RETRY spiral
Run 3	JS detection + cap of 6	41s	Clean response

Problem 4: HTML entities leaking into RACI table output

Separate issue, same root cause: content from article HTML not being cleaned properly before it reached the LLM.

RACI tables in responses were rendering like:

R&rsquo;esponsible | C&mdash;onsulted

The article HTML had encoded entities. Our stripHtml function was removing tags but not decoding entities, so they passed straight through.

function stripHtml(html) {
  if (!html) return '';

  // Strip script/style content entirely
  var c = html.replace(/<script[^>]*>[\s\S]*?<\/script>/gi, ' ');
  c = c.replace(/<style[^>]*>[\s\S]*?<\/style>/gi, ' ');

  // Preserve paragraph breaks
  c = c.replace(/<\/(p|div|h[1-6]|li|tr)>/gi, '\n');
  c = c.replace(/<br\s*\/?>/gi, '\n');
  c = c.replace(/<[^>]+>/g, ' ');

  // Decode entities — do this here, in the tool, not downstream
  c = c
    .replace(/&amp;/g,   '&')
    .replace(/&lt;/g,    '<')
    .replace(/&gt;/g,    '>')
    .replace(/&quot;/g,  '"')
    .replace(/&#39;/g,   "'")
    .replace(/&nbsp;/g,  ' ')
    .replace(/&rsquo;/g, '\u2019')
    .replace(/&lsquo;/g, '\u2018')
    .replace(/&mdash;/g, '\u2014')
    .replace(/&ndash;/g, '\u2013');

  return c.replace(/[ \t]+/g, ' ').replace(/\n{3,}/g, '\n\n').trim();
}

Rule we applied consistently: clean data at the point where it enters the system, not somewhere downstream where the mess has already spread.

What the final fetch flow looks like

URL comes in
    │
    ├─ isAuthRequired?      → skip, return 🔐 markdown link
    │
    ├─ fetchCount >= 6?     → skip, return 🔗 markdown link
    │
    ├─ HTTP fetch
    │     └─ isJsGatedContent?  → skip, return 🔗 markdown link
    │
    └─ stripHtml + decode entities → return fetched_content

Four checks before anything useful happens. Most URLs hit one of the first three. Only a handful actually make it to content extraction — which is exactly what we want.

The agent instruction that ties it together:

For each link returned by the fetch tool:

  fetch_status = success/partial  → summarise fetched_content, append 🔗 [text](url)
  fetch_status = skipped          → render fallback_message verbatim. Do NOT paraphrase.
  fetch_status = failed           → omit silently

That "verbatim" instruction matters more than it looks. Without it, the LLM rewrites the fallback in slightly different ways each run — dropping the emoji, rephrasing the auth message, forgetting the URL.

Passing a pre-formatted string and telling the model not to touch it gives you consistent output every time.

Karteek KVLN

Why we didn't use the out-of-the-box profile search

ServiceNow's AI Agent framework does offer a built-in profile-based search — a higher-level abstraction that handles retrieval for you. We looked at it and deliberately chose not to use it for this agent.

The reason is straightforward: profile search operates on the indexed, processed representation of an article. By the time it hands content to the agent, the article has already been through the platform's own text extraction pipeline. What you get back is clean text — but only text. The embedded links are gone. The <a href> tags that point to catalog items, cross-referenced articles, SharePoint pages, and external documentation are stripped out during indexing and never surface in the search result.

That's fine if your articles are self-contained. Ours weren't.