So you want to use Generative AI on your ServiceNow PDI but don't know where to start? Well, at the time of writing this article, we cannot access the Now Assist GenAI capability on a ServiceNow PDI. However, that doesn't mean all is lost. With access to an open-source Large Language Model (LLM) and the ability to build a simple REST integration, we can do it DIY style!
There are multiple ways to access an LLM via REST API on the internet. Recently, I stumbled across a YouTube video from NetworkChuck that inspired me to host my own local LLM (LLLM?). Then I thought, why not connect it to my ServiceNow PDI, hence this article! Now I should mention that my little experiment is really just a REST integration to a local LLM via MID Server. It's certainly no match for the immense OOTB capability that lies in ServiceNow's Now Assist. If you want to know what Now Assist can do for you, follow this link!
Before I go any further, let me hit you with my high-level diagram. After all, a picture paints a thousand words (or "tokens" as they say in the world of LLM).
Similar to NetworkChuck, I decided to install the Ollama API to host my LLM. I then pulled one of the latest open-source models available, llama3.1 (8b version = 4.7GB). This is where my post differs a bit, as I didn't need to install a user interface or play with image generation. The Ollama API provides a REST API to interface with the model, as well as a CLI (command line interface) if that's your thing. I should mention that to successfully host an LLM you need some reasonably powerful hardware. I'm certainly no expert on the subject, but my advice is to go for a newer CPU with at least 6 cores, 16GB RAM, and an SSD with enough space to download a couple of models to play with. I also highly recommend a discrete GPU (with compute support) and enough VRAM to store your model of choice. For example, the llama3.1 8b model is almost 5GB so you'll ideally want a GPU with 8GB or more VRAM.
Here are my specs along with a picture showing the techy bits:
- Intel i5-8400 6-core CPU
- 48GB DDR4 RAM
- 512GB NVMe SSD
- Nvidia RTX 3060 12GB
- Microsoft Windows 10 Professional 64bit
- WSL2 (Ubuntu 22.04.4)
- Ollama 0.2.7 for Linux (via WSL2)
If you're following along at home, here are a couple of tips to confirm your Ollama is running on your GPU hardware rather than your CPU (which is the default when no compatible GPU is found):
- Go to the WSL command prompt and start the CLI (e.g. ollama run llama3.1)
- Ask the CLI a question (e.g. Why is the sky blue?)
- Check Resource/Task Manager to see if the CPU or GPU is under load while the response is generating (similar to below).
Ok, well that wraps it up for part 1. In the next article, we'll install the MID Server and then make a REST call to the Ollama API.
Links to each article:
GenAI on a ServiceNow PDI - Part 1 - Overview
GenAI on a ServiceNow PDI - Part 2 - MID Server
GenAI on a ServiceNow PDI - Part 3 - Gen AI Integration
