Securing AI for the enterprise: Our research vision
AI agents for automating tasks in the workplace have arrived and are likely to become ubiquitous in the future. On a challenging benchmark designed by ServiceNow Research reflecting enterprise knowledge worker tasks, the recent Claude model with computer use capabilities from Anthropic achieved significant performance.
ServiceNow has long been a leader in several areas of enterprise IT, including IT service management (ticketing systems to deal with employee IT issues), IT operations management (responding to faults and alerts in complex enterprise IT infrastructure), and customer service management (responding to issues customers face).
Using AI to assist and, in some cases, autonomously resolve issues in enterprise use cases has shown significant productivity improvements—40% to 60% increases in key metrics across many ServiceNow customers.
Cybersecurity risks of AI agents
At the same time, the use of AI to replace or augment traditional software for IT services raises an interesting proposition: Is it possible for an attacker to trick AI into doing something dangerous, such as leak privileged information to an untrusted source or enable unauthorized access to an enterprise’s IT systems?
AI systems based on neural networks have been known to be vulnerable to attacks of various kinds. Imperceptible changes in the pixels of an image caused neural networks to mistake a stop sign for a speed limit. Tools such as Microsoft Copilot and ChatGPT have been vulnerable to leaking private chat histories and training data, respectively.
Despite nearly a decade of research, the foundational adversarial example problem is nowhere close to being solved. This underscores the need for proceeding with caution when developing and deploying AI-based agents in the workplace. It also emphasizes the need for safeguards and interventions, along with evolving research, on AI security risks and mitigations as the landscape evolves.
The reliable and secure AI team at ServiceNow Research was created with precisely this motivation. Our mission is to innovate and deliver secure, private, and reliable AI through research-driven adversarial testing, enterprise-compatible mitigations, and mathematically certified algorithms.
In these next sections, we’ll explore the broad challenges our team expects to focus on, with the eventual scope of work adapting to the evolving cyberthreat landscape of the global enterprise AI ecosystem.
Addressing these challenges will be difficult, so we’ll also engage with a diverse community of security-minded professionals who share a common interest in understanding and securing AI that is open for the benefit of the enterprise and society in general.
Enterprise-aligned adversarial testing
The scope of attacks and malicious exploitation of AI-enabled systems is broad and covers a wide range of possibilities, including extremes such as child pornography and bioterrorism.
However, most of the scenarios studied in the AI safety and security literature aren’t relevant to enterprise settings. In these settings, models are often only used directly by employees of a company (for whom one can reasonably assume benign intent) and not by adversarial or malicious users.
An adversary attacking an AI-enabled system deployed by an enterprise is far more likely to be motivated to gain privileged access to the enterprise’s IT infrastructure. This could be done by sneaking documents or web pages that contain malicious instructions into the enterprise’s workflows.
This brings up the focus of our first research question: How do we develop attack algorithms that expose the risks associated with specific deployments of AI capabilities in the enterprise and use these to inform the design of safeguards or defenses?
Traceability and provenance
AI agents in the enterprise will be most useful when they’re able to access diverse information sources. This includes proprietary or privileged internal information, along with public information from the web.
However, when something goes wrong (for example, the system is subject to an attack or takes an incorrect but catastrophic action, such as deleting a critical database due to incorrect reasoning or hallucination), it’s imperative to perform a post hoc analysis that traces the root of the issue.
Endowing AI agents with the ability to trace information flows and identify the provenance of various sources or inputs that led to a downstream failure is a critical capability the team expects to work on.
This capability will help us address the second research challenge: How do we robustly attribute the output or decision made by a complex multiagent workflow involving both AI and non-AI software tools and human inputs to specific sources, especially when the system may be under attack?
Provable security and privacy
The holy grail for designing secure systems is to have a mathematical guarantee of security, or an assurance that an attacker cannot hope to compromise the system, no matter what they do.
Although this is extremely challenging for modern AI systems, progress is being made on approaches that offer mathematical guarantees, including randomized smoothing, differential privacy, and formal verification.
The key questions we plan to address here are: What are realistic scenarios of AI deployment where we can offer provable security, privacy, and safety guarantees? And what (if any) fundamental limitations prevent us from making such guarantees more broadly?
AI for security
Securing (even non-AI) enterprise IT remains a hurdle. AI-based assistants are poised to significantly help security analysts deal with emerging threats. Developing and securing such assistants is a key opportunity and challenge that the team hopes to make progress on.
We will work on the following challenge: How do we use AI both offensively and defensively to improve security analysis in an enterprise setting?
How can you get involved?
If you’re excited about this research agenda, here’s how you can join our efforts:
- Apply for one of the open roles on the team.
- Engage with us in our open scientific community effort via the AI Alliance or as we put out relevant benchmarks and challenges.
- Reach out to collaborate with us if you’re an academic AI researcher working on safety and security.
Find out more about ServiceNow Research.