Planning and Reinforcement Learning workshop at ICAPS 2021

  • ServiceNow Research
  • 2021
December 17, 2021

Bridging the gap between AI Planning and Reinforcement Learning - ICAPS 2021 Planning and Reinforcement Learning workshop

Written by Hector Palacios, research scientist

The second edition of the Bridging the Gap Between AI Planning and Reinforcement Learning workshop was held during ICAPS 2021, the International Conference on Automated Planning and Scheduling. This was a joint workshop for AI researchers who work at the intersection of AI planning and reinforcement learning. The goal was to encourage discussion and collaboration between these two communities. Each focuses on sequential decision problems, but with different emphases and methods and little awareness of the other’s specific issues, techniques, methodologies, and evaluation protocols.

Before we jump into the workshop content, I’d like to provide a bit of background for anyone new to AI planning or reinforcement learning for intelligent decision-making in the context of enterprise AI research.

AI planning vs. reinforcement learning

On the one hand, the AI planning community aims to create algorithms to achieve defined goals in specific worlds. On the other hand, the reinforcement learning (RL) community seeks to develop learning algorithms that produce a policy with a low average error rate in future instances of a specific world, but with no guarantees for any instance since one instance could be an uncommon case that a "good" policy ignores.

RL algorithms require that the new instances be from the same world used for training, emphasizing the obtention of effective policies at the expense of specializing in a particular fixed world. In contrast, AI planning emphasizes flexibility, robustness, and adaptation to new instances and worlds at the expense of requiring a world description.

The state of the art in RL consists of algorithms that need applied research to be used in specific domains—sometimes with great success—such as AlphaGo, a superhuman player of the board game Go.

State-of-the-art algorithms in AI planning can obtain plans with thousands of actions for unseen world descriptions. For some problems, AI planning is particularly effective. For instance, the logistics domain is one of the many benchmarks used for evaluating planning techniques. The planning community relies on a standard language for describing the planning domain and problems: the Planning Domain Definition Language (PDDL). New domains are introduced during international planning competitions.

You can see and solve an instance of logistics using this example of online tool, where you can see other instances of logistics and import other domains.

PDDL Editor online tool from

The same logistics example can be explored using Visual Studio Code. Check out the repo PDDLGym if you want to test RL algorithms in planning domains, both deterministic and with probabilistic effects. Documentation and educational material about planning are available at and

Why AI planning and RL matter for enterprise AI

First, although RL methods are becoming state of the art in many AI use cases and applications, they do not scale as well as AI planning in terms of generalization over a family of situations in the same domain, changes in the domain, and guarantees per instance. The relationship between learning and reasoning is at the core of this mismatch in scalability and calls for tighter integrations to overcome the weaknesses of each family of methods.

Second, ServiceNow offers customers a unified platform for enterprise AI. A team of human agents using the Now Platform can execute multiple actions to achieve their goals—e.g., to resolve a new IT incident. In such a scenario, AI planning and RL can provide AI methods to recommend sequences of actions that resolve such incidents. In contrast, standard machine learning methods might be unaware of the consequences of intermediate actions.

Bridging the gap between AI planning and RL

The organizers of the second edition of the AI planning and reinforcement learning (PRL) website accepted 25 of the 35 papers submitted and hosted five invited talks, 11 oral presentations of papers, a poster session, and five discussion sessions about topics transversal to the accepted papers. The recordings, papers, and posters are available on the PRL website. We are organizing a new edition of the PRL workshop.

Invited talks:

Discussion session topics included:

  • Abstractions in Planning & RL

  • Safe, Risk-Sensitive, and Robust Planning and RL

  • Domain Generalization in Planning and RL

The papers detailed the following problems and techniques:

  • Problems considered:

    • How to improve RL generalization in the same domain/world
    • How to use planning models to improve RL
    • How to ensure safe RL and verify RL guarantees

  • Techniques used:

    • Planning and RL algorithms
    • World simulators, some of them using a hidden planning model as ground truth
    • Graph neural networks (GNNs)
    • Optimization under constraints
    • Hierarchical representations

Thank you to the co-organizers

We want to thank the co-organizers of the workshop: Hector Palacios from Element AI, a ServiceNow company; Vicenç Gómez and Anders Jonsson from Universitat Pompeu Fabra in Barcelona, Spain; Scott Sanner from the University of Toronto; Andrey Kolobov from Microsoft Research in Redmond, Washington; and Alan Fern from Oregon State University.

Related content

In addition to being the co-chair of the PRL workshop, Hector Palacios participated in the main track of ICAPS 2021 with an invited talk titled “Planning for Controlling Business-to-business Applications” (recording).

© 2021 ServiceNow, Inc. All rights reserved. ServiceNow, the ServiceNow logo, Now, and other ServiceNow marks are trademarks and/or registered trademarks of ServiceNow, Inc. in the United States and/or other countries. Other company names, product names, and logos may be trademarks of the respective companies with which they are associated.

ICAPS 2021 screenshot: Opportunities and Challenges


  • Digital transformation in healthcare for nurse retention: smiling nurses walking
    Boosting nurse retention through digital transformation in healthcare
    Digital transformation in healthcare is the key to nurse retention. Find out how it can free nurses to focus on their passion: caring for patients.
  • Knowledge 2023 takeaways: a group of women raising their hands
    4 takeaways from Knowledge 2023
    Knowledge 2023 united thousands of ServiceNow customers and colleagues at the first single global user conference in three years. Here are four key takeaways.
  • Artifacts of pride: 3 ServiceNow employees
    Showcasing artifacts of pride
    In honor of Pride Month, three ServiceNow employees share personal artifacts that have shaped their lives. Watch a video and get inspired.

Trends & Research

  • ESG technology: green surrounding a river, woman smiling, 2 government employees in conversation
    Cybersecurity and Risk
    Survey says ESG technology drives results
  • RPA: group of workers gathered around a conference table looking at a laptop
    AI and Automation
    Forrester report: ServiceNow debuts as a Strong Performer in RPA
  • Digital innovation: three workers looking at a computer monitor
    AI and Automation
    Survey says digital innovation is the way to navigate macro uncertainty