Workspace Service Delivery with Microsoft Online Exchange Spoke issue

jasonkist · ‎06-02-2025

Hello Community, I wanted to blog about some technical debt we've been chasing for the past 6 months. This is not a question but an article to hopefully help the next person. With Return to Office, we implemented Workplace Service Delivery (WSD) in 2024. With that, we enabled the Microsoft Online Exchange Spoke to synchronize MS Exchange calendars for flex offices and conference rooms(also referred in this write-up as Graph). This was a big hit with the users and senior management. It allowed users to use a fun tool to seek the workspace of their choice and to reserve it. Also, users can reserve flex offices and conference rooms from WSD or Exchange and the 2 systems remain in sync.

Issue:

After implementation of WSD with Exchange Sync, we started noticing this error message repeating in the system log:

Active Webhook with subscription ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx not found in Microsoft Exchange Online Spoke Subscriptions Table.

The short of it is:

Subscriptions that were deactivated on sn_wsd_rsvsync_reservable_sync_configuration but the deactivate button does not remove the subscription from Graph. You need to do this manually. One possible tool is using the Unsubscribe Webhook flow action.

The long of it is:

This error comes from script include: ExchangeOnlineWebhookAuthenticator. This informational message also occurred with each subscription error: Webhook notification received. On the sn_wsd_rsvsync_reservable_sync_configuration table, no subscription from the errors in any of the subscriptions configured in our lower environments. In the 'Outbound HTTP Logs', there are 10 times the amount of outbound rest calls out to MS Graph than there are actual sync'd reservations for each day. On the transaction log, for app scope sn_ex_online_spke, there are 10-20 times the amount of inbound api calls, with the 'IP Address' mapping to Microsoft(www.whois.com). Nothing made sense yet. What is all this traffic? Where are these subscriptionID's in error messages coming from that we don't have configured. Is this from another App? Is it an outside attack on the API?

We engaged our internal Microsoft team but not much luck, I guess we weren't asking the right question yet. We opened a Hi Portal case and they advised us to go through the internal channels to turn sys property glide.rest.debug to true for a 1 hour window on a Sunday to capture debugging on the inbound traffic from Microsoft(I would advise not doing this during normal business hours). The inbound traffic on the API is constant from Graph, looking at the logs captured, the traffic was legitimate. The light bulb turned on, it's sync'ing EVERYTHING and refreshing old calendar entries. See #1 #tip below.

With the open case, we challenged them that these errors shouldn’t be ignored as advised. Your system log is a representation of your system’s health and steps should be taken to resolve common repeating errors, warning, and informational messages. To be good stewards of your system, the whole team should be monitoring and fixing common repeating system logs messages.

But the question remained, what are these subscription IDs errors in the system Logs. The Graph Explorer is your friend. You will need an account and a valid token. What we eventually rooted out as the issue, when you Activate an email address on ‘Reservable Sync Configuration’, there’s a flow action ‘Subscribe Webhook’ goes out to Graph and creates the Subscription. But when you deactivate a subscription from ‘Reservable Sync Configuration’, it does not call the flow action ‘Unsubscribe Webhook’ to remove the Subscription in Graph, this is manual. When I asked the obvious question, ‘why does the deactivate button not call the flow action to remove the subscription?” I got no answer. When I asked the product owner, they didn’t get back to me. Months have gone by and we still have 1900 of these to clean up. You could use the ‘Unsubscribe Webhook’ action manually and remove the subscription, but how do you know what it is?? Obviously, it’s orphaned in Graph but what is it and what mailbox is it tied to. WSD does not have a tool to list the Webhook Subscription to provide any useful info, we are flying blind folks. See tip #4.

The cleanup is manual. From here many things can be done to verify and cleanup. What we did was build a widget on a simple portal page to enter stale subscription, call flow action create in Tip #4 below, list the detail of the stale subscription, lookup email of stale subscription on current sn_wsd_rsvsync_reservable_sync_configuration table and list current info and list a delete button to call ‘unsubscribe webhook’ flow action and list success/fail.

Tips:

Be careful when you enable the reservation synchronization. For each reservable sync config you enable, it syncs everything past and present and refreshes everything daily. Consider down tuning how much is being sync'd, how often it refreshes and how far back in the past does it sync? We haven't gotten that far yet....
In 'Reservable Sync Configurations', do not configure any email addresses without an associated location, this will cause sync's to stop with no warning or error logs.
On the sn_wsd_rsv_reservation table, add a read acl to the 'Reservation Subject' dictionary for reserved for and attendees is dynamic(me). This is considered sensitive and general users shouldn't be able to see others info
For a good way to view the stale subscriptionID, clone the ‘Unsubscribe Webhook’ to a new action. Name it something like, ‘Get Webhook Info’. Change the http method from delete to get. In the flow action, send the response body to something you can fetch such as the system log.
TEST TEST TEST and validate everything to ensure a seamless cleanup

Hope people find this helpful, happy hunting.