O365 Service Alerts & the Service Status widget, and outages table etc

jamesmcwhinney · ‎05-07-2020

For anyone else out there who has started moving into Office 365, you are probably already familiar with the "M365 Service Health Notifications" or "365 Sercice Alerts" that go out on a daily basis.

These alerts outline aspects of the Office 365 services which are currently experiencing issues.

From my perspective, it would be ideal to expose some of these to our end users via our the service status widget in ESS, where they could see that the given service is experiencing a service degradation, or show a notification, or a full outage state. From there they could drill in to read the alert verbatim as written by microsoft.

Has anyone attempted anything of this nature already, either via a manual process, or via some sort of automation via the Microsoft graph API?

Thanks,

- James

jamesmcwhinney · ‎06-29-2020

Here is the rest of the solution for others attempting to achieve something similar:

Overview:

System properties are used to store the keys for the integration.

A scheduled job runs hourly which calls a script include, which:

connects to Azure to create a token.
Uses the token to pull down the service status and update the outage records.
Uses the token to pull down the messages and update the outage records.

Creation of Azure Active Directory “Enterprise Application”

Open Azure Active Directory Admin portal as global admin
Note tenant ID on “Overview” page
Click on “Enterprise Applications”
1. Click on “New Application”
2. Create a new application and note the Application (client) ID and Object ID
Click on “App Registrations”
1. Open app that was created
2. Create client secret and note #
3. Go to API permissions and add “user.Read” and “ServiceHealth.Read”
4. Ensure global admin approves permissions

Creation of REST Messages:

Create new rest Message “Microsoft Azure”
1. Endpoint: https://login.microsoftonline.com/${TenantID}/oauth2/token?api-version=1.0
2. Authentication type: no authentication
3. Create HTTP Method “Get Access Token”
4. Endpoint: https://login.microsoftonline.com/${TenantID}/oauth2/token?api-version=1.0
5. Authentication type: inherit from parent
6. Variable Substitutions (for testing):
7. 1. ClientID: (your client ID)
  2. ClientSecret: (your client secret)
  3. TenantID: (your tenant ID)
8. HTTP Request
9. 1. Content:grant_type=client_credentials&resource=https://manage.office.com&client_id=${ClientID}&client_secret=${ClientSecret}
Create HTTP Method “Get All Messages”
1. Endpoint: https://manage.office.com/api/v1.0/${TenantID}/ServiceComms/Messages
2. Authentication type: inherit from parent
3. HTTP Headers
4. 1. Authorization: “Bearer ${Token}”
5. Variable Substitutions (for testing):
6. 1. Filter: ?$filter=MessageType eq Incident
  2. Token: (your token)
  3. TenantID: (your tenant ID)
Create HTTP Method “Get Service Status”
1. Endpoint: https://manage.office.com/api/v1.0/${TenantID}/ServiceComms/Messages
2. Authentication type: inherit from parent
3. HTTP Headers
4. 1. Authorization: “Bearer ${Token}”
5. Variable Substitutions (for testing):
6. 1. Token: (your token)
  2. TenantID: (your tenant ID)

Testing:

If you have configured the test values with your client secret, tenant ID etc, then you should be able to click the test button inside the HTTP method to test the interface.

Test the “Get Access Token” method

Copy the token from the response into the “token” variable for the other 2 methods and then test them as well to ensure they work.

Create System Properties:

Create 3 system properties to hold the client secret, tenant Id etc (type “string”):

AzureAccessToken.TenantID
AzureAccessToken.ClientID
AzureAccessToken.ClientSecret

Create Script Include:

Create Script include “O365OutageSync”

var O365OutageSync = Class.create(); 

O365OutageSync.prototype = { 

    initialize: function() { 

    }, 

    refreshO365OutageRecords: function(token){ 

        var date_now = gs.nowDateTime(); 

        var parser = new JSONParser(); 

        var json_responseBody = "failed to load"; 

        try {  

            var r = new sn_ws.RESTMessageV2('Microsoft Azure', 'Get Service Status'); 

            r.setStringParameterNoEscape('TenantID', gs.getProperty('AzureAccessToken.TenantID')); 

            r.setStringParameterNoEscape('Token', token); 

            var response = r.execute(); 

            var httpStatus = response.getStatusCode(); 

            if(httpStatus == '200'){ 

                //Success 

                var str_responseBody = response.getBody(); 

                json_responseBody = parser.parse(str_responseBody); 

            } else { 

                gs.log("Error requesting O365 Service Status.  httpStatus:" + httpStatus); 

            } 

        } catch(ex) { 

            var message = ex.message; 

            gs.log("Error requesting O365 Service Status.  Error Message:" + message); 

        } 

         

        if(json_responseBody != "failed to load"){ 

            for (s = 0; s < json_responseBody.value.length; s++) { 

                //gs.log(json_responseBody.value[s].WorkloadDisplayName + " : " + json_responseBody.value[s].StatusDisplayName); 

                var str_bs_sys_id = ""; 

                 

                //***Create Business Service Record if it does not already exist*** 

                var gr_bs = new GlideRecord('cmdb_ci_service'); 

                gr_bs.addQuery('name', 'O365 - ' + json_responseBody.value[s].WorkloadDisplayName); 

                gr_bs.query(); 

                if(gr_bs.next()){ 

                    str_bs_sys_id = gr_bs.sys_id + ""; 

                } else if (json_responseBody.value[s].IncidentIds.length > 0){ 

                    //Doesnt exist, create it 

                    var gr_new_bs = new GlideRecord('cmdb_ci_service'); 

                    gr_new_bs.initialize(); 

                    gr_new_bs.setValue('name', 'O365 - ' + json_responseBody.value[s].WorkloadDisplayName); 

                    gr_new_bs.setValue('used_for', 'Production'); 

                    str_bs_sys_id = gr_new_bs.insert(); 

                } 

                 

                 

                //***Create Outages for open incidents*** 

                var arr_openOutages = []; 

                for (i = 0; i < json_responseBody.value[s].IncidentIds.length; i++) { 

                    arr_openOutages.push(json_responseBody.value[s].IncidentIds[i]); 

                    var gr_existing_outage = new GlideRecord('cmdb_ci_outage'); 

                    gr_existing_outage.addQuery('u_number',json_responseBody.value[s].IncidentIds[i]); 

                    gr_existing_outage.query(); 

                    if(gr_existing_outage.next()){ 

                        //Set Outage Status for existing Outage Record 

                        switch(json_responseBody.value[s].Status){ 

                            case 'ServiceDegradation': 

                                gr_existing_outage.setValue('type', 'degradation'); 

                                break; 

                            case 'ServiceInterruption': 

                                gr_existing_outage.setValue('type', 'outage'); 

                                break; 

                        } 

                        gr_existing_outage.update(); 

                         

                         

                    } else { 

                        var gr_new_outage = new GlideRecord('cmdb_ci_outage'); 

                        gr_new_outage.initialize(); 

                        gr_new_outage.setValue('u_number', json_responseBody.value[s].IncidentIds[i]); 

                        gr_new_outage.setValue('cmdb_ci', str_bs_sys_id); 

                        gr_new_outage.begin = date_now; 

                         

                        //Set Outage Status for existing Outage Record 

                        switch(json_responseBody.value[s].Status){ 

                            case 'Investigating': 

                                gr_new_outage.setValue('type', 'degradation'); 

                                break; 

                            case 'ServiceDegradation': 

                                gr_new_outage.setValue('type', 'degradation'); 

                                break; 

                            case 'ServiceInterruption': 

                                gr_new_outage.setValue('type', 'outage'); 

                                break; 

                            case 'RestoringService': 

                                gr_new_outage.setValue('type', 'degradation'); 

                                break; 

                            case 'ExtendedRecovery': 

                                gr_new_outage.setValue('type', 'degradation'); 

                                break; 

                            case 'VerifyingService': 

                                gr_new_outage.setValue('type', 'degradation'); 

                                break; 

                        } 

                        gr_new_outage.insert(); 

                    } 

                } 

                     

                     

                //***End Outages for resolved incidents*** 

                //gs.log("Closing incidents for :" + json_responseBody.value[s].WorkloadDisplayName +", cisysid:" + str_bs_sys_id +  "Status:" + json_responseBody.value[s].Status + ", open outages:" + arr_openOutages.join(',')); 

                if(str_bs_sys_id.length > 0){ 

                    var arr_closedcodes = ['ServiceRestored', 'PostIncidentReportPublished', 'ServiceOperational', 'FalsePositive']; 

                    var arrUtil = new ArrayUtil(); 

                    var gr_active_outage = new GlideRecord('cmdb_ci_outage'); 

                    gr_active_outage.addQuery('cmdb_ci', str_bs_sys_id); 

                    gr_active_outage.addNullQuery('end'); 

                    gr_active_outage.query(); 

                    while(gr_active_outage.next()){ 

                        if((arrUtil.indexOf(arr_closedcodes, json_responseBody.value[s].Status) >= 0)||(arrUtil.indexOf(arr_openOutages, gr_active_outage.u_number) < 0)){ 

                            //Service is fully operational. 

                            //Close out all active outages 

                            gr_active_outage.end = date_now; 

                            gr_active_outage.update(); 

                        } else { 

                            

                        } 

                    } 

                } 

            } 

        }         

    }, 

    refreshO365OutageMessages: function(token){ 

        //***Pull Down Outage Messages*** 

        var parser = new JSONParser(); 

        var json_responseBody = "failed to load"; 

        try {  

            var r = new sn_ws.RESTMessageV2('Microsoft Azure', 'Get All Messages'); 

            r.setStringParameterNoEscape('TenantID', gs.getProperty('AzureAccessToken.TenantID')); 

            r.setStringParameterNoEscape('Token', token); 

            var response = r.execute(); 

            var httpStatus = response.getStatusCode(); 

            if(httpStatus == '200'){ 

                //Success 

                var str_responseBody = response.getBody(); 

                json_responseBody = parser.parse(str_responseBody); 

            } else { 

                gs.log("Error requesting O365 Service Messages.  httpStatus:" + httpStatus); 

            } 

        } catch(ex) { 

            var message = ex.message; 

            gs.log("Error requesting O365 Service Messages.  Error Message:" + message); 

        } 

        if(json_responseBody != "failed to load"){ 

            for (i = 0; i < json_responseBody.value.length; i++) { 

                if(json_responseBody.value[i].Messages.length > 0){ 

                    this.updateOutage(json_responseBody.value[i].Id, json_responseBody.value[i].Messages[json_responseBody.value[i].Messages.length-1].MessageText + "<BR><BR><u>Last Updated: " + json_responseBody.value[i].Messages[json_responseBody.value[i].Messages.length-1].PublishedTime + "</u>"); 

                } else { 

                    this.updateOutage(json_responseBody.value[i].Id, json_responseBody.value[i].ImpactDescription); 

                } 

            } 

        } 

    }, 

    updateOutage: function(str_number, str_details){ 

        var gr_outage = new GlideRecord('cmdb_ci_outage'); 

        gr_outage.addQuery('u_number', str_number); 

        gr_outage.query(); 

        if(gr_outage.next()){ 

            gr_outage.setValue('details', str_details); 

            gr_outage.update(); 

        } 

    }, 

    getAzureAPIAccessToken: function(){ 

        var token = ""; 

        try {  

            var r = new sn_ws.RESTMessageV2('Microsoft Azure', 'Get Access Token'); 

            r.setStringParameterNoEscape('ClientSecret', gs.getProperty('AzureAccessToken.ClientSecret')); 

            r.setStringParameterNoEscape('ClientID', gs.getProperty('AzureAccessToken.ClientID')); 

            r.setStringParameterNoEscape('TenantID', gs.getProperty('AzureAccessToken.TenantID')); 

            var response = r.execute(); 

            var httpStatus = response.getStatusCode(); 

            if(httpStatus == '200'){ 

                //Success 

                var str_responseBody = response.getBody(); 

                var parser = new JSONParser(); 

                var json_responseBody = parser.parse(str_responseBody); 

                token = json_responseBody.access_token; 

            } else { 

                gs.log("Error requesting Azure API Token.  httpStatus:" + httpStatus); 

            } 

        } catch(ex) { 

            var message = ex.message; 

            gs.log("Error requesting Azure API Token.  Error Message:" + message); 

        } 

             

        return token; 

    },     

    type: ‘O365OutageSync’ 

};

Create Scheduled Job:

Create new scheduled job “Refresh Office 365 Business Service Outages”

Script:

var obj_outageTools = new O365OutageSync(); 

var token = obj_outageTools.getAzureAPIAccessToken(); 

obj_outageTools.refreshO365OutageRecords(token); 

obj_outageTools.refreshO365OutageMessages(token);

View solution in original post

John Shores1 · ‎08-19-2020

This is great! Has anyone done this with Azure DevOps? Its system status is exposed via endpoint too.

BDE95 · ‎06-29-2020

Hi James,

I found this very helpful, thank you so much for posting.

How should we handle the authentications if the test response of the run show "Invalid username/password combo"? In our 365 tenant, we do have legacy authentication blocked, and all of our accounts also use MFA, so I'm not sure if maybe that has some contributing factor. The "Get Access Token" method worked fine and I was able to retrieve the token.

jamesmcwhinney · ‎06-29-2020

Sorry I'm still finding my way around O365 and azure AD as well.

All I can suggest is to double check you have granted the right permissions to your application inside "App Registrations" and that your global admin has approved the request.

josjordan · ‎03-25-2025

it appears that the links are no longer valid.. anyone have updated instructions? Thanks! --joe