Scraping data from Service Now without the API

[BDunbar]

I need to use the API without being able to use the API

We have CMDB data in our Service Now environment.  It would be tremendously helpful for me access this data programatically.  However, I cannot use the API for complicated political reasons.

What options do I have to use my standard Service Now account to scrape or obtain the data?

Simple Use Case

I have a report: server X, Y, and Z need an update.

My script would  grab the server name, look at the server record in CMDB for the field 'Supported By', and fire off an email to that person 'Your server needs attention, please attend to this before the bad guys discover the unpatched resource'.

Thank you in advance,

Brian Dunbar

[Roger Poore]

Check out my article about mass exporting KB articles to PDF's:  https://community.servicenow.com/community?id=community_article&sys_id=6c46130edb135c109e691ea668961...

Essentially, it automates opening up Internet Explorer, going to a form and printing it to the Microsoft PDF printer.  Now, it sounds like you're wanting to get at the content at the DOM level.  Depending on the authentication used, that's going to be the trick part.  Basic auth is easy but if you're doing SSO, that'll take some work.  I've never been successful.  

At any rate, the article I linked to will give you a head start in at least opening a web page through an automated process.  You just need to to scrape it rather than save as a PDF.

I'll be happy to help if you need it.

 -Roger

[BDunbar]

That's really helpful!

I was digging around last night after I posted this question: if I can export the data to text/csv .. well then I can just use standard unix text processing commands to sort out the data.

I only need to schedule a task to export the data I want and hey presto: a reasonably up-to-date local copy of our CMDB.

Thanks!

bdunbar

[Dariussiriuse]

Heya!

I'm aware that the article you shared is from a few years ago, but it is now archived and therefore not accessible anymore. Would you have any articles in mind that I could use for this?

Darius

[Roger Poore]

Hey @Dariussiriuse 

I think that article was regarding exporting KB articles as PDF's in bulk. Below is the PowerShell script I used to do that.  Rather than screen scraping, I'm cheating.  I'm just programmatically hitting the print button and saving the article to a PDF.  Make sense?

HTH

 -Roger

# used to export KB articles to PDF.
# The $kblist is generated from the get_kb_sysids.js script
# This is a proof-of-conecpt and currently doesn't handle authentication.  I used my
#   PDI and opened up this particular Knowledge Base for anonymous access.
$window_title = "Save Print Output As"

$kblist = Get-Content C:\temp\kbs.csv
foreach ($line in $kblist) {
    $m = $line.Split(',')
    $sysid = $m[0]
    $desc = $m[1].Replace("/", "-") + ".pdf"    

    $url = "https://<YOUR-INSTANCE>.service-now.com/kb_view.do?sys_kb_id=" + $sysid + "&sysparm_stack=no&sysparm_force_row_count=999999999&sysparm_media=print"

    $ie = New-Object -comObject InternetExplorer.Application
    $ie.navigate($url)
    while ($ie.busy) { sleep 1 }
    $ie.visible = $true
    $ie.ExecWB(6,2)

    $wshell = New-Object -ComObject wscript.shell;
    $wshell.AppActivate($window_title)
    Start-Sleep -Seconds 3
  
    if ($wshell.AppActivate($window_title)) {
        Start-Sleep -Seconds 2

        [System.Windows.Forms.SendKeys]::SendWait($desc)
        [System.Windows.Forms.SendKeys]::SendWait('~')
        # Break
    } 

    Start-Sleep -Seconds 5
    
    taskkill /F /IM iexplore.exe  
}