UCF "duplicate" citations

[Maros Takac1]

Hi,

would anyone who has been using UCF be able to explain the rationale behind their duplicate citations ? in the screenshot below, we have a bunch of citations, imported from UCF, with the same name/description; linked to the same authority document. The only unique attribute value is a source ID. Each citation is linked to a single control objective. My expectation is that there is only one citation per authority document with the same name/description; which is then linked to one or many control objectives. More importantly, things get very tricky when one has to create a new control objective internally and then link it back to a UCF citation. In this case, which citation would i link it back to ? all of them ? Reporting on this becomes a nightmare as you can imagine.

@Eric Feron - would you be able to look into it and advise on the reason SNOW creates multiple "content identical" citations ? Is it imposed by the UCF data model ? 

You can re-create the "issue" using the info below.

UCF Authority Document: "Hong Kong Monetary Authority: TM-G-1: General Principles for Technology Risk Management"

UCF Citation Reference: '4.3.3'

Any help would be appreciated.

Many thanks

[Maros Takac1]

Hello Rafael,

thank you for your reply. I followed your link to the problem and read through its description: "PRB1372593 - Gathering data from UCF is generating duplicate records and losing relationships during the process has a new comment". The second bit which I highligted is not something we have experienced. This issue or "feature" has been around for ages and is baked into the entire UCF integration if you look closely at their scripts. In fact a while back I came across a knowledge article explaining the need to have multiple citations with a different source ID, which is weird. What I want to know is how citations come back from UCF into SNOW. If each citation comes in with a unique id irrespective of its duplicate content, then it is up to UCF to explain why that is. Otherwise, it can be recognized as aSNOW issue which I hope will be addressed in the next release as you said.

there is another issue which I forgot to mention in my initial description. If you happen to have a UCF account, I suggest you log in, select the Hong Kong authority document and then export it into a depth report. Under the citation guidance column, you`ll find mandates; primary/secondary nouns and verbs used to find a match between a citation and common control. That crucial piece of information does NOT get imported into ServiceNow, making it impossible for someone to know why those two entities were linked. The entire context is gone, missing, which is really the core of the UCF magic... Having said that, I believe the current SNOW-UCF model is still immature and somewhat incomplete unless of course I am lacking in my understanding of the rationale behind all this...

Cheers

Maros

[Community Alums]

Yes, thats exactly the issue.In the 1st time you load data from UCF is working fine but if you try to push again it will generate duplicate records losing sine relationships during the process. The fix behind the PRB will fix this issue.

Regarding your other question, you can probably tweak the UCF integration  in order to capture your fields. Everything starts in the UI script (sn_comp_ucf.DisplayUCFPopup) which points indirectly to the client script  (sn_comp_ucf.UCFAjax) and eventual script includes.


1. UI script: sn_comp_ucf.DisplayUCFPopup

updateSharedListAndShowDownloadPopup()

2. Client script: sn_comp_ucf.UCFAjax:

var result = new UCFDownloader().downloadAvailableSharedLists(configurationId);

3. Script include: sn_comp_ucf.UCFDownloaderBase

			else if (downloadType == 'authority document')
				this._saveAuthDocAndCitations(responseObj, integrationRecId);
			else if (downloadType == 'control') {
				this._saveControl(responseObj);

Never tried but the chances are you need to start to tweak this functions. Hope this gives you a good starting point

[Maros Takac1]

1. have a look at this community thread: looks like you have already had a similar discussion in the past. What is interesting are the replies from Richard Taylor and Jan Spurlin who support the idea of SNOW creating multiple citations to support multiple mandates, which is not the same as the issue you are referrring to. To them it is not an issue, it is a design decision.  

2. the starting point would be to check whether the UCF API exposes that information. there is no need to tweak scripts, all you need to do is to tweak transform maps where all the action happens.

hope that makes sense ?

[dsarazen]

So, it appears that this issue hasn't been resolved in the last two years. 

[Maros Takac1]

@Rafael Cardoso  it would appear that UCF treats every citation, even with the same name/description, linked to the same authority document, as a unique record. To prove this, I have connected to their API, the same endpoint used by SNOW, and have retrieved their JSON payload in which each citation is assigned a unique ID. Please note that this ID field is then used to coalesce on in UCF import transform maps. Furthermore, As you`ll see from the payload attached, a citation to common control mapping is a 1:1 relationship. I was not able to find in the payload any information on mandates used to match a citation to a common control either. 

You can check for yourself by searching my payload JSON file for citations with the following IDs: '119082', '119178', '119083', '119177'. These IDs correspond to the records shown in my initial screenshot. 

Using the script below, I was not able to find a single UCF citation linked to more than one UCF control objective in our pool of over 9000 UCF citation-control objective mapping records.

My aim is to organize a session with UCF to get them to clarify their design model and rationale behind it. Once I have received some feedback from them, I`ll let you know.

P.S: if you want a code snippet to quickly connect to and retrieve data from UCF, let me know and I`ll share it with you. The assumption is that you have a valid API key... 

Cheers

Maros

======== SCRIPT ==========

var gr = new GlideAggregate('sn_compliance_m2m_statement_citation');
gr.addAggregate('COUNT', 'sn_compliance_citation');
gr.groupBy('sn_compliance_citation');
gr.addEncodedQuery('has_imported=true^sn_compliance_citation.source=UCF^sn_compliance_policy_statement.source=UCF');
gr.addHaving('COUNT', 'sn_compliance_citation', '>', 1);
gr.query();

while(gr.next()) {
gs.info(gr.sn_compliance_citation)
}

======== END OF SCRIPT ==========