ETL - Script operation - Create entity record in stage/temp entity

Not applicable

My company has an interesting use case where we bring outage data from Azure and store it into the Outage table.  Multiple validations are performed before transferring the data to the Outage table.  One possible case is if an incoming outage started within a change window but ended after the change window.  The expected result should be 2 outage records;  

 

Outage 1

Begin:  <incoming outage start date>

End:  <end of change window>
Outage type:  Planned

 

Outage record #2

Begin:  <end of change window + 1 minute>

End:  <incoming outage end date>
Outage type:  Outage

 

My ETL definition is the following:

  • 3 entities:  ImportSet, stage, Outage (cmdb_ci_outage)
  • stage entity has 1 input script operation (called Process outage)
  • Mappings:  ImportSet -> stage, stage -> Outage [cmdb_ci_outage]

 

The business logic is built within stage's script operation.  Is there any way I can create another record within the stage entity that captures Outage record #2 above?

 

Within the stage script operation, I tried adding it to the end of the output array but was getting an "infinite recursion" error.

 

3 REPLIES 3

chaithu1489
Kilo Sage

I am not an expert in ETL. As you are loading only data to outage table.i would use transform maps and just write this logic in a transform map script.

Not applicable

Thanks for the response.

 

Currently that it was I am using is a transform map but we're planning on using Stream Connect/Kafka which requires RTE/ETL for the transform.