How to ignore a row in ETL mapping based on a condition

Blessy2
Tera Contributor

‎04-22-2025 06:52 AM - edited ‎04-22-2025 06:54 AM

Hi Team,

 

We are working on GitHub integration and trying to pull Repos into 'A' table. This is done using Integration Hub ETL.

The requirement is to load all the 'Code' type repos in 'A' table and 'document' type repos in 'B' table.

From the payload I tried to distinguish between code and doc type repos. I created a column in ETL source data step to store the type of the repo(attachment 1).

Now from the data I know which record is of what type.

 

During the mapping, I want to ignore all doc type repos and load only code type to 'A' table, vice-versa. 

I'm trying to check if I can do it using any transform types available (attachment 2).

How do I achieve this, request to suggest?

Please find the attachments.

 

Blessy2_0-1745329750464.png

 

Blessy2_1-1745329843347.png

 

Thanks in Advance.

Labels:
0 Helpfuls
  • 351 Views
1 REPLY 1

Community Alums
Not applicable

‎04-22-2025 11:56 PM

Hi @Blessy2 ,

Hi, I'm not entirely sure if this is the definitive best practice, but based on my understanding of Integration Hub ETL

To load 'Code' repositories into table 'A' and 'Document' repositories into table 'B' using Integration Hub ETL, you should clone your initial data stream. In the first cloned stream, add a "Filter" transform to only allow records where the 'repo_type' is "Code" and then use a "Table Load" transform to load this filtered data into table 'A'. For the second cloned stream, apply a "Filter" transform to include only records where the 'repo_type' is "Document" and subsequently use a "Table Load" transform to load this filtered data into table 'B'. This approach leverages the "Filter" transform within separate data streams to achieve the conditional routing of your GitHub repository data.