How do we best model parts of a data lake?

Kristine Naess · ‎03-15-2022

Hi all,

We are a company utilizing Teradata's enterprice data warehouse solution as our data lake. Since we use it for all kinds of purposes (multiple business processes, financials as well as risk & compliance), we need to create a structure that's meaningful to all of our stakeholders - be it business analysts, data analysts, other business applications, compliance officers, etc. At the same time we need to be able to address alerts and events, plus incidents and changes. Not to mention GDPR DPIA's and other regulatory policies.

The databases that are created frequently as part of the architecture need to be tagged and identified so that we have good information governance and access management.

How should we go about this in a "best practice" CSDM 4.0 way?

All recommendations are very welcome since we are already in need of tags for our newly created databases.

Cheers,

Kristine

Mark Bodman · ‎10-05-2023

The APM product provides Data Domain on top of the Information Object that we have in CSDM. The intended use when we implemented Information Objects in APM is to understand what data is being used / stored in each Business Application. This would typically trigger a specific type of audit, such as PCI audits when storing and using Credit Cards for example.

What you seem to be describing is "data as a product" in that data is a deliverable stream or package that one team provides another team to consume and use in some way. If this is the "product" use case, then extending the the product model types to include a Data Product Model this may be an appropriate approach. This would also allow you to articulate a product catalog and catalog item that would allow consumers to obtain these data as a type of product.

My main caution is that this will add a lot of extra management overhead, and I would encourage you to work with those who provide and consume data in this way be at the table to understand what level of manual labor is required to initially document this level of detail, and then to maintain going forward. It's franky hard enough for most companies to manage their Business Applications and Services at an appropriate level, let alone data products in this way. If the providers and consumers are willing to do it, then knock yourself out. : )

View solution in original post

Mark Bodman · ‎10-05-2023

The APM product provides Data Domain on top of the Information Object that we have in CSDM. The intended use when we implemented Information Objects in APM is to understand what data is being used / stored in each Business Application. This would typically trigger a specific type of audit, such as PCI audits when storing and using Credit Cards for example.

What you seem to be describing is "data as a product" in that data is a deliverable stream or package that one team provides another team to consume and use in some way. If this is the "product" use case, then extending the the product model types to include a Data Product Model this may be an appropriate approach. This would also allow you to articulate a product catalog and catalog item that would allow consumers to obtain these data as a type of product.

My main caution is that this will add a lot of extra management overhead, and I would encourage you to work with those who provide and consume data in this way be at the table to understand what level of manual labor is required to initially document this level of detail, and then to maintain going forward. It's franky hard enough for most companies to manage their Business Applications and Services at an appropriate level, let alone data products in this way. If the providers and consumers are willing to do it, then knock yourself out. : )

Kristine Naess · ‎10-10-2023

Thanks for a very clear and good answer, @Mark Bodman .

It is actually the devops teams that are delivering Master Data as a Service that have brought up this question. For instance the Supplier Master Hub that delivers data both to TPRM, S2P, Accounting, ESG, CRM, ERP, etc. They have a data model describing how the information objects they provide should be used and understood. For example, some information objects within the Supplier Master data model, such as the Third Party contact, belong to the PII data domain, whereas the Supplier Carbon Footprint score belong to the ESG Data Domain. There's also a relationship, maybe even many, between these information object which they need to describe. This is done in relation to what they call their "Data Products". In other words, using just the Data Domains as an entity will not solve all of their needs. So I have trouble dedicating ownership to the Supplier Master as a Data model without something to pin it to. Of course, they deliver a set of service offerings, basically in the shape of various APIs and API scopes one may subscribe to, but if the data model itself were to change, this would also impact the usage of those.

The Common Service Data Model is another good example. In version 5 a set of new artifacts turn up, such as the Value stream. Then I need to create descriptions on what to put in the value stream table, what it is for, and what columns we are currently using. We may also need to add some norms around the governance of the records in this table, like we have done for BusApps, BusCaps, etc. Should the understanding of a Value stream change in the future, this needs to be versioned.

The catalog items that could be related to a Data Product Model as you suggest, can in our case either be solved by the new Digital Integration Management solution, and be catalog items under each service offering we have for our BusApps. Or: ensuring that all those who subscribe to APIs on the previous Data Product Model version are notified when there is a change, and to let them stay on the old version until they are ready to move to the next, could be a good use case for having data product model-based catalog items for those API-types of service offerings.

I do dread the user adoption side of introducing Data as a Product, as well as Data as Assets, but at the same time I want to be able to provide something that makes sense to these highly specialised, highly skilled architects who knock on my door. So I think we will give it a try ;-).

Cheers,

Kristine