Kubernetes Discovery: life cycle strategy for batch containers

[danielherre]

Hi colleagues!

We have been running discovery against our openshift environment, and we see for [cmdb_ci_oslv_container] all CIs have install_status set to Installed, while the [cmdb_ci_oslv_container] items show a correct kubernetes status (running/failed/succeeded).

We realize we are missing a retirement strategy, otherwise containers in install status accumulate over time, 

and since they remain represented as CIs, the container count only grows.

Is there any best practice to implement a life cycle to retire succeeded or failed containers which are no longer running?

We've searched around but found nothing, if anyone could provide some experience it will be much appreciated, so we can implement a robust standard solution for these ephemeral workloads.
(We are still using the legacy install_status).

Thanks!

[danielherre · Accepted solution]

Here after some testing I think I can answer myself, in case it helps anyone.

The docker containers we see on Succeeded State are batch ones. Its operational_status is Non-Operational and kubernetes colleagues have confirmed such kind of containers are deleted from the kubernetes clusters  after 72h from the Succeeded event. 

So to align with this behavior, our strategy is to enable a Data Manager Policy to retire docker containers after 72h when state is succeeded.

[MattSN]

It should auto delete the containers marked absent within a few hours. 

Discussed in detail here:

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB2601800

[danielherre]

Hi @MattSN here we are not implementing KVA yet, but just kubernetes Discovery. We have explored enabling Data Manager policy to retire succeeded containers, on the initial tests it behaves well apparently.