Toggling serial number after Australia update

[johanalande]

Hi,

After upgrading to Australia two weeks back we are experienceing toggling/flipping of serial number and incorrect serial numbers in the CMDB. From Discovery. This seems to affect Linux server only, but not all of them, guestimate is about 40% of the machines have incorrect serial number in cmdb_ci_linux_server.serial_number. Physical servers and virtual are both affected. Windows servers seems unaffected.

Looking at the history on one of the affected CIs I can see that the issue started the first Discovery-run after the upgrade. And that serial number on the CI is set twice, first to the correct (system), 3 seconds later to the incorrect (uuid, or baseboard, or... ).

Looking at the CI I can see that the system property
sn_itom_pattern.linux_server_preferred_serial_number_type
is being ignored.
The only "thing" around the error that seems consistent, but still incorrect, is that the serial number from the cmdb_serial_number table that is "top" when sorting on Absent (unsorted?) is the one being picked for being set to the CI.

Data from Discovery (IRE payload) contains the cmdb_serial_number data that is being updated, and the CI data that is being updated. The CI data does not contain a serial number.
From this I understand that it is not Discovery that is the issue, but rather a process afterwards. IRE? BRs?

Version Australia. Patterns OOB.
Any ideas? Anyone else seen this?
BR/Johan

[kmohammed]

Hi Johan,

Your analysis is correct, and the 3-second gap is the fingerprint: the serial_number field on the CI is maintained from the Serial Number [cmdb_serial_number] child records after the IRE payload is processed, so when that sync runs it re-derives the CI's serial from whichever serial record it picks up first. With several types present for the same CI (system, baseboard, UUID) and no deterministic ordering on that lookup, the "top" record wins — which matches exactly what you see in cmdb_serial_number sorted by Absent (all rows absent = false, so the order is effectively arbitrary), and why the value flips between runs.

Why the property doesn't protect you: sn_itom_pattern.linux_server_preferred_serial_number_type is consumed pattern-side, when Discovery builds the payload. Since your payload (correctly) contains no serial_number on the CI itself — only cmdb_serial_number rows — nothing pattern-side can control which row the post-IRE sync promotes to the CI field. That's also why it hits physical and virtual alike, and only ~40%: only hosts whose dmidecode returns multiple serial types have more than one candidate row; single-serial hosts can't flip.

How to pin the process down:

- In the affected CI's audit history, compare the two serial_number writes — the second write's origin/updated-by tells you the mechanism (a business rule on cmdb_serial_number insert/update vs. a scheduled or async process).

- Check whether any business rules or script includes touching cmdb_serial_number or the CI serial_number were changed by the upgrade (System Diagnostics > Upgrade History, filter the Australia release). Starting exactly at the first post-upgrade Discovery run points to a changed out-of-the-box component rather than your data.

Given OOB patterns and a clean payload, this is defect-shaped — worth a Support case with the CI audit trail and a payload excerpt attached. As an interim mitigation, a small scheduled repair job can re-align the field between runs: query cmdb_serial_number for your Linux server class where serial_number_type matches your preferred type and absent = false, and set the CI's serial_number from that row when it differs. It doesn't fix the sync, but it stops the wrong value from persisting.

Hope that helps narrow it down.

Karim