Garbled Japanese characters of company records imported by MID server

marloii
Tera Contributor

3 weeks ago

Hello! We discovered some core_company records imported through the MID Server that have garbled characters (Shift-JIS encoding) in their names, which are originally in Japanese.

When we restore these garbled records, we usually find that a separate record with the same name already exists. As a result, we end up deleting the garbled records and using the existing records instead.

What could be the possible cause(s) of this issue? Could this be related to character encoding or the way the MID Server handles the imported data?

I apologize if this is a basic question, as I don't have much experience with MID Servers. Any advice or suggestions on where we should investigate would be greatly appreciated.

#mid-server #itom #import #discovery

 

0 Helpfuls
  • 357 Views
1 REPLY 1

BharatC
Giga Guru

3 weeks ago

Hi @marloii 

 

Likely cause: A character encoding mismatch. Shift-JIS bytes are being decoded as UTF-8 or Windows-1252 somewhere in the import path. The garbled name then fails to match the existing correct record when the transform map coalesces on name, so a new duplicate record gets created.

Where to check:

Staging table: Is the name already garbled in the import set row? If yes, the problem is in the read (data source, MID Server, or source). If it's clean in staging but garbled in core_company, check the transform map or its scripts.

Data Source record: Check the encoding/charset setting (File) or the connection URL charset parameter (JDBC, e.g. characterEncoding=MS932).

MID Server JVM: If the host has a non-Japanese locale, the default file.encoding may be wrong. Set -Dfile.encoding=MS932 (or UTF-8, depending on the source) in wrapper-override.conf and restart the MID Server.


Source file: Confirm the actual encoding in Notepad++ or VS Code (Shift-JIS vs. CP932 vs. UTF-8).

Multiple imports or MID Servers: Another integration may be loading the same companies correctly. Compare their settings and the creation timestamps of the garbled records.

Prevention: Coalesce on a stable ID instead of name, and re-point references before deleting garbled records.

Menu paths and field names vary by release, so please verify against the docs for your version.

 

 

Hope this helps!

If this response helped, please mark it as Helpful.
If it resolves your issue, please Accept it as Solution.

Kind Regards,

Bharat C

0 Helpfuls