Agent Client Collector not connecting correctly

[Ansly21]

Hello everyone, we have an issue in which Agent Client Collector is connecting to the wrong instance.

To give a background, we have three instances: Dev, Test, and Prod. We have first started testing ACC in our DEV environment, so we enabled the WebSocket and generated an API-Key.

So, we first enabled the WebSocket for a MID servers, let's name it MID A. Few agents have already connected successfully and is reporting in our DEV instance. We wanted to migrate it to our Test instance, so first we have disabled the WebSocket for MID A in Dev instance and enabled MID A WebSocket in Test instance. Somehow, it was able to report in our Test instance, even though we haven't configured the api-key. (Note: IP address and port is the same + API key is uniquely generated in each instance and applies to all MID servers)

We are thinking on how it was able to connect, even if the api-key is wrongly configured.

Previously, we were able to repoint our agents with by changing the api-key.

Now, when we do that, it somehow goes to our Prod and Test instances randomly.

Also, when we performed the "acc self-test,"  we saw that the authentication is failing, but in the logs, it is showing the "successfully connected..."

Has anyone encountered the same issue of repointing agents to a different instance? If so, any help is appreciated. Thanks!

Regards,

Ansly

[ayushraj7012933]

Hi @Ansly21 

I faced something similar a while back. In my case, I first checked whether the ACC agent was using the correct instance URL and API key. I also made sure the MID Server was connected only to the intended instance.

If your Dev, Test, and Prod instances are cloned, it's worth generating a new API key and re-enrolling the agent. That helped rule out any old configuration.

Also, seeing "Successfully connected" in the logs doesn't always mean the authentication was successful. It just means the WebSocket connection was established. Since the ACC self-test is showing authentication failed, I'd focus on the agent configuration and authentication rather than network connectivity.

Just curious, are your Dev, Test, and Prod instances cloned? And are you using the same MID Server across all three environments?

[Ansly21]

Hello @ayushraj7012933 ,

Actually, yes, recently our Dev and Test instances got cloned and we are using the same MID servers across all environments. That is the time when our 2k+ agents that were previously connected in Dev instance, moved to our Test instance, even without changing anything. (Note that all of our WebSockets in Dev instance have been disabled, since we enabled them in our Prod instance instead.) There is also one WebSocket active in Test, in which we also used in Dev previously.

I was also thinking if that cloning was the issue, but I don't see any changes in those API keys across all three instances, they are still the same records they used to be before the cloning.

It's reassuring to know that someone actually faced a similar issue.

Thank you for the response!

So yeah, in parallel, we have raised a case to ServiceNow to find the fix.

[drbob]

Cloning MID server [ecc_agent] records doesn't work - you end up with dead records. If you have MIDs A/B/C in PROD and clone down to TEST which has D/E/F then after the clone the correct situation is that PROD still has A/B/C and TEST has D/E/F. This is the OOB behavior - there are clone rules that exclude (prevent copying to target) and preserve (keep what is in the target) these records so that they are unchanged by a clone.

As PROD is the source (in my example) that will never vary, it will always be A/B/C but depending what has been changed with the clone rules TEST could end up with any of these combinations...

A/B/C - both rules are deleted so TEST becomes a copy of PROD, as is the case with most tables

A/B/C/D/E/F - preserved but not excluded

<none> - excluded but not preserved

D/E/F - excluded _and_ preserved (what we want)

It makes no sense* to copy these records between instances as the MID server itself is configured to communicate with a specific instance - if A/B/C are copied down to TEST they will get no updates and eventually trigger some warning in TEST because those MID servers are talking to PROD. ...and if D/E/F go away due to the clone then the MID servers will talk to the instance, find no related record and create new ones (which will need to be validated before they work). You lose some history and have to do the validation but otherwise no harm.

*I worked somewhere where we _did_ copy the MID server records because we wanted to monitor them for issues all in PROD. We didn't want to have to log into PROD, TEST, and DEV to run the same tests ...and mail is disabled in non-PROD so we opted to copy all the records to PROD and we had a process where changes were copied up too so if a MID server went down in TEST we could see that in the dashboard in PROD. The clone rules were still OOB though so TEST only had TEST MIDs listed, DEV only DEV MIDs ...just PROD that had them all.