Elasticsearch Connector Term filters - class.java.util error

[Kelly Logan]

We are configuring Health Log Analytics on a Yokohama instance. One of our sources is Elasticsearch and we are using the OOB pull connectors. We have been able to connect and pull logs successfully, but now we are trying to set the 'Term filters' field, these are meant to limit the logs pulled to those matching the values given.

The documentation example is:

{"severity": ["error", "warning"]}

(How do these code inputs not have JSON?!?)

The example works, but is useless for our purposes. The values we need to match are in a second layer of JSON, and I think that's what the problem is.

When I try to use an actual value:

{"data_stream":{"dataset":"fileexchange.generic"}}

I get the following error:

 Elasticsearch full activation failed. Error while starting data input: class java.util.LinkedHashMap cannot be cast to class java.util.List (java.util.LinkedHashMap and java.util.List are in module java.base of loader 'bootstrap')

The ultimate goal is to query for multiple datastream types, something like:

{"data_stream":{"dataset":["fileexchange.generic","babelmessaging.generic"]}}

Any ideas? It seems like whoever wrote the parser for that 'Term filters' field didn't consider that there might be multiple layers of JSON, but if there is another formatting trick I'm missing, please let me know.

[charleselite · Accepted solution]

HLA Elasticsearch connector, only accepts flat key-value pairs in the Term filters JSON. This is because of the how the pull method function works via the mid server.  If you write the term filter in the flat model, something like: {"data_stream.dataset": ["fileexchange.generic", "babelmessaging.generic"]} the filter should work. 

[Kelly Logan]

Yes, that's what we found. So the solution was to start from the base log index prefix "logs-*" and then use a flat term filter and have an input mapping script that assigns out the different logs types. 

{"data_stream.dataset" : [ "acct.extract.ent.import", "acct.extract.ent.delivery", "fileexchange.generic", "fileexchange_uat.generic", "fileexchange.entitlement", "stale.fe.ent.generic", "controlm.generic", "controlmjob_capacity.generic", "controlm.metrics", "babelmessaging.generic" ] }