1. uberAgent Support
  2. Community
  3. General Discussion
New post
0

Strange Error in Splunk regarding index name (variants of u, ub, ube

Avatar
mel strathdee

Periodically we see the following errors in Splunk:

Search peer serverxxxxx has the following message: Received event for unconfigured/disabled/deleted index=ube with source="source::tcp:19500" host="host::machinexxx" sourcetype="sourcetype::dummy". So far received events from 4 missing index(es).

We have uberagent data going to the default index as specified in the config. - uberagent

 

Any ideas how we can fix these strange events errors

Date Votes

1 comment

  • 0
    Avatar
    Dominik Britz

    This is not the first time we see this issue. Any time we saw this issue, uberAgent was not the cause. The cause was either an overloaded Splunk server (like a heavy forwarder sitting between endpoint and indexer) or something in the network which manipulated the TCP packages. While we do not have a solution to the problem, we have a list of steps you could do to narrow down the issue. The goal is to isolate a (possibly) corrupted link in the chain.

    Let me start by showing you what a healthy event should look like:

    ***SPLUNK*** host=HOSTNAME index=uberagent source=uberAgent sourcetype=uberAgent:Process:ProcessDetail
    1541605682327,dllhost.exe,0,0,0,0,0,0,0,0,,,11.6,0,Domain\Username,,,MsWnOS,,29052,,

    And this how these corrupted events look like:

    ***SPLUNK*** host=HOSTNAME index=uber..^...........E.3|h.@......\...\...9L,'.$0{/#.P. ..{..agent source=uberAgent sourcetype=uberAgent:Process:ProcessDetail
    1541605682327,dllhost.exe,0,0,0,0,0,0,0,0,,,11.6,0,Domain\Username,,,MsWnOS,,29052,,

    As you can see the index name is manipulated and Splunk does not know where to put the event.

    1. Configure a lastChanceIndex
    If an event arrives whose index destination key points to an index that is not there, it will route that event to the index specified by this setting. The index destination key of that event will be overwritten with the specified index name before routing.
    Note that <index name> must name an existing enabled index. Splunk will not start if this is not the case!
    Source: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf

    lastChanceIndex brings the following benefits:

    • You will see the number of corrupted events, as they are all together in one dedicated index
    • You will see if all, a subset, or only one indexer are affected
    • You will see if all, a subset, or only one endpoint are affected

    2. Add additional receivers in the uberAgent config
    uberAgent sends data to all receivers in its configuration. You likely have only one configured at the moment, your production Splunk environment. Please add two more receivers and, with that, two more Splunk servers:

    1. A Splunk server sitting in your network. This could either be your existing test Splunk environment or a new one dedicated to this test.
    2. Install Splunk locally so that uberAgent does not have to send data through the network, ruling out the network as a problem.

    3. Do a Wireshark trace on endpoint and server

    If you identified affected endpoints and servers, do a Wireshark trace on both ends and search for the corrupted event on the endpoint. The expected result is that you see the corrupted event only on the server but not on the endpoint. That means uberAgent sends the event correctly, and it gets manipulated somewhere.

Please sign in to leave a comment.