Upgrading from 5.2.1 to 6.1.1 in Distributed Splunk Environment


We're looking to upgrade to uberAgent 6.1 from uberAgent 5.2.1 in a Distributed Splunk environment with a multi-site index cluster, multiple search heads and a few heavy forwarders running uberAgent Splunk apps.

According to the upgrade documentation (link below) the high level plan seems to be:

1) Clear KV stores and uninstall Splunk apps in all required locations.
2) Install new Splunk apps in all required locations.
3) Upgrade the endpoint agents.

Once I do step 1) what happens to all the data coming in from uberAgent endpoints before I install the new Splunk apps in step 2)? We have the Splunk index defined in our cluster master managed indexes.conf file so the splunk Index will still exist. This means that, theoretically, data can still write there if required. But if I literally uninstall the indexer app there will be Splunk ingest time parsing config that will no longer be accessible to incoming data. Theoretically this data will still get written to the index, this time without proper parsing, for a period of time until the new ones are installed. Is that ideal?

What happens after I've done 1) and 2) and endpoints are still running 5.2.1? Will the data coming in from them work with the new Splunk apps until which time all the endpoints are upgraded too? Or is the data coming in from 5.2.1 endpoints incompatible with the Splunk ingest time parsing config present in the new 6.1.1 indexer app?

Basically I have some concerns that at various stages of this plan various Splunk components, e.g. Indexers & Heavy Forwarders, will potentially be running different, or no, version of the Splunk apps and am keen to understand what this will mean for the fleet of endpoints running in our environment and the data they are sending in.

Perhaps there is an alternative plan where I disable all inputs collecting the incoming endpoint data so it can't get into Splunk until all the apps are upgraded and re-installed. I'm unsure what that would do the endpoints in terms of them having to buffer that data until the inputs are enabled again. Also that still doesn't solve the second question about the endpoints being different version... unless I didn't re-enable the inputs until they were all upgraded too.

Cheers for any answers/tips!



  • Avatar
    Martin Kretzschmar Official comment

    Hi Michael,

    When updating the uberAgent_indexer app within an indexer cluster, Splunk ensures service availability by performing a rolling restart as described here.

    As for your second question, some dashboards will probably show no or faulty data for endpoints running an older version. You don't have to expect parsing issues though.

    Disabling the uberAgent data input until all Splunk apps have been upgraded, is also a possible approach. The default uberAgent buffer size on the endpoints (10 MB) should be sufficient for a few hours.

    Kind regards, Martin

  • 0
    Michael Jorgensen

    Thanks for the response Martin. I'm still looking into our exact detailed plans. I might pop back on here with future questions as they come up.

Please sign in to leave a comment.