0

several Splunk dashboards stopped working

Hello,

We only have the uberAgent app inside Splunk. Since a few weeks we are encountering issues when opening them. 

We are getting the following erorr:

"The search job terminated unexpectedly"

Some dashboards work perfectly, others won't load at all. We haven't made any changes for several weeks. Rebooted Splunk and the server did not resolve anything. 

I can't seem to find the reason behind it. How do I start troubleshooting this?

8 comments

  • Avatar
    Timm Brochhaus Official comment

    Hi Sven, 

    Could you please share the following additional information with us: 

    1. Which dashboards are affected? 
    2. Which Splunk and uberAgent version are you using?
  • 0
    Avatar
    Sven Goossens

    I can see some errors regarding but I'm not sure if related.

    ERROR SearchOperator:newchart - Error in 'timechart' command: The data field '[stats count | addinfo | eval range = info_max_time-info_min_time | eval span = "span=" . case(range < 1200, "15s", range < 15000, "1m", range < 90000, "5m", range < 300000, "15m", range < 700000, "1h", 1=1, "1d") | return $span]' is malformed.

  • 0
    Avatar
    Sven Goossens

    Hi Tim,

    I have run through all the dashboards. The problems only occur at these dashboards:

    • User Logon Duration
    • User Logon Duration -> only data table element
    • User Logoff Duration

     

  • 0
    Avatar
    Sven Goossens

    Splunk 8.0.6

    Uberagent app in splunk is version 5.2.1

  • 1
    Avatar
    Timm Brochhaus

    Hi Sven, 

    This issue related to missing data in the uberAgent dashboards User Logon Duration and User Logoff Duration was reported by other customers. During further investigation, it turned out that this issue exists since Splunk Enterprise version 8.0.4. One of our customers downgraded to Splunk Enterprise version 8.0.3, and the issue doesn't exist anymore.

    From our perspective, it seems that there is a problem with new Splunk releases. We are aware of open support cases at Splunk regarding this.

    There are only two dashboards affected because we are using a feature called Splunk transactions only there. Independent of this issue, we decided to get rid of Splunk transactions in the upcoming release uberAgent 6.

    Based on that, I would recommend one of the following actions:

    • Downgrade to Splunk Enterprise 8.0.3. A download is available here.
    • Try out the public beta 2 of uberAgent 6. You could download the beta version here. We are planning to release the final version soon.

    I hope that helps.

  • 0
    Avatar
    Sven Goossens

    Hi Timm,

    Thank you for your answer.

    Can I install the public beta 2 app on my Splunk installation without any issues? Or will this break something?

    I'm probably safe if I just don't use the UXM agents in a bèta version, correct?

    best regards,

    Sven

  • 0
    Avatar
    Timm Brochhaus

    Hi Sven, 

    We highly recommend not to use the beta version of uberAgent in production. When using uberAgent 6, you have to update both the Splunk app and the endpoint agent.  

    So, in this case, I recommend either to downgrade your Splunk version or to wait for the final uberAgent 6.0 release. We are planning to release the final version at the end of this month.

     

  • 0
    Avatar
    Sven Goossens

    Issues have dissappeared since updating Splunk to 8.1.0.0

Please sign in to leave a comment.