There are two critical issues that all Splunk users need to be aware of around the use of timestamps in Splunk, one of which is highly time sensitive (see bottom of this post for info on how to fix the issue).
- Beginning on January 1, 2020, Splunk will no longer correctly identify timestamps that use two digits to signify the year.
- Beginning on September 13, 2020, at 12:26:39 PM UTC, Splunk will no longer identify timestamps based on Unix time.
Patches to the Splunk software will need to be applied before the start of the new year to avoid indexing data in Splunk with the wrong timestamp. Unpatched Splunk instances will either add timestamps using the current year or will misinterpret the date and add a timestamp with the misinterpreted date.
What does this mean for you?
Applying the correct timestamp to your events is critical to healthy operation of your Splunk environment. Here are some issues that could come from using an incorrect timestamp:
- Basic searching: searching your data in Splunk depends on each of your events having the correct timestamp. Searching the time range of the “Last 24 hours” could miss the events with incorrect timestamps showing up in a completely different year
- Reports/Dashboards: if new incoming data does not have the correct timestamp, your reports, alerts, and dashboards could miss the events
- Security incidents: your security team could be missing potential security incidents or suspicious activity on your network
How to fix this issue?
Before January 1, 2020, the patch provided by Splunk will need to be applied to resolve this issue. Splunk Cloud customers should be contacted by Splunk support and will automatically receive the update to the Splunk components in the cloud. Splunk Cloud customers will still need to patch any on-premise Splunk instances. For Splunk Enterprise customers, there are 3 options to apply this patch to your Splunk instances:
- Download an updated version of the datetime.xml configuration file and apply it to each of your Splunk instances
- Manually modify the datetime.xml configuration file on each of your Splunk instances
- Upgrade Splunk to a version that includes the patch
Additional details on how to patch this issue can be found in the Splunk Documentation.
If you need assistance or have questions about this Splunk issue, please reach out to us.