If you haven’t used tstats and data model acceleration (DMA) yet, it’s time to start! Splunk slides aren’t marketing fluff claiming false information regarding DMA and how much it can speed up your searches. You can literally make your searches ten to one hundred times faster. That being said it’s important to understand how tstats and DMA works before you start implementing this.
DMA works by essentially running a scheduled search (defaults to every 5 minutes) that populates what is known as a “high performance analytics store.” What does that mean? In the simplest terms it’s the opposite of how Splunk normally works.
Schema-on-the-fly aka Schema-on-read is utilized by Splunk when conducting searches, this means Splunk stores the data in its native format and enriches said data on read (ie when a search against the data is conducted). There are obviously many benefits to this schema on the fly approach, such as when new data arrives no changes are needed on how to ingest or store (index) the data.
Schema-on-write is utilized by traditional methods of storing and indexing data. Without going into a lot of details on this method, the key importance is that while being inflexible (breaks when the data changes or when users want to ask different questions) it is extremely fast. Schema-on-write also takes up additional storage space when comparing it against Schema-on-the-fly.
Because this is Splunk we are talking about, we have options, and can use both schema’s via tstats and DMA!
Check out Splunk’s official documentation for a more in depth understanding. http://docs.splunk.com/Documentation/Splunk/7.1.2/Knowledge/Acceleratedatamodels
Is data model acceleration right for you?
Ask yourself the following questions
- Do my users require real-time results?
- If Yes, stop
- If no, proceed
- Can my infrastructure handle additional data storage?
- If no, stop
- If yes, proceed
- If maybe, build a test and see what the impact will be. (HINT-the GUI will tell you the size of the data model after it has been accelerated).
- Do I have fairly static dashboards that are used heavily?
- If yes, proceed
- If no, proceed, but you’ll likely being doing a lot of work for your users!
If you’ve answered yes, you’re likely a good candidate for DMA. If not, try giving it a go regardless. DMA is a great exercise and you’re likely to run into this throughout your Splunk career!