Skip to main content
NetApp Solutions

TR-4869: NetApp StorageGRID with Splunk SmartStore

Contributors kevin-hoke
PDFs

Karthikeyan Nagalingam, Bobby Oommen, Joseph Kandatilparambil

Splunk Enterprise is the market-leading Security Information and Event Management (SIEM) solution that drives outcomes across the Security, IT, and DevOps teams. Data volumes continue to grow at exponential rates, creating massive opportunities for enterprises that can leverage this vast resource. Splunk Enterprise continues to gain adoption across a wider variety of use cases. As the use cases grow, so does the amount of data that Splunk Enterprise ingests and processes. The traditional architecture of Splunk Enterprise is a distributed scale-out design providing excellent data access and availability. However, enterprises using this architecture are faced with growing costs associated with scaling to meet the rapidly growing volume of data.

Splunk SmartStore with NetApp StorageGRID solves this challenge by delivering a new deployment model in which compute and storage is decoupled. This solution also unlocks unmatched scale and elasticity for Splunk Enterprise environments by allowing customers to scale across single and multiple sites, all while reducing costs by allowing compute and storage to scale independently and adding intelligent tiering to cost-effective cloud- based S3 object storage.

The solution optimizes the amount of data in local storage while maintaining search performance, allowing compute and storage to be scaled on demand. SmartStore automatically evaluates data access patterns to determine which data needs to be accessible for real-time analytics and which data should reside in lower-cost S3 object storage.

This technical report outlines the benefit NetApp provides to a Splunk SmartStore solution while demonstrating a framework for designing and sizing Splunk SmartStore in your environment. The result is a simple, scalable, and resilient solution that delivers a compelling TCO. StorageGRID provides the scalable and cost-effective S3 protocol/API-based object storage, also known as remote storage, allowing organizations to scale their Splunk solution at a lower cost while increasing resiliency.

Note Splunk SmartStore refers to object storage as remote stores or remote storage tiers.

About NetApp StorageGRID

NetApp StorageGRID is a software-defined object storage solution for large archives, media repositories, and web data stores. With StorageGRID, NetApp leverages two decades of experience in delivering industry-leading innovation and data management solutions while helping organizations manage and maximize the value of their information both on-premises and in public, private, or hybrid cloud deployments.

StorageGRID provides secure, durable storage for unstructured data at scale. Integrated, metadata-driven lifecycle management policies optimize where your data lives throughout its life. Content is placed in the right location, at the right time, and on the right storage tier to reduce cost. The single namespace allows the data to be accessed via a single call regardless of geographical location of the StorageGRID storage. Customers can deploy and manage multiple StorageGRID instances between datacenters and in the cloud infrastructure.

A StorageGRID system is composed of globally distributed, redundant, heterogeneous nodes that can be integrated with both existing and next-generation client applications.

Figure showing input/output dialog or representing written content

IDC MarketScape recently named NetApp as a leader in the latest report, IDC MarketScape: Worldwide Object-Based Storage 2019 Vendor Assessment. With nearly 20 years of production deployments in the most demanding industries, StorageGRID is a recognized leader in unstructured data.

With StorageGRID, you can achieve the following:

  • Deploy multiple StorageGRID instances to access data from any location between data centers and the cloud through a single namespace that easily scales to hundreds of petabytes.

  • Provide flexibility to deploy and centrally manage across infrastructures.

  • Provide unmatched durability with fifteen-nines of durability leveraging layered Erasure Coding (EC).

  • Enable more hybrid multi-cloud capabilities with validated integrations into Amazon S3 Glacier and Azure Blob.

  • Meet regulatory obligations and facilitate compliance through tamper-proof data retention, without proprietary APIs or vendor lock-in.

For more information about how StorageGRID can help you solve your most complex unstructured data management problems, see the NetApp StorageGRID homepage.

About Splunk Enterprise

Splunk Enterprise is a platform for turning data into doing. Data generated by various sources such as log files, websites, devices, sensors, and applications are sent to and parsed by the Splunk Indexers, allowing you to derive rich insights from the data. It might identify data breaches, point out customer and product trends, find opportunities to optimize infrastructure, or create actionable insights across a wide variety of use cases.

About Splunk SmartStore

Splunk SmartStore expands on the benefits of the Splunk architecture while simplifying its ability to scale cost-effectively. The decoupling of compute and storage resources results in indexer nodes optimized for I/O with significantly reduced storage needs because they only store a subset of data as cache. You do not have to add extra compute or storage when only one of those resources is necessary, which allows you to realize significant cost savings. You can use cost-effective and easily scalable S3-based object storage, which further simplifies the environment, reduces costs, and allows you to maintain a more massive data set.

Splunk SmartStore delivers significant value to organizations, including the following:

  • Lowering storage cost by moving warm data to cost-optimized S3 object storage

  • Scaling seamlessly by decoupling storage and compute

  • Simplifying business continuity by leveraging resilient cloud-native storage