This NetApp StorageGRID and Apache Kafka solution is a distributed system designed for streams, and both programs are horizontally scalable and fault tolerant. They provide a commit log and allow distributed data streaming and stream processing. NetApp storage arrays decouple compute and data storages resources so that they can be scaled independently. The following figure depicts the NetApp StorageGRID and Confluent Kafka solution.

Error: Missing Graphic Image

Solution architecture details

This section covers the hardware and software used for Confluent certification. This information is applicable to Kafka deployment with NetApp storage. The following table covers the tested solution architecture and base components.

Solution components Details

Confluent Kafka version 6.2

  • Three zookeepers

  • Five broker servers

  • Five tools’ servers

  • One Grafana

  • One control center

Linux (ubuntu 18.04)

All servers

NetApp StorageGRID for warm buckets

  • 4 x Storage Grid

  • 1 x SG1000 (load balancer)

  • 4 x SGD6024

  • 4 x 24 x 800 SSDs

  • S3 protocol

  • 100GbE

15 Fujitsu PRIMERGY RX2540 servers

Each equipped with:
* 2 CPUs, 16 physical cores total
* Intel Xeon
* 256GB physical memory
* 100GbE dual port