Skip to main content

Troubleshoot load balancer caching

Contributors netapp-lhalbert
PDFs

Learn about possible issues with load balancer caching and how to troubleshoot them.

Determine if a request was a cache hit

  • The X-Cache header is set in the response to requests handled by the cache service. Possible codes:

    • HIT: The object was served from cache

    • PARTIAL-HIT: The bucket/key had a record in the cache but not all of the requested range was able to be served from cache

    • STALE: The bucket/key had a record in cache but the object was updated since the last time it was served from cache.

    • MISS: The object was not in cache

  • The nginx-gw/endpoint-access.log.gz record for the request includes "unix:/run/cache-svc/cache-svc.sock" for requests handled by the cache.

  • The cache-svc/cache-svc.log reports a message such as, "Request 320390: completed successfully (cache hit)" or "Request 320375: completed successfully (cache miss)." Find the requested path by searching for other records with the same "Request <number>" string.

Low cache hit rate

  • Low cache hit rates might be expected when a new workload is added or the working set being accessed by a workload changes. In these situations, the hit rate is expected to increase over time.

  • If multiple workloads are using caching, consider adding traffic classification policies to isolate parts of the workloads being serviced by the cache. Cache hit rate metrics are available per traffic classification policy. If some workloads are not experiencing good cache hit rates consider moving those workloads to other endpoints which do not have caching enabled.

  • Evaluate the cache eviction rate. If the cache is too small to fit the working set it will experience high eviction rates and might contribute to lower hit rates.

  • Options for improving hit rates of certain workloads might be available under FPVR.

Low performance

  • Evaluate the cache hit rate. Low cache hit rates can lead to low overall performance.

  • Evaluate the cache eviction rate. During eviction some storage resources are used for removing existing objects from disk. If the eviction process does not keep up with the access of new objects the system may hit the hard watermark thresholds and begin bypassing the cache.

  • Check for network limits using the “Network interfaces receive usage” and “Network interfaces transmit usage” diagnostics.