Skip to main content

Troubleshoot S3 PUT Object size too large alert

PDFs

The S3 PUT Object size too large alert is triggered if a tenant attempts a non-multipart PutObject operation that exceeds the S3 size limit of 5 GiB.

Before you begin

Determine which tenants use objects that are larger than 5 GiB, so you can notify them.

Steps

  1. Go to CONFIGURATION > Monitoring > Audit and syslog server.

  2. If Client Writes are Normal, access the audit log:

    1. Enter ssh admin@primary_Admin_Node_IP

    2. Enter the password listed in the Passwords.txt file.

    3. Enter the following command to switch to root: su -

    4. Enter the password listed in the Passwords.txt file.

      When you are logged in as root, the prompt changes from $ to #.

    5. Change to the directory where the audit logs are located.

      The audit log directory and applicable nodes depend on your audit destination settings.

      Option Destination

      Local nodes (default)

      /var/local/log/localaudit.log

      Admin Nodes/local nodes

      • Admin Nodes (Primary and Non-Primary): /var/local/audit/export/audit.log

      • All nodes: The /var/local/log/localaudit.log file is typically empty or missing in this mode.

      External syslog server

      /var/local/log/localaudit.log

      Depending on your audit destination settings, enter: cd /var/local/log or /var/local/audit/export/

      To learn more, refer to Select audit information destinations.

    6. Identify which tenants are using objects larger than 5 GiB.

      1. Enter zgrep SPUT * | egrep "CSIZ\(UI64\):([5-9]|[1-9][0-9]+)[0-9]{9}"

      2. For each audit message in the results, look at S3AI field to determine the tenant account ID. Use the other fields in the message to determine which IP address was used by the client, the bucket, and the object:

        Code Description

        SAIP

        Source IP

        S3AI

        Tenant ID

        S3BK

        Bucket

        S3KY

        Object

        CSIZ

        Size (bytes)

        Example audit log results

        audit.log:2023-01-05T18:47:05.525999 [AUDT:[RSLT(FC32):SUCS][CNID(UI64):1672943621106262][TIME(UI64):804317333][SAIP(IPAD):"10.96.99.127"][S3AI(CSTR):"93390849266154004343"][SACC(CSTR):"bhavna"][S3AK(CSTR):"06OX85M40Q90Y280B7YT"][SUSR(CSTR):"urn:sgws:identity::93390849266154004343:root"][SBAI(CSTR):"93390849266154004343"][SBAC(CSTR):"bhavna"][S3BK(CSTR):"test"][S3KY(CSTR):"large-object"][CBID(UI64):0x077EA25F3B36C69A][UUID(CSTR):"A80219A2-CD1E-466F-9094-B9C0FDE2FFA3"][CSIZ(UI64):6040000000][MTME(UI64):1672943621338958][AVER(UI32):10][ATIM(UI64):1672944425525999][ATYP(FC32):SPUT][ANID(UI32):12220829][AMID(FC32):S3RQ][ATID(UI64):4333283179807659119]]

  3. If Client Writes aren't Normal, use the tenant ID from the alert to identify the tenant:

    1. Go to SUPPORT > Tools > Logs. Collect application logs for the Storage Node in the alert. Specify 15 minutes before and after the alert.

    2. Extract the file and go to bycast.log:

      /GID<grid_id>_<time_stamp>/<site_node>/<time_stamp>/grid/bycast.log

    3. Search the log for method=PUT and identify the client in the clientIP field.

      Example bycast.log

      Jan  5 18:33:41 BHAVNAJ-DC1-S1-2-65 ADE: |12220829 1870864574 S3RQ %CEA 2023-01-05T18:33:41.208790| NOTICE   1404 af23cb66b7e3efa5 S3RQ: EVENT_PROCESS_CREATE - connection=1672943621106262 method=PUT name=</test/4MiB-0> auth=<V4> clientIP=<10.96.99.127>

  4. Inform tenants that the maximum PutObject size is 5 GiB and to use multipart uploads for objects greater than 5 GiB.

  5. Ignore the alert for one week if the application has been changed.