使用 audit-sum 工具彙總 StorageGRID 稽核訊息
建議變更
PDF
您可以使用 `audit-sum`工具來統計寫入、讀取、head 及刪除稽核訊息的數量，並查看每種操作類型的最小值、最大值和平均時間（或大小）。
-
您有"特定存取權限"。
-
您擁有
Passwords.txt檔案。 -
${post_edited_translations.segment}
主要管理節點上提供的 audit-sum 工具可摘要說明已記錄的寫入、讀取和刪除作業數量，以及這些作業所花費的時間。
|
此 `audit-sum`工具主要供技術支援在進行疑難排解作業時使用。處理 `audit-sum`查詢可能會消耗大量的 CPU 運算能力，這可能會影響 StorageGRID 作業。 |
此範例顯示 audit-sum 工具的典型輸出。此範例顯示協定作業所花費的時間。
message group count min(sec) max(sec) average(sec) ============= ===== ======== ======== ============ IDEL 274 SDEL 213371 0.004 20.934 0.352 SGET 201906 0.010 1740.290 1.132 SHEA 22716 0.005 2.349 0.272 SPUT 1771398 0.011 1770.563 0.487
此 audit-sum 工具提供稽核日誌中下列 S3 和 ILM 稽核訊息的計數與時間。
|
|
稽核代碼會隨著功能被取代而從產品和說明文件中移除。如果您遇到此處未列出的稽核代碼，請檢查此主題的先前版本以取得較舊的 StorageGRID 版本。例如， "${post_edited_translations.segment}"。 |
| 程式碼 | 說明 | 請參閱 |
|---|---|---|
IDEL |
${post_edited_translations.segment} |
|
SDEL |
S3 DELETE：記錄成功刪除物件或儲存貯體的交易。 |
|
SGET |
S3 GET：記錄成功擷取物件或列出儲存貯體中物件的交易。 |
|
SHEA |
S3 HEAD：記錄成功的交易，以檢查物件或儲存貯體是否存在。 |
|
SPUT |
S3 PUT：記錄建立新物件或儲存桶的成功交易。 |
此 `audit-sum`工具可以執行下列操作：
-
處理純文字或壓縮的稽核日誌。例如：
audit-sum audit.logaudit-sum 2019-08-12.txt.gz -
同時處理多個檔案。例如：
audit-sum audit.log 2019-08-12.txt.gz 2019-08-13.txt.gzaudit-sum /var/local/audit/export/* -
接受來自管道的輸入，這可讓您使用
grep命令或其他方式來篩選和預先處理輸入。例如：grep WGET audit.log | audit-sumgrep bucket1 audit.log | audit-sumneoplasms_of_the_central_nervous_system.json (1).zip (1.1 KB)
I am trying to load this JSON file into a pandas dataframe. The file contains a list of dictionaries, where each dictionary represents a neoplasm of the central nervous system.
I have tried the following code:
However, I get the following error:
I have checked the JSON file and it seems to be valid. What am I doing wrong?
The JSON file is attached.
The file you attached is not a valid JSON file. It contains multiple JSON objects, one after another, but they are not enclosed in a list (i.e., there are no and at the beginning and end of the file, and the objects are not separated by commas). This format is often called "JSON Lines" or "newline-delimited JSON" (ndjson).
Pandas has a built-in way to read this format directly using with the argument.
Here is how you can load it:
If you want to read it using the module first (though is preferred and faster), you would have to read it line by line:
+
grep SPUT audit.log | grep bucket1 | audit-sum neoplasms_of_the_central_nervous_system.json (1).zip (1.1 KB)
I am trying to load this JSON file into a pandas dataframe. The file contains a list of dictionaries, where each dictionary represents a neoplasm of the central nervous system.
I have tried the following code:
However, I get the following error:
I have checked the JSON file and it seems to be valid. What am I doing wrong?
The JSON file is attached.
The file you attached is not a valid JSON file. It contains multiple JSON objects, one after another, but they are not enclosed in a list (i.e., there are no and at the beginning and end of the file, and the objects are not separated by commas). This format is often called "JSON Lines" or "newline-delimited JSON" (ndjson).
Pandas has a built-in way to read this format directly using with the argument.
Here is how you can load it:
If you want to read it using the module first (though is preferred and faster), you would have to read it line by line:
|
|
此工具不接受壓縮檔案作為管道輸入。若要處理壓縮檔案，請提供其檔案名稱作為命令列引數，或使用
I am trying to load this JSON file into a pandas dataframe. The file contains a list of dictionaries, where each dictionary represents a neoplasm of the central nervous system. I have tried the following code: However, I get the following error: I have checked the JSON file and it seems to be valid. What am I doing wrong? The JSON file is attached. The file you attached is not a valid JSON file. It contains multiple JSON objects, one after another, but they are not enclosed in a list (i.e., there are no and at the beginning and end of the file, and the objects are not separated by commas). This format is often called "JSON Lines" or "newline-delimited JSON" (ndjson). Pandas has a built-in way to read this format directly using with the argument. Here is how you can load it: If you want to read it using the module first (though is preferred and faster), you would have to read it line by line: |
您可以使用命令列選項，將貯存庫上的作業與物件上的作業分開進行摘要，或是按貯存庫名稱、時間週期或目標類型來將訊息摘要分組。預設情況下，摘要會顯示最小、最大和平均作業時間，但您可以改用 size (-s) 選項來查看物件大小。
使用 help (-h) 選項來查看可用的選項。例如：
$ audit-sum -h
-
${post_edited_translations.segment}
-
輸入以下命令：
ssh admin@primary_Admin_Node_IP -
請輸入 `Passwords.txt`檔案中列出的密碼。
-
輸入以下命令以切換至根目錄：
su - -
請輸入 `Passwords.txt`檔案中列出的密碼。
當您以根目錄身分登入時，提示符號會從
$`變為 `#。
-
-
${post_edited_translations.segment}
-
輸入下列命令，其中
/var/local/audit/export/audit.log代表您要分析之檔案的名稱與位置：$ audit-sum /var/local/audit/export/audit.log此範例顯示
audit-sum工具的典型輸出。此範例顯示協定作業所花費的時間。message group count min(sec) max(sec) average(sec) ============= ===== ======== ======== ============ IDEL 274 SDEL 213371 0.004 20.934 0.352 SGET 201906 0.010 1740.290 1.132 SHEA 22716 0.005 2.349 0.272 SPUT 1771398 0.011 1770.563 0.487
${post_edited_translations.segment}
-
若要顯示速度最慢的 10 個擷取操作，請使用 grep 命令僅選擇 SGET 訊息，並新增 long 輸出選項(`-l`以包含物件路徑：
grep SGET audit.log | audit-sum -l結果包括類型（物件或儲存桶）和路徑，可讓您在稽核日誌中搜尋與這些特定物件相關的其他訊息。
Total: 201906 operations Slowest: 1740.290 sec Average: 1.132 sec Fastest: 0.010 sec Slowest operations: time(usec) source ip type size(B) path ========== =============== ============ ============ ==== 1740289662 10.96.101.125 object 5663711385 backup/r9O1OaQ8JB-1566861764-4519.iso 1624414429 10.96.101.125 object 5375001556 backup/r9O1OaQ8JB-1566861764-6618.iso 1533143793 10.96.101.125 object 5183661466 backup/r9O1OaQ8JB-1566861764-4518.iso 70839 10.96.101.125 object 28338 bucket3/dat.1566861764-6619 68487 10.96.101.125 object 27890 bucket3/dat.1566861764-6615 67798 10.96.101.125 object 27671 bucket5/dat.1566861764-6617 67027 10.96.101.125 object 27230 bucket5/dat.1566861764-4517 60922 10.96.101.125 object 26118 bucket3/dat.1566861764-4520 35588 10.96.101.125 object 11311 bucket3/dat.1566861764-6616 23897 10.96.101.125 object 10692 bucket3/dat.1566861764-4516+ 從此範例輸出中，您可以看到三個最慢的 S3 GET 要求是針對大小約為 5 GB 的物件，這比其他物件大得多。較大的大小是導致最差情況下擷取時間緩慢的原因。
-
-
如果您想確定擷取至網格及從網格擷取的物件大小，請使用 size 選項(
-s）：audit-sum -s audit.logmessage group count min(MB) max(MB) average(MB) ============= ===== ======== ======== ============ IDEL 274 0.004 5000.000 1654.502 SDEL 213371 0.000 10.504 1.695 SGET 201906 0.000 5000.000 14.920 SHEA 22716 0.001 10.504 2.967 SPUT 1771398 0.000 5000.000 2.495
在此範例中、SPUT 的平均物件大小小於 2.5 MB、但 SGET 的平均大小則大得多。SPUT 訊息的數量遠高於 SGET 訊息的數量、這表示大多數物件從未被擷取。
-
如果您想確定昨天的檢索速度是否緩慢：
-
在相應的稽核日誌上發出命令，並使用 group-by-time 選項(
-gt），後接時間段（例如，15M、1H、10S）：grep SGET audit.log | audit-sum -gt 1Hmessage group count min(sec) max(sec) average(sec) ============= ===== ======== ======== ============ 2019-09-05T00 7591 0.010 1481.867 1.254 2019-09-05T01 4173 0.011 1740.290 1.115 2019-09-05T02 20142 0.011 1274.961 1.562 2019-09-05T03 57591 0.010 1383.867 1.254 2019-09-05T04 124171 0.013 1740.290 1.405 2019-09-05T05 420182 0.021 1274.511 1.562 2019-09-05T06 1220371 0.015 6274.961 5.562 2019-09-05T07 527142 0.011 1974.228 2.002 2019-09-05T08 384173 0.012 1740.290 1.105 2019-09-05T09 27591 0.010 1481.867 1.354
結果顯示，S3 GET 請求流量在 06:00 至 07:00 之間出現峰值。在此期間，最大請求時間和平均請求時間均顯著升高，且並非隨著請求數量的增加而逐漸增加。這些計量表明，容量可能已超出負荷，可能是網路方面，或是網格處理請求的能力方面。
-
若要確定昨天每小時擷取的物件大小，請在命令中加入大小選項 (
-s）：grep SGET audit.log | audit-sum -gt 1H -smessage group count min(B) max(B) average(B) ============= ===== ======== ======== ============ 2019-09-05T00 7591 0.040 1481.867 1.976 2019-09-05T01 4173 0.043 1740.290 2.062 2019-09-05T02 20142 0.083 1274.961 2.303 2019-09-05T03 57591 0.912 1383.867 1.182 2019-09-05T04 124171 0.730 1740.290 1.528 2019-09-05T05 420182 0.875 4274.511 2.398 2019-09-05T06 1220371 0.691 5663711385.961 51.328 2019-09-05T07 527142 0.130 1974.228 2.147 2019-09-05T08 384173 0.625 1740.290 1.878 2019-09-05T09 27591 0.689 1481.867 1.354
${post_edited_translations.segment}
-
若要查看更多詳細資訊，請使用 "audit-explain 工具" 審查該小時內的所有 SGET 作業：
grep 2019-09-05T06 audit.log | grep SGET | audit-explain | less
如果 grep 命令的輸出預期會有多行，請加入
less命令，以一次一頁（一個畫面）的方式顯示稽核日誌檔案的內容。 -
-
${post_edited_translations.segment}
-
請先使用
-go選項，此選項會分別將物件和儲存貯體作業的訊息進行分組：grep SPUT sample.log | audit-sum -gomessage group count min(sec) max(sec) average(sec) ============= ===== ======== ======== ============ SPUT.bucket 1 0.125 0.125 0.125 SPUT.object 12 0.025 1.019 0.236
${post_edited_translations.segment}
-
若要確定哪些儲存桶的 SPUT 操作速度最慢，請使用 `-gb`選項，該選項可依儲存桶對訊息進行分組：
grep SPUT audit.log | audit-sum -gbmessage group count min(sec) max(sec) average(sec) ============= ===== ======== ======== ============ SPUT.cho-non-versioning 71943 0.046 1770.563 1.571 SPUT.cho-versioning 54277 0.047 1736.633 1.415 SPUT.cho-west-region 80615 0.040 55.557 1.329 SPUT.ldt002 1564563 0.011 51.569 0.361
-
若要確定哪些貯存庫具有最大的 SPUT 物件大小，請同時使用
-gb和-s選項：grep SPUT audit.log | audit-sum -gb -s
message group count min(B) max(B) average(B) ============= ===== ======== ======== ============ SPUT.cho-non-versioning 71943 2.097 5000.000 21.672 SPUT.cho-versioning 54277 2.097 5000.000 21.120 SPUT.cho-west-region 80615 2.097 800.000 14.433 SPUT.ldt002 1564563 0.000 999.972 0.352
-