== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:16:06 (1750590966) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) oleg104-server: oleg104-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1226 (time 1750591023.77 total 10.00 last 122.55) - open/close 2472 (time 1750591033.77 total 20.00 last 124.59) - open/close 3775 (time 1750591043.78 total 30.01 last 130.29) - open/close 5073 (time 1750591053.78 total 40.01 last 129.76) - open/close 6294 (time 1750591063.78 total 50.01 last 122.02) - open/close 7360 (time 1750591073.79 total 60.02 last 106.51) - open/close 8514 (time 1750591083.80 total 70.03 last 115.33) - open/close 9896 (time 1750591093.80 total 80.03 last 138.12) - open/close 11117 (time 1750591103.81 total 90.04 last 122.05) - open/close 12379 (time 1750591113.81 total 100.04 last 126.12) - open/close 13538 (time 1750591123.82 total 110.05 last 115.81) - open/close 14836 (time 1750591133.83 total 120.06 last 129.69) - open/close 16163 (time 1750591143.83 total 130.06 last 132.70) - open/close 17347 (time 1750591153.83 total 140.06 last 118.37) - open/close 18540 (time 1750591163.84 total 150.07 last 119.23) - open/close 19682 (time 1750591173.84 total 160.07 last 114.18) - open/close 20000 (time 1750591176.63 total 162.86 last 114.09) - open/close 21151 (time 1750591186.64 total 172.87 last 115.01) - open/close 22248 (time 1750591196.64 total 182.87 last 109.66) - open/close 23372 (time 1750591206.65 total 192.88 last 112.32) - open/close 24607 (time 1750591216.65 total 202.88 last 123.43) - open/close 25971 (time 1750591226.65 total 212.88 last 136.39) - open/close 27290 (time 1750591236.66 total 222.89 last 131.84) - open/close 28402 (time 1750591246.66 total 232.89 last 111.17) - open/close 29509 (time 1750591256.67 total 242.90 last 110.61) - open/close 30000 (time 1750591260.70 total 246.93 last 121.95) - open/close 31346 (time 1750591270.70 total 256.93 last 134.50) - open/close 32534 (time 1750591280.71 total 266.94 last 118.75) - open/close 33979 (time 1750591290.71 total 276.94 last 144.48) - open/close 35199 (time 1750591300.71 total 286.94 last 121.94) - open/close 36372 (time 1750591310.72 total 296.95 last 117.23) - open/close 37598 (time 1750591320.72 total 306.95 last 122.56) - open/close 38939 (time 1750591330.72 total 316.95 last 134.08) - open/close 40000 (time 1750591340.05 total 326.28 last 113.79) - open/close 41121 (time 1750591350.06 total 336.29 last 111.99) - open/close 42382 (time 1750591360.06 total 346.29 last 126.07) - open/close 43651 (time 1750591370.06 total 356.29 last 126.86) - open/close 44937 (time 1750591380.08 total 366.31 last 128.45) - open/close 46066 (time 1750591390.08 total 376.31 last 112.83) - open/close 47477 (time 1750591400.08 total 386.31 last 141.08) - open/close 48838 (time 1750591410.09 total 396.32 last 136.03) - open/close 50000 (time 1750591418.90 total 405.13 last 131.94) - open/close 51231 (time 1750591428.90 total 415.13 last 123.07) - open/close 52503 (time 1750591438.90 total 425.13 last 127.16) - open/close 53910 (time 1750591448.91 total 435.14 last 140.64) - open/close 55181 (time 1750591458.91 total 445.14 last 127.09) - open/close 56456 (time 1750591468.91 total 455.14 last 127.41) - open/close 57748 (time 1750591478.92 total 465.15 last 129.16) - open/close 59162 (time 1750591488.92 total 475.15 last 141.35) - open/close 60000 (time 1750591494.96 total 481.19 last 138.84) - open/close 61324 (time 1750591504.97 total 491.20 last 132.27) - open/close 62626 (time 1750591514.97 total 501.20 last 130.16) - open/close 63937 (time 1750591524.98 total 511.21 last 131.00) total: 64768 open/close in 517.14 seconds: 125.24 ops/second - /unlink 2806 (time 1750591542.77 total 10.00 last 280.58) - /unlink 5769 (time 1750591552.77 total 20.00 last 296.22) - /unlink 7905 (time 1750591562.77 total 30.01 last 213.53) - /unlink 10000 (time 1750591572.30 total 39.54 last 219.84) - /unlink 12475 (time 1750591582.30 total 49.54 last 247.46) - /unlink 15109 (time 1750591592.31 total 59.54 last 263.33) - /unlink 17359 (time 1750591602.31 total 69.54 last 224.95) - /unlink 19765 (time 1750591612.31 total 79.54 last 240.56) - /unlink 22367 (time 1750591622.31 total 89.55 last 260.16) - /unlink 24836 (time 1750591632.31 total 99.55 last 246.84) - /unlink 27395 (time 1750591642.32 total 109.55 last 255.80) - /unlink 29694 (time 1750591652.32 total 119.56 last 229.82) - /unlink 32173 (time 1750591662.32 total 129.56 last 247.83) - /unlink 34601 (time 1750591672.33 total 139.56 last 242.75) - /unlink 37326 (time 1750591682.33 total 149.56 last 272.49) - /unlink 39799 (time 1750591692.33 total 159.56 last 247.25) - /unlink 42736 (time 1750591702.33 total 169.56 last 293.69) - /unlink 45112 (time 1750591712.33 total 179.56 last 237.60) - /unlink 47429 (time 1750591722.33 total 189.57 last 231.60) - /unlink 50000 (time 1750591731.34 total 198.57 last 285.61) - /unlink 52970 (time 1750591741.34 total 208.57 last 296.90) - /unlink 55839 (time 1750591751.34 total 218.58 last 286.81) - /unlink 58222 (time 1750591761.34 total 228.58 last 238.28) - /unlink 60000 (time 1750591768.35 total 235.59 last 253.60) - /unlink 62671 (time 1750591778.36 total 245.59 last 267.06) total: 64768 /unlink in 252.54 seconds: 256.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1418 (time 1750591814.49 total 10.00 last 141.75) - open/close 2841 (time 1750591824.49 total 20.00 last 142.29) - open/close 4311 (time 1750591834.49 total 30.01 last 146.98) - open/close 5845 (time 1750591844.50 total 40.01 last 153.32) - open/close 7457 (time 1750591854.50 total 50.02 last 161.12) - open/close 8975 (time 1750591864.50 total 60.02 last 151.74) - open/close 10000 (time 1750591872.64 total 68.16 last 125.94) - open/close 11473 (time 1750591882.64 total 78.16 last 147.29) - open/close 12622 (time 1750591892.66 total 88.18 last 114.71) - open/close 13684 (time 1750591902.67 total 98.19 last 106.11) - open/close 15019 (time 1750591912.67 total 108.19 last 133.49) - open/close 16223 (time 1750591922.67 total 118.19 last 120.40) - open/close 17633 (time 1750591932.67 total 128.19 last 140.94) - open/close 18962 (time 1750591942.68 total 138.19 last 132.85) - open/close 20000 (time 1750591949.99 total 145.51 last 141.96) - open/close 21377 (time 1750591960.00 total 155.51 last 137.63) - open/close 22623 (time 1750591970.00 total 165.51 last 124.59) - open/close 24083 (time 1750591980.01 total 175.52 last 145.86) - open/close 25433 (time 1750591990.01 total 185.52 last 134.96) - open/close 26788 (time 1750592000.02 total 195.53 last 135.40) - open/close 27982 (time 1750592010.02 total 205.53 last 119.37) - open/close 29594 (time 1750592020.02 total 215.54 last 161.12) - open/close 30000 (time 1750592022.70 total 218.22 last 151.44) - open/close 31183 (time 1750592032.71 total 228.22 last 118.27) - open/close 32564 (time 1750592042.71 total 238.22 last 138.09) - open/close 33703 (time 1750592052.71 total 248.23 last 113.86) - open/close 34807 (time 1750592062.72 total 258.24 last 110.30) - open/close 36210 (time 1750592072.72 total 268.24 last 140.25) - open/close 37760 (time 1750592082.73 total 278.24 last 154.96) - open/close 39171 (time 1750592092.73 total 288.24 last 141.08) - open/close 40000 (time 1750592098.34 total 293.86 last 147.62) - open/close 41317 (time 1750592108.36 total 303.87 last 131.52) - open/close 42538 (time 1750592118.37 total 313.88 last 122.01) - open/close 43664 (time 1750592128.37 total 323.89 last 112.54) - open/close 44942 (time 1750592138.38 total 333.89 last 127.73) - open/close 46225 (time 1750592148.39 total 343.91 last 128.11) - open/close 47457 (time 1750592158.39 total 353.91 last 123.18) - open/close 48657 (time 1750592168.40 total 363.91 last 119.96) - open/close 49892 (time 1750592178.40 total 373.91 last 123.48) - open/close 51127 (time 1750592188.40 total 383.92 last 123.47) - open/close 52380 (time 1750592198.41 total 393.92 last 125.23) - open/close 53688 (time 1750592208.41 total 403.92 last 130.79) - open/close 54768 (time 1750592218.42 total 413.93 last 107.88) - open/close 56110 (time 1750592228.42 total 423.94 last 134.14) - open/close 57288 (time 1750592238.42 total 433.94 last 117.80) - open/close 58486 (time 1750592248.43 total 443.94 last 119.73) - open/close 59726 (time 1750592258.43 total 453.94 last 123.99) - open/close 60000 (time 1750592260.66 total 456.17 last 123.04) - open/close 61181 (time 1750592270.66 total 466.18 last 118.06) - open/close 62258 (time 1750592280.67 total 476.18 last 107.62) - open/close 63572 (time 1750592290.67 total 486.19 last 131.36) total: 64768 open/close in 495.84 seconds: 130.62 ops/second - /unlink 2067 (time 1750592311.82 total 10.00 last 206.64) - /unlink 4555 (time 1750592321.82 total 20.01 last 248.75) - /unlink 6807 (time 1750592331.82 total 30.01 last 225.08) - /unlink 8691 (time 1750592341.83 total 40.01 last 188.38) - /unlink 10000 (time 1750592349.08 total 47.27 last 180.45) - /unlink 12069 (time 1750592359.08 total 57.27 last 206.88) - /unlink 13977 (time 1750592369.08 total 67.27 last 190.77) - /unlink 16215 (time 1750592379.09 total 77.27 last 223.72) - /unlink 18389 (time 1750592389.09 total 87.28 last 217.29) - /unlink 20000 (time 1750592397.02 total 95.20 last 203.31) - /unlink 21937 (time 1750592407.02 total 105.21 last 193.57) - /unlink 24037 (time 1750592417.02 total 115.21 last 209.95) - /unlink 26134 (time 1750592427.03 total 125.21 last 209.63) - /unlink 28389 (time 1750592437.03 total 135.21 last 225.47) - /unlink 30000 (time 1750592445.32 total 143.51 last 194.27) - /unlink 32264 (time 1750592455.32 total 153.51 last 226.35) - /unlink 34599 (time 1750592465.33 total 163.51 last 233.46) - /unlink 36855 (time 1750592475.33 total 173.51 last 225.55) - /unlink 39215 (time 1750592485.33 total 183.51 last 235.98) - /unlink 40000 (time 1750592488.67 total 186.86 last 234.77) - /unlink 42282 (time 1750592498.67 total 196.86 last 228.15) - /unlink 45009 (time 1750592508.68 total 206.87 last 272.56) - /unlink 47525 (time 1750592518.68 total 216.87 last 251.58) - /unlink 50000 (time 1750592527.09 total 225.27 last 294.35) - /unlink 52334 (time 1750592537.09 total 235.27 last 233.39) - /unlink 54766 (time 1750592547.10 total 245.28 last 243.04) - /unlink 56607 (time 1750592557.10 total 255.28 last 184.05) - /unlink 58508 (time 1750592567.10 total 265.29 last 190.04) - /unlink 60000 (time 1750592574.83 total 273.02 last 192.92) - /unlink 61761 (time 1750592584.84 total 283.02 last 176.03) - /unlink 63692 (time 1750592594.84 total 293.03 last 193.03) total: 64768 /unlink in 298.25 seconds: 217.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1162 (time 1750592629.50 total 10.00 last 116.20) - open/close 2302 (time 1750592639.50 total 20.00 last 113.96) - open/close 3318 (time 1750592649.50 total 30.01 last 101.56) - open/close 4328 (time 1750592659.51 total 40.01 last 100.98) - open/close 5312 (time 1750592669.51 total 50.02 last 98.32) - open/close 6386 (time 1750592679.52 total 60.02 last 107.32) - open/close 7532 (time 1750592689.52 total 70.03 last 114.59) - open/close 8660 (time 1750592699.52 total 80.03 last 112.78) - open/close 9835 (time 1750592709.53 total 90.03 last 117.42) - open/close 10922 (time 1750592719.53 total 100.04 last 108.67) - open/close 12041 (time 1750592729.54 total 110.04 last 111.86) - open/close 13119 (time 1750592739.54 total 120.05 last 107.74) - open/close 14244 (time 1750592749.55 total 130.05 last 112.46) - open/close 15605 (time 1750592759.55 total 140.05 last 136.09) - open/close 16833 (time 1750592769.55 total 150.05 last 122.74) - open/close 18159 (time 1750592779.56 total 160.06 last 132.53) - open/close 19399 (time 1750592789.56 total 170.06 last 123.99) - open/close 20000 (time 1750592794.11 total 174.61 last 132.08) - open/close 21288 (time 1750592804.11 total 184.61 last 128.79) - open/close 22672 (time 1750592814.11 total 194.61 last 138.39) - open/close 24161 (time 1750592824.11 total 204.61 last 148.87) - open/close 25553 (time 1750592834.12 total 214.62 last 139.13) - open/close 26759 (time 1750592844.13 total 224.63 last 120.42) - open/close 27954 (time 1750592854.14 total 234.64 last 119.41) - open/close 29311 (time 1750592864.15 total 244.65 last 135.61) - open/close 30000 (time 1750592870.42 total 250.92 last 109.83) - open/close 31023 (time 1750592880.43 total 260.93 last 102.24) - open/close 32038 (time 1750592890.43 total 270.93 last 101.46) - open/close 33088 (time 1750592900.43 total 280.94 last 104.96) - open/close 34250 (time 1750592910.43 total 290.94 last 116.18) - open/close 35378 (time 1750592920.44 total 300.94 last 112.73) - open/close 36472 (time 1750592930.44 total 310.95 last 109.38) - open/close 37561 (time 1750592940.45 total 320.95 last 108.85) - open/close 38711 (time 1750592950.45 total 330.96 last 114.93) - open/close 39904 (time 1750592960.46 total 340.96 last 119.23) - open/close 41117 (time 1750592970.46 total 350.97 last 121.26) - open/close 42236 (time 1750592980.47 total 360.97 last 111.85) - open/close 43429 (time 1750592990.47 total 370.97 last 119.27) - open/close 44529 (time 1750593000.47 total 380.98 last 109.97) - open/close 45627 (time 1750593010.47 total 390.98 last 109.79) - open/close 46719 (time 1750593020.48 total 400.99 last 109.09) - open/close 47927 (time 1750593030.49 total 410.99 last 120.74) - open/close 49168 (time 1750593040.50 total 421.00 last 124.00) - open/close 50000 (time 1750593047.72 total 428.23 last 115.13) - open/close 51128 (time 1750593057.73 total 438.24 last 112.68) - open/close 52409 (time 1750593067.74 total 448.24 last 128.04) - open/close 53585 (time 1750593077.74 total 458.25 last 117.52) - open/close 54859 (time 1750593087.75 total 468.25 last 127.34) - open/close 56005 (time 1750593097.76 total 478.26 last 114.50) - open/close 57025 (time 1750593107.76 total 488.26 last 102.00) - open/close 58094 (time 1750593117.76 total 498.26 last 106.88) - open/close 59119 (time 1750593127.77 total 508.27 last 102.43) - open/close 60000 (time 1750593135.96 total 516.46 last 107.55) - open/close 61074 (time 1750593145.96 total 526.47 last 107.35) - open/close 62061 (time 1750593155.96 total 536.47 last 98.69) - open/close 62999 (time 1750593165.97 total 546.48 last 93.72) - open/close 64041 (time 1750593175.97 total 556.48 last 104.17) total: 64768 open/close in 563.99 seconds: 114.84 ops/second - /unlink 1707 (time 1750593196.06 total 10.01 last 170.61) - /unlink 3523 (time 1750593206.07 total 20.01 last 181.52) - /unlink 5427 (time 1750593216.07 total 30.02 last 190.28) - /unlink 7261 (time 1750593226.07 total 40.02 last 183.38) - /unlink 8926 (time 1750593236.08 total 50.02 last 166.44) - /unlink 10000 (time 1750593241.29 total 55.23 last 206.07) - /unlink 12166 (time 1750593251.29 total 65.23 last 216.59) - /unlink 14422 (time 1750593261.29 total 75.23 last 225.58) - /unlink 17056 (time 1750593271.29 total 85.24 last 263.32) - /unlink 19699 (time 1750593281.30 total 95.24 last 264.26) - /unlink 22226 (time 1750593291.30 total 105.24 last 252.68) - /unlink 24732 (time 1750593301.30 total 115.24 last 250.59) - /unlink 26891 (time 1750593311.30 total 125.24 last 215.83) - /unlink 29105 (time 1750593321.30 total 135.25 last 221.35) - /unlink 30000 (time 1750593324.79 total 138.74 last 256.35) - /unlink 32414 (time 1750593334.79 total 148.74 last 241.39) - /unlink 34817 (time 1750593344.79 total 158.74 last 240.29) - /unlink 36867 (time 1750593354.79 total 168.74 last 204.98) - /unlink 38852 (time 1750593364.80 total 178.74 last 198.46) - /unlink 40000 (time 1750593371.10 total 185.05 last 181.99) - /unlink 41796 (time 1750593381.11 total 195.05 last 179.57) - /unlink 43895 (time 1750593391.11 total 205.05 last 209.87) - /unlink 45720 (time 1750593401.11 total 215.05 last 182.49) - /unlink 47717 (time 1750593411.11 total 225.05 last 199.67) - /unlink 49705 (time 1750593421.11 total 235.05 last 198.77) - /unlink 51415 (time 1750593431.11 total 245.05 last 171.00) - /unlink 53058 (time 1750593441.11 total 255.05 last 164.30) - /unlink 54862 (time 1750593451.11 total 265.06 last 180.34) - /unlink 56720 (time 1750593461.12 total 275.06 last 185.75) - /unlink 58496 (time 1750593471.12 total 285.06 last 177.55) - /unlink 60000 (time 1750593478.73 total 292.68 last 197.59) - /unlink 62046 (time 1750593488.73 total 302.68 last 204.58) - /unlink 63854 (time 1750593498.74 total 312.69 last 180.64) total: 64768 /unlink in 318.04 seconds: 203.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 997 (time 1750593530.79 total 10.01 last 99.62) - open/close 2071 (time 1750593540.79 total 20.01 last 107.36) - open/close 3065 (time 1750593550.80 total 30.02 last 99.31) - open/close 4108 (time 1750593560.81 total 40.03 last 104.25) - open/close 5175 (time 1750593570.81 total 50.03 last 106.68) - open/close 6348 (time 1750593580.81 total 60.03 last 117.28) - open/close 7699 (time 1750593590.81 total 70.03 last 135.03) - open/close 8730 (time 1750593600.82 total 80.04 last 103.02) - open/close 9817 (time 1750593610.82 total 90.04 last 108.67) - open/close 11052 (time 1750593620.83 total 100.05 last 123.42) - open/close 12164 (time 1750593630.83 total 110.05 last 111.16) - open/close 13333 (time 1750593640.84 total 120.05 last 116.89) - open/close 14441 (time 1750593650.84 total 130.06 last 110.72) - open/close 15509 (time 1750593660.84 total 140.06 last 106.79) - open/close 16671 (time 1750593670.85 total 150.07 last 116.11) - open/close 17825 (time 1750593680.85 total 160.07 last 115.37) - open/close 18937 (time 1750593690.86 total 170.08 last 111.16) - open/close 20000 (time 1750593700.35 total 179.57 last 111.97) - open/close 21070 (time 1750593710.36 total 189.58 last 106.92) - open/close 22137 (time 1750593720.36 total 199.58 last 106.68) - open/close 23141 (time 1750593730.36 total 209.58 last 100.38) - open/close 24166 (time 1750593740.37 total 219.59 last 102.43) - open/close 25156 (time 1750593750.37 total 229.59 last 98.97) - open/close 26193 (time 1750593760.38 total 239.60 last 103.65) - open/close 27219 (time 1750593770.38 total 249.60 last 102.55) - open/close 28274 (time 1750593780.38 total 259.60 last 105.48) - open/close 29231 (time 1750593790.39 total 269.61 last 95.66) - open/close 30000 (time 1750593797.54 total 276.75 last 107.60) - open/close 31027 (time 1750593807.55 total 286.77 last 102.56) - open/close 32091 (time 1750593817.56 total 296.78 last 106.31) - open/close 33163 (time 1750593827.57 total 306.79 last 107.08) - open/close 34277 (time 1750593837.57 total 316.79 last 111.34) - open/close 35443 (time 1750593847.58 total 326.80 last 116.58) - open/close 36518 (time 1750593857.58 total 336.80 last 107.47) - open/close 37564 (time 1750593867.59 total 346.81 last 104.49) - open/close 38629 (time 1750593877.60 total 356.82 last 106.39) - open/close 39620 (time 1750593887.61 total 366.83 last 99.01) - open/close 40000 (time 1750593891.15 total 370.36 last 107.44) - open/close 41076 (time 1750593901.15 total 380.37 last 107.52) - open/close 41987 (time 1750593911.16 total 390.38 last 91.07) - open/close 42970 (time 1750593921.16 total 400.38 last 98.26) - open/close 44011 (time 1750593931.16 total 410.38 last 104.08) - open/close 45047 (time 1750593941.17 total 420.39 last 103.54) - open/close 46177 (time 1750593951.17 total 430.39 last 112.94) - open/close 47476 (time 1750593961.18 total 440.40 last 129.85) - open/close 48683 (time 1750593971.18 total 450.40 last 120.65) - open/close 50000 (time 1750593980.99 total 460.21 last 134.24) - open/close 51005 (time 1750593990.99 total 470.21 last 100.47) - open/close 51913 (time 1750594001.01 total 480.23 last 90.69) - open/close 52868 (time 1750594011.01 total 490.23 last 95.48) - open/close 53900 (time 1750594021.01 total 500.23 last 103.16) - open/close 55013 (time 1750594031.02 total 510.23 last 111.26) - open/close 56088 (time 1750594041.02 total 520.24 last 107.41) - open/close 57112 (time 1750594051.03 total 530.25 last 102.32) - open/close 58194 (time 1750594061.04 total 540.26 last 108.13) - open/close 59385 (time 1750594071.04 total 550.26 last 119.04) - open/close 60000 (time 1750594076.07 total 555.29 last 122.38) - open/close 61168 (time 1750594086.07 total 565.29 last 116.75) - open/close 62362 (time 1750594096.08 total 575.30 last 119.36) - open/close 63523 (time 1750594106.08 total 585.30 last 116.07) total: 64768 open/close in 594.94 seconds: 108.87 ops/second - /unlink 2043 (time 1750594127.84 total 10.00 last 204.21) - /unlink 4287 (time 1750594137.84 total 20.00 last 224.40) - /unlink 6232 (time 1750594147.84 total 30.01 last 194.46) - /unlink 8239 (time 1750594157.84 total 40.01 last 200.66) - /unlink 10000 (time 1750594167.29 total 49.46 last 186.36) - /unlink 11938 (time 1750594177.30 total 59.46 last 193.78) - /unlink 13926 (time 1750594187.30 total 69.47 last 198.63) - /unlink 15818 (time 1750594197.30 total 79.47 last 189.20) - /unlink 17845 (time 1750594207.31 total 89.47 last 202.61) - /unlink 20000 (time 1750594216.50 total 98.66 last 234.53) - /unlink 22122 (time 1750594226.50 total 108.66 last 212.13) - /unlink 24199 (time 1750594236.50 total 118.67 last 207.65) - /unlink 26571 (time 1750594246.51 total 128.67 last 237.14) - /unlink 28591 (time 1750594256.51 total 138.67 last 201.93) - /unlink 30000 (time 1750594264.65 total 146.81 last 173.07) - /unlink 32051 (time 1750594274.65 total 156.82 last 205.07) - /unlink 33828 (time 1750594284.66 total 166.82 last 177.56) - /unlink 35993 (time 1750594294.66 total 176.82 last 216.50) - /unlink 38134 (time 1750594304.66 total 186.83 last 214.05) - /unlink 40000 (time 1750594313.43 total 195.59 last 212.94) - /unlink 42186 (time 1750594323.43 total 205.59 last 218.59) - /unlink 44309 (time 1750594333.43 total 215.59 last 212.29) - /unlink 46417 (time 1750594343.43 total 225.59 last 210.79) - /unlink 48303 (time 1750594353.44 total 235.60 last 188.44) - /unlink 50000 (time 1750594362.93 total 245.09 last 178.81) - /unlink 51795 (time 1750594372.93 total 255.09 last 179.41) - /unlink 53610 (time 1750594382.93 total 265.10 last 181.47) - /unlink 55552 (time 1750594392.94 total 275.10 last 194.12) - /unlink 57412 (time 1750594402.94 total 285.10 last 185.95) - /unlink 59265 (time 1750594412.94 total 295.10 last 185.28) - /unlink 60000 (time 1750594416.77 total 298.93 last 191.88) - /unlink 62120 (time 1750594426.77 total 308.93 last 211.99) - /unlink 64438 (time 1750594436.77 total 318.94 last 231.71) total: 64768 /unlink in 320.50 seconds: 202.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1265 (time 1750594464.32 total 10.00 last 126.48) - open/close 2498 (time 1750594474.33 total 20.01 last 123.22) - open/close 3805 (time 1750594484.33 total 30.01 last 130.69) - open/close 5011 (time 1750594494.33 total 40.01 last 120.58) - open/close 6149 (time 1750594504.34 total 50.01 last 113.77) - open/close 7223 (time 1750594514.34 total 60.02 last 107.38) - open/close 8274 (time 1750594524.34 total 70.02 last 105.06) - open/close 9408 (time 1750594534.35 total 80.03 last 113.31) - open/close 10000 (time 1750594539.64 total 85.32 last 111.92) - open/close 11071 (time 1750594549.64 total 95.32 last 107.04) - open/close 12280 (time 1750594559.65 total 105.33 last 120.82) - open/close 13357 (time 1750594569.66 total 115.33 last 107.64) - open/close 14543 (time 1750594579.66 total 125.34 last 118.57) - open/close 15746 (time 1750594589.66 total 135.34 last 120.28) - open/close 16866 (time 1750594599.66 total 145.34 last 111.97) - open/close 18049 (time 1750594609.66 total 155.34 last 118.29) - open/close 19141 (time 1750594619.67 total 165.35 last 109.12) - open/close 20000 (time 1750594626.42 total 172.10 last 127.20) - open/close 21448 (time 1750594636.43 total 182.10 last 144.75) - open/close 22708 (time 1750594646.43 total 192.11 last 125.91) - open/close 23890 (time 1750594656.44 total 202.12 last 118.07) - open/close 25069 (time 1750594666.44 total 212.12 last 117.90) - open/close 26255 (time 1750594676.45 total 222.12 last 118.59) - open/close 27282 (time 1750594686.45 total 232.13 last 102.62) - open/close 28349 (time 1750594696.46 total 242.13 last 106.67) - open/close 29559 (time 1750594706.46 total 252.14 last 120.93) - open/close 30000 (time 1750594709.98 total 255.66 last 125.41) - open/close 31354 (time 1750594719.98 total 265.66 last 135.38) - open/close 32598 (time 1750594729.98 total 275.66 last 124.40) - open/close 33749 (time 1750594739.99 total 285.67 last 115.01) - open/close 34949 (time 1750594749.99 total 295.67 last 119.95) - open/close 36254 (time 1750594759.99 total 305.67 last 130.48) - open/close 37584 (time 1750594770.00 total 315.67 last 132.98) - open/close 38742 (time 1750594780.00 total 325.68 last 115.74) - open/close 39935 (time 1750594790.01 total 335.69 last 119.22) - open/close 41056 (time 1750594800.01 total 345.69 last 112.04) - open/close 42196 (time 1750594810.01 total 355.69 last 114.00) - open/close 43315 (time 1750594820.02 total 365.69 last 111.87) - open/close 44548 (time 1750594830.02 total 375.69 last 123.29) - open/close 45671 (time 1750594840.02 total 385.70 last 112.22) - open/close 46936 (time 1750594850.03 total 395.71 last 126.41) - open/close 48003 (time 1750594860.03 total 405.71 last 106.66) - open/close 49331 (time 1750594870.04 total 415.71 last 132.77) - open/close 50000 (time 1750594875.47 total 421.15 last 123.09) - open/close 51209 (time 1750594885.48 total 431.16 last 120.77) - open/close 52354 (time 1750594895.48 total 441.16 last 114.49) - open/close 53404 (time 1750594905.49 total 451.17 last 104.95) - open/close 54523 (time 1750594915.49 total 461.17 last 111.84) - open/close 55635 (time 1750594925.51 total 471.19 last 110.99) - open/close 56825 (time 1750594935.51 total 481.19 last 118.98) - open/close 57802 (time 1750594945.52 total 491.20 last 97.67) - open/close 58832 (time 1750594955.52 total 501.20 last 103.00) - open/close 59926 (time 1750594965.52 total 511.20 last 109.35) - open/close 60903 (time 1750594975.53 total 521.20 last 97.67) - open/close 62003 (time 1750594985.53 total 531.21 last 109.92) - open/close 63197 (time 1750594995.54 total 541.22 last 119.31) - open/close 64229 (time 1750595005.54 total 551.22 last 103.16) total: 64768 open/close in 556.41 seconds: 116.40 ops/second - /unlink 2080 (time 1750595022.66 total 10.00 last 207.98) - /unlink 3995 (time 1750595032.67 total 20.01 last 191.42) - /unlink 5733 (time 1750595042.67 total 30.01 last 173.69) - /unlink 7593 (time 1750595052.68 total 40.02 last 185.93) - /unlink 9564 (time 1750595062.68 total 50.02 last 197.07) - /unlink 10000 (time 1750595064.94 total 52.28 last 193.05) - /unlink 11661 (time 1750595074.94 total 62.28 last 166.08) - /unlink 13485 (time 1750595084.94 total 72.28 last 182.39) - /unlink 15269 (time 1750595094.94 total 82.28 last 178.33) - /unlink 17048 (time 1750595104.94 total 92.28 last 177.90) - /unlink 18971 (time 1750595114.95 total 102.28 last 192.27) - /unlink 20000 (time 1750595120.46 total 107.80 last 186.50) - /unlink 22068 (time 1750595130.47 total 117.80 last 206.76) - /unlink 23932 (time 1750595140.47 total 127.80 last 186.39) - /unlink 25903 (time 1750595150.47 total 137.81 last 197.06) - /unlink 27962 (time 1750595160.47 total 147.81 last 205.85) - /unlink 29879 (time 1750595170.47 total 157.81 last 191.63) - /unlink 31897 (time 1750595180.47 total 167.81 last 201.79) - /unlink 34097 (time 1750595190.47 total 177.81 last 219.98) - /unlink 36193 (time 1750595200.48 total 187.81 last 209.55) - /unlink 38166 (time 1750595210.48 total 197.82 last 197.29) - /unlink 40000 (time 1750595218.06 total 205.40 last 241.86) - /unlink 41840 (time 1750595228.06 total 215.40 last 183.97) - /unlink 43880 (time 1750595238.07 total 225.40 last 203.92) - /unlink 45729 (time 1750595248.07 total 235.41 last 184.88) - /unlink 47560 (time 1750595258.07 total 245.41 last 183.02) - /unlink 49675 (time 1750595268.07 total 255.41 last 211.50) - /unlink 51880 (time 1750595278.07 total 265.41 last 220.47) - /unlink 53881 (time 1750595288.07 total 275.41 last 200.09) - /unlink 55868 (time 1750595298.08 total 285.41 last 198.65) - /unlink 57768 (time 1750595308.08 total 295.42 last 189.88) - /unlink 59710 (time 1750595318.08 total 305.42 last 194.18) - /unlink 61588 (time 1750595328.09 total 315.43 last 187.69) - /unlink 63431 (time 1750595338.09 total 325.43 last 184.22) total: 64768 /unlink in 332.80 seconds: 194.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2