== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:50:02 (1746132602) Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) oleg330-server: oleg330-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg330-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg330-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1394 (time 1746132657.38 total 10.00 last 139.38) - open/close 2453 (time 1746132667.39 total 20.00 last 105.86) - open/close 3552 (time 1746132677.39 total 30.01 last 109.88) - open/close 4699 (time 1746132687.39 total 40.01 last 114.68) - open/close 5973 (time 1746132697.39 total 50.01 last 127.35) - open/close 6937 (time 1746132707.39 total 60.01 last 96.40) - open/close 8006 (time 1746132717.40 total 70.02 last 106.83) - open/close 9011 (time 1746132727.40 total 80.02 last 100.47) - open/close 9959 (time 1746132737.41 total 90.03 last 94.72) - open/close 10895 (time 1746132747.42 total 100.04 last 93.53) - open/close 11986 (time 1746132757.43 total 110.04 last 109.04) - open/close 13244 (time 1746132767.43 total 120.04 last 125.79) - open/close 14302 (time 1746132777.43 total 130.05 last 105.77) - open/close 15367 (time 1746132787.44 total 140.06 last 106.38) - open/close 16540 (time 1746132797.44 total 150.06 last 117.28) - open/close 17748 (time 1746132807.45 total 160.07 last 120.73) - open/close 18735 (time 1746132817.46 total 170.08 last 98.61) - open/close 19763 (time 1746132827.46 total 180.08 last 102.76) - open/close 20000 (time 1746132829.72 total 182.33 last 105.17) - open/close 21185 (time 1746132839.73 total 192.35 last 118.36) - open/close 22275 (time 1746132849.74 total 202.36 last 108.90) - open/close 23483 (time 1746132859.74 total 212.36 last 120.72) - open/close 24644 (time 1746132869.76 total 222.37 last 115.96) - open/close 25647 (time 1746132879.76 total 232.38 last 100.24) - open/close 26655 (time 1746132889.77 total 242.39 last 100.70) - open/close 27693 (time 1746132899.78 total 252.39 last 103.76) - open/close 28912 (time 1746132909.78 total 262.40 last 121.88) - open/close 30000 (time 1746132919.12 total 271.74 last 116.41) - open/close 31210 (time 1746132929.13 total 281.74 last 120.97) - open/close 32530 (time 1746132939.14 total 291.75 last 131.88) - open/close 33719 (time 1746132949.14 total 301.76 last 118.79) - open/close 34926 (time 1746132959.15 total 311.77 last 120.65) - open/close 36156 (time 1746132969.15 total 321.77 last 122.95) - open/close 37502 (time 1746132979.16 total 331.77 last 134.55) - open/close 38657 (time 1746132989.17 total 341.79 last 115.30) - open/close 39872 (time 1746132999.19 total 351.80 last 121.35) - open/close 41376 (time 1746133009.19 total 361.81 last 150.33) - open/close 42758 (time 1746133019.19 total 371.81 last 138.19) - open/close 44255 (time 1746133029.19 total 381.81 last 149.68) - open/close 45489 (time 1746133039.19 total 391.81 last 123.39) - open/close 46661 (time 1746133049.20 total 401.82 last 117.14) - open/close 47955 (time 1746133059.21 total 411.83 last 129.22) - open/close 49300 (time 1746133069.22 total 421.84 last 134.43) - open/close 50000 (time 1746133075.46 total 428.08 last 112.17) - open/close 51510 (time 1746133085.46 total 438.08 last 150.96) - open/close 53022 (time 1746133095.46 total 448.08 last 151.18) - open/close 54352 (time 1746133105.47 total 458.09 last 132.88) - open/close 55463 (time 1746133115.49 total 468.11 last 110.91) - open/close 56472 (time 1746133125.50 total 478.11 last 100.82) - open/close 57781 (time 1746133135.50 total 488.11 last 130.89) - open/close 58850 (time 1746133145.50 total 498.11 last 106.90) - open/close 59845 (time 1746133155.50 total 508.12 last 99.43) - open/close 60936 (time 1746133165.51 total 518.13 last 108.99) - open/close 62017 (time 1746133175.53 total 528.14 last 107.98) - open/close 63149 (time 1746133185.53 total 538.15 last 113.17) - open/close 64276 (time 1746133195.53 total 548.15 last 112.69) total: 64768 open/close in 552.12 seconds: 117.31 ops/second - /unlink 2339 (time 1746133210.99 total 10.00 last 233.88) - /unlink 4416 (time 1746133220.99 total 20.01 last 207.55) - /unlink 6559 (time 1746133231.00 total 30.01 last 214.22) - /unlink 8332 (time 1746133241.00 total 40.01 last 177.29) - /unlink 10000 (time 1746133249.08 total 48.10 last 206.28) - /unlink 12242 (time 1746133259.09 total 58.11 last 224.03) - /unlink 14592 (time 1746133269.09 total 68.11 last 234.95) - /unlink 16632 (time 1746133279.10 total 78.11 last 203.86) - /unlink 18623 (time 1746133289.10 total 88.12 last 199.01) - /unlink 20000 (time 1746133296.91 total 95.92 last 176.48) - /unlink 21790 (time 1746133306.91 total 105.93 last 178.92) - /unlink 23751 (time 1746133316.92 total 115.93 last 195.97) - /unlink 25858 (time 1746133326.92 total 125.94 last 210.63) - /unlink 27774 (time 1746133336.92 total 135.94 last 191.52) - /unlink 29855 (time 1746133346.93 total 145.94 last 208.08) - /unlink 32052 (time 1746133356.93 total 155.94 last 219.67) - /unlink 33839 (time 1746133366.93 total 165.94 last 178.68) - /unlink 36086 (time 1746133376.93 total 175.95 last 224.66) - /unlink 39336 (time 1746133386.93 total 185.95 last 324.98) - /unlink 40000 (time 1746133389.38 total 188.40 last 270.80) - /unlink 42150 (time 1746133399.38 total 198.40 last 214.97) - /unlink 44546 (time 1746133409.38 total 208.40 last 239.60) - /unlink 46904 (time 1746133419.39 total 218.40 last 235.78) - /unlink 49365 (time 1746133429.39 total 228.40 last 246.07) - /unlink 50000 (time 1746133431.94 total 230.96 last 248.58) - /unlink 52256 (time 1746133441.94 total 240.96 last 225.57) - /unlink 54447 (time 1746133451.95 total 250.96 last 219.03) - /unlink 56457 (time 1746133461.95 total 260.96 last 200.96) - /unlink 58477 (time 1746133471.95 total 270.96 last 201.98) - /unlink 60000 (time 1746133479.16 total 278.17 last 211.25) - /unlink 61886 (time 1746133489.16 total 288.18 last 188.54) - /unlink 63879 (time 1746133499.16 total 298.18 last 199.29) total: 64768 /unlink in 302.76 seconds: 213.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1230 (time 1746133529.62 total 10.00 last 122.97) - open/close 2290 (time 1746133539.63 total 20.00 last 105.99) - open/close 3513 (time 1746133549.63 total 30.00 last 122.29) - open/close 5050 (time 1746133559.63 total 40.01 last 153.67) - open/close 6294 (time 1746133569.64 total 50.01 last 124.32) - open/close 7530 (time 1746133579.64 total 60.01 last 123.59) - open/close 8879 (time 1746133589.64 total 70.02 last 134.85) - open/close 10000 (time 1746133597.91 total 78.28 last 135.61) - open/close 11293 (time 1746133607.91 total 88.29 last 129.28) - open/close 12535 (time 1746133617.91 total 98.29 last 124.17) - open/close 13806 (time 1746133627.92 total 108.30 last 126.99) - open/close 15128 (time 1746133637.92 total 118.30 last 132.17) - open/close 16338 (time 1746133647.92 total 128.30 last 120.99) - open/close 17503 (time 1746133657.93 total 138.30 last 116.44) - open/close 18763 (time 1746133667.93 total 148.31 last 125.98) - open/close 20000 (time 1746133677.15 total 157.53 last 134.18) - open/close 21378 (time 1746133687.15 total 167.53 last 137.74) - open/close 22627 (time 1746133697.15 total 177.53 last 124.90) - open/close 23792 (time 1746133707.16 total 187.54 last 116.38) - open/close 24910 (time 1746133717.17 total 197.54 last 111.77) - open/close 26177 (time 1746133727.17 total 207.54 last 126.68) - open/close 27469 (time 1746133737.17 total 217.55 last 129.17) - open/close 28648 (time 1746133747.17 total 227.55 last 117.86) - open/close 29854 (time 1746133757.18 total 237.56 last 120.51) - open/close 31096 (time 1746133767.20 total 247.57 last 123.99) - open/close 32455 (time 1746133777.20 total 257.58 last 135.87) - open/close 33676 (time 1746133787.20 total 267.58 last 122.06) - open/close 34975 (time 1746133797.20 total 277.58 last 129.87) - open/close 36261 (time 1746133807.21 total 287.59 last 128.50) - open/close 37411 (time 1746133817.21 total 297.59 last 114.97) - open/close 38403 (time 1746133827.22 total 307.60 last 99.14) - open/close 39390 (time 1746133837.22 total 317.60 last 98.65) - open/close 40000 (time 1746133843.23 total 323.61 last 101.60) - open/close 41146 (time 1746133853.23 total 333.61 last 114.58) - open/close 42267 (time 1746133863.24 total 343.61 last 112.03) - open/close 43502 (time 1746133873.24 total 353.62 last 123.43) - open/close 44834 (time 1746133883.24 total 363.62 last 133.17) - open/close 45935 (time 1746133893.25 total 373.63 last 110.00) - open/close 47076 (time 1746133903.26 total 383.63 last 114.08) - open/close 48346 (time 1746133913.26 total 393.63 last 126.99) - open/close 49630 (time 1746133923.26 total 403.64 last 128.37) - open/close 50000 (time 1746133926.18 total 406.56 last 126.72) - open/close 51182 (time 1746133936.20 total 416.58 last 117.91) - open/close 52314 (time 1746133946.21 total 426.59 last 113.08) - open/close 53321 (time 1746133956.26 total 436.64 last 100.25) - open/close 54278 (time 1746133966.26 total 446.64 last 95.64) - open/close 55288 (time 1746133976.27 total 456.65 last 100.95) - open/close 56298 (time 1746133986.27 total 466.65 last 101.00) - open/close 57387 (time 1746133996.31 total 476.69 last 108.43) - open/close 58339 (time 1746134006.31 total 486.69 last 95.19) - open/close 59345 (time 1746134016.31 total 496.69 last 100.60) - open/close 60000 (time 1746134022.15 total 502.52 last 112.34) - open/close 61062 (time 1746134032.16 total 512.54 last 106.02) - open/close 62047 (time 1746134042.17 total 522.55 last 98.44) - open/close 62968 (time 1746134052.17 total 532.55 last 92.07) - open/close 63933 (time 1746134062.19 total 542.57 last 96.33) total: 64768 open/close in 552.03 seconds: 117.33 ops/second - /unlink 1571 (time 1746134083.88 total 10.01 last 157.01) - /unlink 3119 (time 1746134093.89 total 20.01 last 154.71) - /unlink 4668 (time 1746134103.89 total 30.02 last 154.84) - /unlink 6221 (time 1746134113.89 total 40.02 last 155.24) - /unlink 7789 (time 1746134123.90 total 50.02 last 156.77) - /unlink 9324 (time 1746134133.90 total 60.03 last 153.40) - /unlink 10000 (time 1746134138.11 total 64.23 last 160.64) - /unlink 11667 (time 1746134148.11 total 74.24 last 166.68) - /unlink 13383 (time 1746134158.12 total 84.24 last 171.53) - /unlink 15161 (time 1746134168.12 total 94.24 last 177.78) - /unlink 17055 (time 1746134178.12 total 104.24 last 189.38) - /unlink 18727 (time 1746134188.12 total 114.24 last 167.20) - /unlink 20000 (time 1746134194.81 total 120.93 last 190.38) - /unlink 21794 (time 1746134204.81 total 130.93 last 179.36) - /unlink 23653 (time 1746134214.81 total 140.93 last 185.89) - /unlink 25365 (time 1746134224.81 total 150.94 last 171.09) - /unlink 27028 (time 1746134234.82 total 160.95 last 166.19) - /unlink 28833 (time 1746134244.82 total 170.95 last 180.49) - /unlink 30000 (time 1746134251.09 total 177.21 last 186.16) - /unlink 31600 (time 1746134261.09 total 187.22 last 159.97) - /unlink 33444 (time 1746134271.09 total 197.22 last 184.36) - /unlink 35438 (time 1746134281.10 total 207.22 last 199.38) - /unlink 37446 (time 1746134291.10 total 217.22 last 200.73) - /unlink 39648 (time 1746134301.10 total 227.23 last 220.12) - /unlink 41666 (time 1746134311.11 total 237.23 last 201.75) - /unlink 43573 (time 1746134321.11 total 247.23 last 190.69) - /unlink 45389 (time 1746134331.11 total 257.23 last 181.60) - /unlink 47261 (time 1746134341.11 total 267.23 last 187.17) - /unlink 49273 (time 1746134351.11 total 277.23 last 201.19) - /unlink 50000 (time 1746134354.78 total 280.91 last 197.87) - /unlink 51946 (time 1746134364.78 total 290.91 last 194.60) - /unlink 53792 (time 1746134374.79 total 300.91 last 184.51) - /unlink 55802 (time 1746134384.79 total 310.92 last 200.92) - /unlink 57681 (time 1746134394.79 total 320.92 last 187.87) - /unlink 59621 (time 1746134404.79 total 330.92 last 193.98) - /unlink 60000 (time 1746134406.83 total 332.96 last 186.06) - /unlink 61842 (time 1746134416.83 total 342.96 last 184.16) - /unlink 63960 (time 1746134426.84 total 352.96 last 211.72) total: 64768 /unlink in 356.47 seconds: 181.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1120 (time 1746134462.23 total 10.00 last 111.95) - open/close 2373 (time 1746134472.23 total 20.01 last 125.26) - open/close 3611 (time 1746134482.24 total 30.02 last 123.69) - open/close 4820 (time 1746134492.25 total 40.03 last 120.79) - open/close 5990 (time 1746134502.25 total 50.03 last 116.98) - open/close 7176 (time 1746134512.26 total 60.04 last 118.50) - open/close 8376 (time 1746134522.26 total 70.04 last 119.99) - open/close 9554 (time 1746134532.26 total 80.04 last 117.75) - open/close 10000 (time 1746134536.37 total 84.14 last 108.68) - open/close 11069 (time 1746134546.37 total 94.15 last 106.87) - open/close 12048 (time 1746134556.40 total 104.18 last 97.59) - open/close 13051 (time 1746134566.41 total 114.19 last 100.22) - open/close 14176 (time 1746134576.41 total 124.19 last 112.50) - open/close 15353 (time 1746134586.41 total 134.19 last 117.65) - open/close 16599 (time 1746134596.42 total 144.20 last 124.55) - open/close 17978 (time 1746134606.42 total 154.20 last 137.88) - open/close 19117 (time 1746134616.42 total 164.20 last 113.90) - open/close 20000 (time 1746134624.01 total 171.79 last 116.36) - open/close 21229 (time 1746134634.01 total 181.79 last 122.84) - open/close 22463 (time 1746134644.01 total 191.79 last 123.39) - open/close 23702 (time 1746134654.02 total 201.80 last 123.83) - open/close 25177 (time 1746134664.02 total 211.80 last 147.47) - open/close 26635 (time 1746134674.02 total 221.80 last 145.77) - open/close 28054 (time 1746134684.03 total 231.81 last 141.85) - open/close 29538 (time 1746134694.03 total 241.81 last 148.34) - open/close 30000 (time 1746134696.97 total 244.75 last 157.34) - open/close 31399 (time 1746134706.97 total 254.75 last 139.88) - open/close 32765 (time 1746134716.97 total 264.75 last 136.57) - open/close 34270 (time 1746134726.97 total 274.75 last 150.46) - open/close 35627 (time 1746134736.98 total 284.76 last 135.56) - open/close 36982 (time 1746134746.99 total 294.77 last 135.45) - open/close 38670 (time 1746134756.99 total 304.77 last 168.79) - open/close 40000 (time 1746134765.54 total 313.32 last 155.56) - open/close 41513 (time 1746134775.54 total 323.32 last 151.25) - open/close 42838 (time 1746134785.54 total 333.32 last 132.44) - open/close 44522 (time 1746134795.55 total 343.33 last 168.36) - open/close 46220 (time 1746134805.55 total 353.33 last 169.77) - open/close 47650 (time 1746134815.55 total 363.33 last 142.94) - open/close 49316 (time 1746134825.56 total 373.34 last 166.52) - open/close 50000 (time 1746134829.77 total 377.55 last 162.50) - open/close 51684 (time 1746134839.77 total 387.55 last 168.34) - open/close 53320 (time 1746134849.78 total 397.56 last 163.52) - open/close 54840 (time 1746134859.78 total 407.56 last 151.95) - open/close 56254 (time 1746134869.78 total 417.56 last 141.35) - open/close 58263 (time 1746134879.78 total 427.56 last 200.87) - open/close 60000 (time 1746134889.51 total 437.29 last 178.57) - open/close 61836 (time 1746134899.52 total 447.30 last 183.47) - open/close 63356 (time 1746134909.52 total 457.30 last 151.95) total: 64768 open/close in 466.90 seconds: 138.72 ops/second - /unlink 3115 (time 1746134930.80 total 10.00 last 311.42) - /unlink 6813 (time 1746134940.80 total 20.01 last 369.69) - /unlink 10000 (time 1746134950.06 total 29.26 last 344.28) - /unlink 12811 (time 1746134960.06 total 39.26 last 281.08) - /unlink 15408 (time 1746134970.06 total 49.26 last 259.69) - /unlink 17760 (time 1746134980.06 total 59.27 last 235.10) - /unlink 20000 (time 1746134989.52 total 68.73 last 236.82) - /unlink 22677 (time 1746134999.52 total 78.73 last 267.69) - /unlink 25681 (time 1746135009.53 total 88.73 last 300.23) - /unlink 28778 (time 1746135019.53 total 98.74 last 309.56) - /unlink 30000 (time 1746135023.53 total 102.74 last 305.49) - /unlink 33171 (time 1746135033.53 total 112.74 last 317.05) - /unlink 35921 (time 1746135043.53 total 122.74 last 274.98) - /unlink 38085 (time 1746135053.54 total 132.74 last 216.36) - /unlink 40000 (time 1746135061.11 total 140.32 last 252.82) - /unlink 42865 (time 1746135071.12 total 150.32 last 286.35) - /unlink 45408 (time 1746135081.12 total 160.32 last 254.23) - /unlink 48523 (time 1746135091.12 total 170.32 last 311.48) - /unlink 50000 (time 1746135096.86 total 176.06 last 257.38) - /unlink 52713 (time 1746135106.86 total 186.07 last 271.21) - /unlink 55315 (time 1746135116.86 total 196.07 last 260.19) - /unlink 58059 (time 1746135126.86 total 206.07 last 274.40) - /unlink 60000 (time 1746135134.13 total 213.34 last 267.02) - /unlink 62987 (time 1746135144.14 total 223.34 last 298.54) total: 64768 /unlink in 230.70 seconds: 280.75 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1390 (time 1746135183.20 total 10.00 last 138.94) - open/close 2824 (time 1746135193.20 total 20.01 last 143.39) - open/close 4208 (time 1746135203.21 total 30.01 last 138.33) - open/close 5723 (time 1746135213.21 total 40.01 last 151.45) - open/close 7371 (time 1746135223.21 total 50.02 last 164.77) - open/close 8837 (time 1746135233.21 total 60.02 last 146.57) - open/close 10000 (time 1746135241.89 total 68.69 last 134.06) - open/close 11429 (time 1746135251.90 total 78.70 last 142.79) - open/close 12875 (time 1746135261.90 total 88.70 last 144.59) - open/close 14211 (time 1746135271.90 total 98.71 last 133.49) - open/close 15644 (time 1746135281.91 total 108.71 last 143.24) - open/close 16969 (time 1746135291.91 total 118.71 last 132.49) - open/close 18301 (time 1746135301.91 total 128.72 last 133.14) - open/close 19601 (time 1746135311.92 total 138.72 last 129.96) - open/close 20000 (time 1746135314.40 total 141.20 last 160.95) - open/close 21346 (time 1746135324.40 total 151.20 last 134.56) - open/close 22714 (time 1746135334.41 total 161.21 last 136.67) - open/close 23931 (time 1746135344.42 total 171.22 last 121.58) - open/close 25185 (time 1746135354.42 total 181.23 last 125.35) - open/close 26663 (time 1746135364.43 total 191.23 last 147.72) - open/close 28146 (time 1746135374.43 total 201.23 last 148.27) - open/close 29616 (time 1746135384.43 total 211.24 last 146.99) - open/close 30000 (time 1746135387.24 total 214.05 last 136.50) - open/close 31383 (time 1746135397.25 total 224.05 last 138.26) - open/close 32798 (time 1746135407.25 total 234.05 last 141.50) - open/close 34340 (time 1746135417.26 total 244.06 last 154.09) - open/close 35926 (time 1746135427.26 total 254.06 last 158.56) - open/close 37489 (time 1746135437.26 total 264.06 last 156.27) - open/close 38886 (time 1746135447.27 total 274.07 last 139.62) - open/close 40000 (time 1746135455.32 total 282.12 last 138.39) - open/close 41338 (time 1746135465.32 total 292.12 last 133.75) - open/close 42694 (time 1746135475.32 total 302.12 last 135.60) - open/close 44034 (time 1746135485.33 total 312.13 last 133.92) - open/close 45573 (time 1746135495.33 total 322.13 last 153.86) - open/close 46921 (time 1746135505.33 total 332.14 last 134.72) - open/close 48219 (time 1746135515.34 total 342.14 last 129.78) - open/close 49553 (time 1746135525.34 total 352.14 last 133.37) - open/close 50000 (time 1746135528.48 total 355.28 last 142.33) - open/close 51398 (time 1746135538.49 total 365.29 last 139.64) - open/close 52679 (time 1746135548.49 total 375.30 last 128.06) - open/close 54023 (time 1746135558.49 total 385.30 last 134.38) - open/close 55377 (time 1746135568.50 total 395.30 last 135.32) - open/close 56724 (time 1746135578.51 total 405.32 last 134.54) - open/close 58201 (time 1746135588.51 total 415.32 last 147.68) - open/close 59647 (time 1746135598.53 total 425.33 last 144.34) - open/close 60000 (time 1746135601.11 total 427.91 last 136.84) - open/close 61469 (time 1746135611.11 total 437.92 last 146.88) - open/close 62862 (time 1746135621.11 total 447.92 last 139.28) - open/close 64306 (time 1746135631.13 total 457.93 last 144.24) total: 64768 open/close in 461.18 seconds: 140.44 ops/second - /unlink 2660 (time 1746135645.99 total 10.00 last 265.95) - /unlink 4790 (time 1746135656.00 total 20.01 last 212.91) - /unlink 6865 (time 1746135666.00 total 30.01 last 207.48) - /unlink 9256 (time 1746135676.00 total 40.01 last 239.03) - /unlink 10000 (time 1746135679.23 total 43.24 last 230.64) - /unlink 12628 (time 1746135689.23 total 53.24 last 262.74) - /unlink 15085 (time 1746135699.23 total 63.24 last 245.67) - /unlink 17459 (time 1746135709.23 total 73.24 last 237.36) - /unlink 20000 (time 1746135718.16 total 82.17 last 284.70) - /unlink 22467 (time 1746135728.16 total 92.17 last 246.58) - /unlink 24225 (time 1746135738.16 total 102.17 last 175.75) - /unlink 26388 (time 1746135748.17 total 112.18 last 216.23) - /unlink 28517 (time 1746135758.17 total 122.18 last 212.87) - /unlink 30000 (time 1746135766.24 total 130.25 last 183.77) - /unlink 32207 (time 1746135776.24 total 140.25 last 220.65) - /unlink 34285 (time 1746135786.24 total 150.25 last 207.78) - /unlink 36196 (time 1746135796.24 total 160.25 last 191.07) - /unlink 38431 (time 1746135806.25 total 170.26 last 223.32) - /unlink 40000 (time 1746135812.81 total 176.82 last 239.07) - /unlink 42179 (time 1746135822.82 total 186.83 last 217.82) - /unlink 44396 (time 1746135832.82 total 196.83 last 221.66) - /unlink 46488 (time 1746135842.83 total 206.84 last 209.10) - /unlink 48784 (time 1746135852.83 total 216.84 last 229.55) - /unlink 50000 (time 1746135858.09 total 222.10 last 231.11) - /unlink 52569 (time 1746135868.09 total 232.10 last 256.87) - /unlink 55137 (time 1746135878.10 total 242.11 last 256.65) - /unlink 57026 (time 1746135888.10 total 252.11 last 188.74) - /unlink 59155 (time 1746135898.11 total 262.12 last 212.87) - /unlink 60000 (time 1746135901.95 total 265.96 last 220.05) - /unlink 62354 (time 1746135911.95 total 275.96 last 235.37) - /unlink 64681 (time 1746135921.95 total 285.96 last 232.70) total: 64768 /unlink in 286.33 seconds: 226.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1435 (time 1746135953.60 total 10.00 last 143.45) - open/close 2928 (time 1746135963.60 total 20.01 last 149.25) - open/close 4385 (time 1746135973.61 total 30.01 last 145.66) - open/close 6099 (time 1746135983.61 total 40.01 last 171.34) - open/close 8119 (time 1746135993.61 total 50.02 last 201.95) - open/close 10000 (time 1746136002.90 total 59.30 last 202.49) - open/close 11646 (time 1746136012.91 total 69.31 last 164.52) - open/close 13182 (time 1746136022.91 total 79.31 last 153.54) - open/close 14788 (time 1746136032.91 total 89.32 last 160.54) - open/close 16246 (time 1746136042.92 total 99.32 last 145.75) - open/close 17933 (time 1746136052.92 total 109.32 last 168.70) - open/close 19423 (time 1746136062.92 total 119.33 last 148.91) - open/close 20000 (time 1746136066.80 total 123.21 last 148.77) - open/close 21510 (time 1746136076.80 total 133.21 last 151.00) - open/close 22912 (time 1746136086.80 total 143.21 last 140.19) - open/close 24255 (time 1746136096.81 total 153.21 last 134.26) - open/close 25650 (time 1746136106.81 total 163.21 last 139.45) - open/close 27012 (time 1746136116.82 total 173.22 last 136.11) - open/close 28477 (time 1746136126.83 total 183.23 last 146.38) - open/close 29774 (time 1746136136.83 total 193.23 last 129.69) - open/close 31148 (time 1746136146.83 total 203.23 last 137.36) - open/close 32483 (time 1746136156.84 total 213.24 last 133.42) - open/close 33910 (time 1746136166.84 total 223.24 last 142.63) - open/close 35341 (time 1746136176.84 total 233.25 last 143.07) - open/close 36784 (time 1746136186.85 total 243.25 last 144.20) - open/close 38313 (time 1746136196.85 total 253.26 last 152.87) - open/close 39726 (time 1746136206.86 total 263.26 last 141.25) - open/close 41101 (time 1746136216.86 total 273.26 last 137.47) - open/close 43054 (time 1746136226.86 total 283.26 last 195.25) - open/close 44752 (time 1746136236.86 total 293.27 last 169.76) - open/close 46336 (time 1746136246.87 total 303.27 last 158.36) - open/close 48112 (time 1746136256.87 total 313.27 last 177.51) - open/close 49645 (time 1746136266.87 total 323.27 last 153.29) - open/close 50000 (time 1746136269.10 total 325.50 last 159.19) - open/close 51483 (time 1746136279.11 total 335.52 last 148.12) - open/close 52899 (time 1746136289.12 total 345.52 last 141.55) - open/close 54476 (time 1746136299.12 total 355.52 last 157.66) - open/close 56085 (time 1746136309.12 total 365.52 last 160.87) - open/close 57658 (time 1746136319.13 total 375.53 last 157.25) - open/close 59482 (time 1746136329.13 total 385.53 last 182.37) - open/close 60000 (time 1746136332.05 total 388.45 last 177.08) - open/close 61754 (time 1746136342.06 total 398.46 last 175.33) - open/close 63300 (time 1746136352.06 total 408.46 last 154.58) - open/close 64700 (time 1746136362.06 total 418.47 last 139.91) total: 64768 open/close in 418.90 seconds: 154.61 ops/second - /unlink 2305 (time 1746136374.12 total 10.00 last 230.47) - /unlink 4641 (time 1746136384.12 total 20.00 last 233.57) - /unlink 7186 (time 1746136394.13 total 30.00 last 254.45) - /unlink 9897 (time 1746136404.13 total 40.01 last 271.04) - /unlink 12629 (time 1746136414.13 total 50.01 last 273.15) - /unlink 15084 (time 1746136424.13 total 60.01 last 245.42) - /unlink 17498 (time 1746136434.13 total 70.01 last 241.35) - /unlink 20000 (time 1746136442.47 total 78.34 last 300.35) - /unlink 22943 (time 1746136452.47 total 88.34 last 294.28) - /unlink 25943 (time 1746136462.47 total 98.35 last 299.92) - /unlink 29721 (time 1746136472.47 total 108.35 last 377.80) - /unlink 33362 (time 1746136482.47 total 118.35 last 364.00) - /unlink 36416 (time 1746136492.47 total 128.35 last 305.30) - /unlink 39143 (time 1746136502.48 total 138.35 last 272.69) - /unlink 40000 (time 1746136506.82 total 142.70 last 197.19) - /unlink 41871 (time 1746136516.82 total 152.70 last 187.07) - /unlink 44035 (time 1746136526.83 total 162.71 last 216.29) - /unlink 46473 (time 1746136536.83 total 172.71 last 243.68) - /unlink 48979 (time 1746136546.83 total 182.71 last 250.55) - /unlink 50000 (time 1746136550.00 total 185.88 last 322.11) - /unlink 52579 (time 1746136560.00 total 195.88 last 257.88) - /unlink 55214 (time 1746136570.00 total 205.88 last 263.50) - /unlink 58009 (time 1746136580.01 total 215.89 last 279.46) - /unlink 60000 (time 1746136586.89 total 222.77 last 289.38) - /unlink 62848 (time 1746136596.89 total 232.77 last 284.74) total: 64768 /unlink in 238.81 seconds: 271.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server unloading modules via unload_modules_local on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2