== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:50:44 (1746129044) Stopping clients: oleg305-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:-f) oleg305-server: oleg305-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg305-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg305-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg305-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg305-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1163 (time 1746129112.55 total 10.00 last 116.29) - open/close 2319 (time 1746129122.56 total 20.01 last 115.54) - open/close 3460 (time 1746129132.56 total 30.01 last 114.05) - open/close 4639 (time 1746129142.56 total 40.01 last 117.88) - open/close 5852 (time 1746129152.57 total 50.02 last 121.22) - open/close 7145 (time 1746129162.57 total 60.02 last 129.25) - open/close 8238 (time 1746129172.58 total 70.03 last 109.24) - open/close 9364 (time 1746129182.58 total 80.03 last 112.54) - open/close 10000 (time 1746129188.39 total 85.84 last 109.56) - open/close 11006 (time 1746129198.40 total 95.85 last 100.45) - open/close 12157 (time 1746129208.42 total 105.87 last 114.94) - open/close 13236 (time 1746129218.42 total 115.87 last 107.86) - open/close 14489 (time 1746129228.42 total 125.87 last 125.28) - open/close 15943 (time 1746129238.43 total 135.88 last 145.32) - open/close 17266 (time 1746129248.43 total 145.88 last 132.24) - open/close 18544 (time 1746129258.43 total 155.88 last 127.77) - open/close 19885 (time 1746129268.44 total 165.89 last 133.99) - open/close 21123 (time 1746129278.45 total 175.90 last 123.71) - open/close 22199 (time 1746129288.45 total 185.90 last 107.58) - open/close 23280 (time 1746129298.46 total 195.91 last 108.00) - open/close 24323 (time 1746129308.46 total 205.91 last 104.28) - open/close 25486 (time 1746129318.48 total 215.93 last 116.13) - open/close 26675 (time 1746129328.49 total 225.94 last 118.71) - open/close 27928 (time 1746129338.50 total 235.95 last 125.28) - open/close 29093 (time 1746129348.50 total 245.95 last 116.49) - open/close 30000 (time 1746129357.50 total 254.95 last 100.77) - open/close 31078 (time 1746129367.51 total 264.96 last 107.68) - open/close 32297 (time 1746129377.51 total 274.96 last 121.83) - open/close 33745 (time 1746129387.52 total 284.97 last 144.77) - open/close 35090 (time 1746129397.52 total 294.97 last 134.47) - open/close 36694 (time 1746129407.52 total 304.97 last 160.37) - open/close 38573 (time 1746129417.52 total 314.97 last 187.88) - open/close 39986 (time 1746129427.53 total 324.97 last 141.24) - open/close 41569 (time 1746129437.53 total 334.98 last 158.27) - open/close 43054 (time 1746129447.54 total 344.98 last 148.38) - open/close 44653 (time 1746129457.54 total 354.99 last 159.81) - open/close 45985 (time 1746129467.55 total 364.99 last 133.15) - open/close 47563 (time 1746129477.55 total 375.00 last 157.77) - open/close 49079 (time 1746129487.56 total 385.00 last 151.48) - open/close 50000 (time 1746129495.25 total 392.70 last 119.73) - open/close 51473 (time 1746129505.25 total 402.70 last 147.25) - open/close 53037 (time 1746129515.25 total 412.70 last 156.37) - open/close 54707 (time 1746129525.26 total 422.70 last 166.95) - open/close 56040 (time 1746129535.26 total 432.71 last 133.26) - open/close 57466 (time 1746129545.26 total 442.71 last 142.59) - open/close 58715 (time 1746129555.27 total 452.71 last 124.83) - open/close 59877 (time 1746129565.27 total 462.72 last 116.19) - open/close 60982 (time 1746129575.27 total 472.72 last 110.44) - open/close 62162 (time 1746129585.28 total 482.73 last 117.87) - open/close 63507 (time 1746129595.29 total 492.73 last 134.45) total: 64768 open/close in 501.21 seconds: 129.22 ops/second - /unlink 2476 (time 1746129615.14 total 10.01 last 247.45) - /unlink 4590 (time 1746129625.14 total 20.01 last 211.37) - /unlink 6580 (time 1746129635.14 total 30.01 last 198.96) - /unlink 8295 (time 1746129645.15 total 40.02 last 171.33) - /unlink 10000 (time 1746129654.70 total 49.57 last 178.51) - /unlink 11922 (time 1746129664.70 total 59.57 last 192.11) - /unlink 14880 (time 1746129674.71 total 69.58 last 295.72) - /unlink 17407 (time 1746129684.71 total 79.58 last 252.66) - /unlink 19788 (time 1746129694.71 total 89.58 last 238.07) - /unlink 21600 (time 1746129704.72 total 99.59 last 181.09) - /unlink 23438 (time 1746129714.72 total 109.59 last 183.78) - /unlink 25358 (time 1746129724.72 total 119.59 last 191.99) - /unlink 27432 (time 1746129734.72 total 129.59 last 207.38) - /unlink 29849 (time 1746129744.72 total 139.59 last 241.70) - /unlink 32150 (time 1746129754.72 total 149.59 last 230.02) - /unlink 34455 (time 1746129764.72 total 159.59 last 230.44) - /unlink 36301 (time 1746129774.73 total 169.60 last 184.59) - /unlink 38312 (time 1746129784.73 total 179.60 last 201.02) - /unlink 40000 (time 1746129793.39 total 188.26 last 194.97) - /unlink 41777 (time 1746129803.39 total 198.26 last 177.62) - /unlink 43738 (time 1746129813.40 total 208.27 last 196.04) - /unlink 45999 (time 1746129823.40 total 218.27 last 226.01) - /unlink 48419 (time 1746129833.40 total 228.27 last 241.93) - /unlink 50000 (time 1746129840.73 total 235.60 last 215.73) - /unlink 52149 (time 1746129850.73 total 245.60 last 214.87) - /unlink 54260 (time 1746129860.73 total 255.60 last 211.06) - /unlink 56230 (time 1746129870.74 total 265.61 last 196.92) - /unlink 58267 (time 1746129880.74 total 275.61 last 203.66) - /unlink 60000 (time 1746129889.72 total 284.59 last 192.89) - /unlink 61404 (time 1746129899.73 total 294.60 last 140.31) - /unlink 62992 (time 1746129909.74 total 304.61 last 158.71) - /unlink 64634 (time 1746129919.74 total 314.61 last 164.19) total: 64768 /unlink in 315.62 seconds: 205.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1099 (time 1746129947.64 total 10.01 last 109.82) - open/close 2335 (time 1746129957.64 total 20.01 last 123.58) - open/close 3641 (time 1746129967.65 total 30.01 last 130.53) - open/close 4991 (time 1746129977.65 total 40.02 last 134.97) - open/close 6319 (time 1746129987.65 total 50.02 last 132.75) - open/close 7590 (time 1746129997.66 total 60.02 last 127.04) - open/close 8717 (time 1746130007.66 total 70.03 last 112.66) - open/close 9954 (time 1746130017.66 total 80.03 last 123.70) - open/close 11202 (time 1746130027.66 total 90.03 last 124.78) - open/close 12291 (time 1746130037.67 total 100.04 last 108.81) - open/close 13451 (time 1746130047.68 total 110.04 last 115.92) - open/close 14496 (time 1746130057.68 total 120.05 last 104.45) - open/close 15555 (time 1746130067.68 total 130.05 last 105.88) - open/close 16536 (time 1746130077.69 total 140.06 last 98.04) - open/close 17699 (time 1746130087.69 total 150.06 last 116.26) - open/close 18752 (time 1746130097.69 total 160.06 last 105.29) - open/close 19982 (time 1746130107.70 total 170.06 last 122.97) - open/close 21119 (time 1746130117.70 total 180.06 last 113.70) - open/close 22195 (time 1746130127.70 total 190.07 last 107.55) - open/close 23347 (time 1746130137.70 total 200.07 last 115.17) - open/close 24469 (time 1746130147.71 total 210.08 last 112.12) - open/close 25689 (time 1746130157.71 total 220.08 last 121.97) - open/close 26753 (time 1746130167.72 total 230.08 last 106.35) - open/close 27870 (time 1746130177.72 total 240.09 last 111.66) - open/close 29006 (time 1746130187.72 total 250.09 last 113.58) - open/close 30000 (time 1746130197.04 total 259.40 last 106.71) - open/close 31044 (time 1746130207.05 total 269.41 last 104.32) - open/close 32217 (time 1746130217.05 total 279.41 last 117.28) - open/close 33456 (time 1746130227.05 total 289.42 last 123.87) - open/close 34547 (time 1746130237.06 total 299.43 last 109.00) - open/close 35515 (time 1746130247.06 total 309.43 last 96.75) - open/close 36758 (time 1746130257.07 total 319.43 last 124.28) - open/close 37923 (time 1746130267.07 total 329.44 last 116.41) - open/close 38843 (time 1746130277.08 total 339.45 last 91.95) - open/close 40000 (time 1746130286.39 total 348.76 last 124.27) - open/close 41314 (time 1746130296.39 total 358.76 last 131.39) - open/close 42327 (time 1746130306.39 total 368.76 last 101.27) - open/close 43457 (time 1746130316.39 total 378.76 last 112.99) - open/close 44652 (time 1746130326.40 total 388.77 last 119.44) - open/close 45580 (time 1746130336.41 total 398.78 last 92.71) - open/close 46472 (time 1746130346.41 total 408.78 last 89.19) - open/close 47553 (time 1746130356.41 total 418.78 last 108.05) - open/close 48786 (time 1746130366.42 total 428.79 last 123.21) - open/close 49884 (time 1746130376.42 total 438.79 last 109.80) - open/close 50961 (time 1746130386.43 total 448.80 last 107.61) - open/close 52055 (time 1746130396.44 total 458.81 last 109.31) - open/close 53242 (time 1746130406.45 total 468.82 last 118.55) - open/close 54228 (time 1746130416.45 total 478.82 last 98.56) - open/close 55118 (time 1746130426.46 total 488.83 last 88.96) - open/close 56090 (time 1746130436.47 total 498.83 last 97.12) - open/close 57106 (time 1746130446.47 total 508.84 last 101.59) - open/close 58140 (time 1746130456.48 total 518.84 last 103.30) - open/close 59137 (time 1746130466.48 total 528.85 last 99.66) - open/close 60000 (time 1746130475.25 total 537.61 last 98.46) - open/close 61264 (time 1746130485.25 total 547.62 last 126.36) - open/close 62465 (time 1746130495.26 total 557.63 last 119.96) - open/close 63559 (time 1746130505.28 total 567.65 last 109.22) - open/close 64709 (time 1746130515.29 total 577.66 last 114.88) total: 64768 open/close in 578.04 seconds: 112.05 ops/second - /unlink 1968 (time 1746130527.26 total 10.00 last 196.79) - /unlink 4016 (time 1746130537.27 total 20.00 last 204.73) - /unlink 6159 (time 1746130547.27 total 30.00 last 214.30) - /unlink 8122 (time 1746130557.27 total 40.01 last 196.24) - /unlink 10000 (time 1746130565.91 total 48.65 last 217.35) - /unlink 12239 (time 1746130575.91 total 58.65 last 223.88) - /unlink 14918 (time 1746130585.91 total 68.65 last 267.87) - /unlink 16999 (time 1746130595.91 total 78.65 last 208.07) - /unlink 19545 (time 1746130605.92 total 88.66 last 254.48) - /unlink 22213 (time 1746130615.92 total 98.66 last 266.75) - /unlink 24528 (time 1746130625.92 total 108.66 last 231.46) - /unlink 26580 (time 1746130635.92 total 118.66 last 205.17) - /unlink 28721 (time 1746130645.93 total 128.66 last 214.08) - /unlink 30000 (time 1746130651.50 total 134.24 last 229.45) - /unlink 31954 (time 1746130661.50 total 144.24 last 195.33) - /unlink 34092 (time 1746130671.51 total 154.24 last 213.74) - /unlink 36455 (time 1746130681.51 total 164.25 last 236.22) - /unlink 38186 (time 1746130691.52 total 174.25 last 172.98) - /unlink 40000 (time 1746130701.45 total 184.19 last 182.54) - /unlink 42202 (time 1746130711.46 total 194.20 last 220.01) - /unlink 44123 (time 1746130721.47 total 204.20 last 192.01) - /unlink 45807 (time 1746130731.47 total 214.21 last 168.35) - /unlink 47494 (time 1746130741.48 total 224.21 last 168.61) - /unlink 49187 (time 1746130751.48 total 234.21 last 169.25) - /unlink 50000 (time 1746130756.28 total 239.02 last 169.17) - /unlink 51671 (time 1746130766.29 total 249.02 last 167.09) - /unlink 53431 (time 1746130776.29 total 259.02 last 175.95) - /unlink 55340 (time 1746130786.29 total 269.03 last 190.79) - /unlink 57190 (time 1746130796.29 total 279.03 last 184.99) - /unlink 59074 (time 1746130806.30 total 289.03 last 188.38) - /unlink 60000 (time 1746130810.73 total 293.46 last 208.91) - /unlink 62068 (time 1746130820.73 total 303.47 last 206.73) - /unlink 64347 (time 1746130830.74 total 313.47 last 227.76) total: 64768 /unlink in 315.91 seconds: 205.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1164 (time 1746130864.06 total 10.00 last 116.38) - open/close 2225 (time 1746130874.06 total 20.01 last 106.05) - open/close 3296 (time 1746130884.06 total 30.01 last 107.09) - open/close 4500 (time 1746130894.06 total 40.01 last 120.40) - open/close 5465 (time 1746130904.09 total 50.03 last 96.29) - open/close 6408 (time 1746130914.09 total 60.03 last 94.27) - open/close 7438 (time 1746130924.10 total 70.04 last 102.90) - open/close 8490 (time 1746130934.10 total 80.05 last 105.13) - open/close 9420 (time 1746130944.10 total 90.05 last 93.00) - open/close 10000 (time 1746130949.95 total 95.89 last 99.25) - open/close 10974 (time 1746130959.95 total 105.89 last 97.39) - open/close 12031 (time 1746130969.95 total 115.90 last 105.66) - open/close 12989 (time 1746130979.96 total 125.90 last 95.76) - open/close 13945 (time 1746130989.97 total 135.91 last 95.49) - open/close 15003 (time 1746130999.97 total 145.92 last 105.78) - open/close 15974 (time 1746131009.98 total 155.93 last 97.00) - open/close 16905 (time 1746131019.98 total 165.93 last 93.08) - open/close 17952 (time 1746131029.99 total 175.94 last 104.59) - open/close 19049 (time 1746131040.00 total 185.94 last 109.64) - open/close 20000 (time 1746131048.64 total 194.58 last 110.10) - open/close 20973 (time 1746131058.64 total 204.58 last 97.27) - open/close 22178 (time 1746131068.65 total 214.59 last 120.44) - open/close 23189 (time 1746131078.65 total 224.60 last 101.02) - open/close 24124 (time 1746131088.66 total 234.61 last 93.42) - open/close 25216 (time 1746131098.67 total 244.61 last 109.15) - open/close 26499 (time 1746131108.67 total 254.62 last 128.18) - open/close 27553 (time 1746131118.69 total 264.63 last 105.29) - open/close 28608 (time 1746131128.69 total 274.63 last 105.47) - open/close 29852 (time 1746131138.69 total 284.63 last 124.38) - open/close 30867 (time 1746131148.69 total 294.64 last 101.49) - open/close 31868 (time 1746131158.69 total 304.64 last 100.07) - open/close 32833 (time 1746131168.69 total 314.64 last 96.49) - open/close 34001 (time 1746131178.70 total 324.64 last 116.77) - open/close 34944 (time 1746131188.70 total 334.65 last 94.23) - open/close 35842 (time 1746131198.71 total 344.65 last 89.76) - open/close 36839 (time 1746131208.71 total 354.66 last 99.66) - open/close 37875 (time 1746131218.72 total 364.67 last 103.49) - open/close 38916 (time 1746131228.73 total 374.67 last 104.05) - open/close 39817 (time 1746131238.73 total 384.67 last 90.10) - open/close 40961 (time 1746131248.73 total 394.68 last 114.34) - open/close 42064 (time 1746131258.75 total 404.70 last 110.10) - open/close 42992 (time 1746131268.76 total 414.70 last 92.73) - open/close 44030 (time 1746131278.77 total 424.71 last 103.68) - open/close 45120 (time 1746131288.77 total 434.72 last 108.99) - open/close 46169 (time 1746131298.78 total 444.72 last 104.84) - open/close 47170 (time 1746131308.79 total 454.73 last 99.96) - open/close 48145 (time 1746131318.80 total 464.74 last 97.42) - open/close 49200 (time 1746131328.80 total 474.75 last 105.47) - open/close 50000 (time 1746131336.54 total 482.48 last 103.40) - open/close 51019 (time 1746131346.54 total 492.48 last 101.88) - open/close 52027 (time 1746131356.55 total 502.49 last 100.75) - open/close 53019 (time 1746131366.55 total 512.49 last 99.16) - open/close 54006 (time 1746131376.55 total 522.50 last 98.65) - open/close 55054 (time 1746131386.56 total 532.50 last 104.74) - open/close 55973 (time 1746131396.56 total 542.50 last 91.90) - open/close 56953 (time 1746131406.56 total 552.51 last 97.96) - open/close 57982 (time 1746131416.57 total 562.51 last 102.86) - open/close 58951 (time 1746131426.57 total 572.51 last 96.86) - open/close 59923 (time 1746131436.58 total 582.52 last 97.15) - open/close 60968 (time 1746131446.59 total 592.53 last 104.38) - open/close 61967 (time 1746131456.59 total 602.53 last 99.89) - open/close 63004 (time 1746131466.59 total 612.53 last 103.69) - open/close 64113 (time 1746131476.59 total 622.54 last 110.87) total: 64768 open/close in 628.99 seconds: 102.97 ops/second - /unlink 974 (time 1746131495.54 total 10.01 last 97.32) - /unlink 2124 (time 1746131505.54 total 20.01 last 115.00) - /unlink 3107 (time 1746131515.54 total 30.01 last 98.29) - /unlink 4189 (time 1746131525.55 total 40.02 last 108.12) - /unlink 5195 (time 1746131535.55 total 50.02 last 100.55) - /unlink 6264 (time 1746131545.55 total 60.02 last 106.88) - /unlink 7253 (time 1746131555.56 total 70.03 last 98.85) - /unlink 8367 (time 1746131565.57 total 80.04 last 111.30) - /unlink 9342 (time 1746131575.57 total 90.04 last 97.49) - /unlink 10000 (time 1746131581.54 total 96.01 last 110.24) - /unlink 11041 (time 1746131591.54 total 106.01 last 104.06) - /unlink 11972 (time 1746131601.55 total 116.02 last 93.06) - /unlink 12925 (time 1746131611.55 total 126.02 last 95.28) - /unlink 13966 (time 1746131621.56 total 136.02 last 104.03) - /unlink 14955 (time 1746131631.56 total 146.03 last 98.88) - /unlink 15781 (time 1746131641.57 total 156.04 last 82.53) - /unlink 16691 (time 1746131651.58 total 166.05 last 90.86) - /unlink 17705 (time 1746131661.59 total 176.06 last 101.34) - /unlink 18769 (time 1746131671.59 total 186.06 last 106.34) - /unlink 19817 (time 1746131681.59 total 196.06 last 104.79) - /unlink 20826 (time 1746131691.60 total 206.07 last 100.87) - /unlink 21887 (time 1746131701.63 total 216.09 last 105.81) - /unlink 22815 (time 1746131711.63 total 226.09 last 92.80) - /unlink 23835 (time 1746131721.63 total 236.10 last 101.99) - /unlink 24803 (time 1746131731.64 total 246.11 last 96.70) - /unlink 25893 (time 1746131741.64 total 256.11 last 108.93) - /unlink 26953 (time 1746131751.65 total 266.12 last 105.95) - /unlink 27994 (time 1746131761.67 total 276.14 last 103.91) - /unlink 28879 (time 1746131771.67 total 286.14 last 88.47) - /unlink 29918 (time 1746131781.68 total 296.15 last 103.80) - /unlink 30957 (time 1746131791.68 total 306.15 last 103.84) - /unlink 32039 (time 1746131801.71 total 316.18 last 107.95) - /unlink 33010 (time 1746131811.71 total 326.18 last 97.06) - /unlink 33906 (time 1746131821.73 total 336.20 last 89.46) - /unlink 34751 (time 1746131831.74 total 346.21 last 84.41) - /unlink 35679 (time 1746131841.75 total 356.22 last 92.73) - /unlink 36648 (time 1746131851.75 total 366.22 last 96.89) - /unlink 37567 (time 1746131861.75 total 376.22 last 91.88) - /unlink 38573 (time 1746131871.75 total 386.22 last 100.59) - /unlink 39769 (time 1746131881.76 total 396.23 last 119.47) - /unlink 40809 (time 1746131891.77 total 406.23 last 103.96) - /unlink 41747 (time 1746131901.77 total 416.24 last 93.74) - /unlink 42821 (time 1746131911.78 total 426.24 last 107.36) - /unlink 43992 (time 1746131921.78 total 436.25 last 117.03) - /unlink 45067 (time 1746131931.79 total 446.26 last 107.46) - /unlink 46106 (time 1746131941.79 total 456.26 last 103.87) - /unlink 47120 (time 1746131951.79 total 466.26 last 101.39) - /unlink 48160 (time 1746131961.79 total 476.26 last 103.99) - /unlink 49265 (time 1746131971.80 total 486.27 last 110.39) - /unlink 50000 (time 1746131980.42 total 494.88 last 85.31) - /unlink 50967 (time 1746131990.43 total 504.89 last 96.61) - /unlink 51868 (time 1746132000.43 total 514.90 last 90.08) - /unlink 53073 (time 1746132010.43 total 524.90 last 120.47) - /unlink 54097 (time 1746132020.43 total 534.90 last 102.35) - /unlink 55108 (time 1746132030.46 total 544.92 last 100.89) - /unlink 56141 (time 1746132040.46 total 554.93 last 103.23) - /unlink 57161 (time 1746132050.46 total 564.93 last 101.99) - /unlink 58274 (time 1746132060.47 total 574.94 last 111.22) - /unlink 59298 (time 1746132070.48 total 584.95 last 102.33) - /unlink 60000 (time 1746132077.15 total 591.62 last 105.17) - /unlink 61150 (time 1746132087.16 total 601.63 last 114.89) - /unlink 62299 (time 1746132097.17 total 611.64 last 114.82) - /unlink 63389 (time 1746132107.18 total 621.65 last 108.85) - /unlink 64475 (time 1746132117.19 total 631.66 last 108.49) total: 64768 /unlink in 634.57 seconds: 102.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1164 (time 1746132150.26 total 10.01 last 116.30) - open/close 2154 (time 1746132160.26 total 20.01 last 98.95) - open/close 3084 (time 1746132170.27 total 30.02 last 92.96) - open/close 4195 (time 1746132180.27 total 40.02 last 111.09) - open/close 5333 (time 1746132190.28 total 50.03 last 113.71) - open/close 6488 (time 1746132200.28 total 60.03 last 115.43) - open/close 7548 (time 1746132210.29 total 70.04 last 105.95) - open/close 8632 (time 1746132220.30 total 80.05 last 108.27) - open/close 9857 (time 1746132230.32 total 90.07 last 122.32) - open/close 11002 (time 1746132240.32 total 100.07 last 114.47) - open/close 12079 (time 1746132250.32 total 110.08 last 107.63) - open/close 13180 (time 1746132260.33 total 120.08 last 110.05) - open/close 14242 (time 1746132270.33 total 130.08 last 106.15) - open/close 15534 (time 1746132280.34 total 140.09 last 129.12) - open/close 16779 (time 1746132290.34 total 150.09 last 124.49) - open/close 18132 (time 1746132300.34 total 160.09 last 135.29) - open/close 19299 (time 1746132310.35 total 170.10 last 116.64) - open/close 20000 (time 1746132315.52 total 175.27 last 135.46) - open/close 21096 (time 1746132325.52 total 185.27 last 109.57) - open/close 22281 (time 1746132335.53 total 195.28 last 118.46) - open/close 23452 (time 1746132345.53 total 205.28 last 117.10) - open/close 24656 (time 1746132355.54 total 215.29 last 120.24) - open/close 25790 (time 1746132365.55 total 225.30 last 113.31) - open/close 26790 (time 1746132375.55 total 235.30 last 99.98) - open/close 27819 (time 1746132385.56 total 245.31 last 102.84) - open/close 28975 (time 1746132395.56 total 255.31 last 115.55) - open/close 29939 (time 1746132405.57 total 265.32 last 96.33) - open/close 30837 (time 1746132415.57 total 275.32 last 89.79) - open/close 32011 (time 1746132425.57 total 285.32 last 117.38) - open/close 33163 (time 1746132435.58 total 295.33 last 115.15) - open/close 34112 (time 1746132445.58 total 305.33 last 94.83) - open/close 35105 (time 1746132455.59 total 315.34 last 99.20) - open/close 36183 (time 1746132465.60 total 325.35 last 107.76) - open/close 37126 (time 1746132475.60 total 335.35 last 94.30) - open/close 38023 (time 1746132485.61 total 345.36 last 89.63) - open/close 38989 (time 1746132495.61 total 355.36 last 96.56) - open/close 40000 (time 1746132504.17 total 363.92 last 118.08) - open/close 41183 (time 1746132514.17 total 373.92 last 118.28) - open/close 42202 (time 1746132524.18 total 383.93 last 101.80) - open/close 43244 (time 1746132534.19 total 393.94 last 104.16) - open/close 44264 (time 1746132544.19 total 403.94 last 101.96) - open/close 45338 (time 1746132554.21 total 413.96 last 107.24) - open/close 46218 (time 1746132564.21 total 423.96 last 87.94) - open/close 47150 (time 1746132574.22 total 433.97 last 93.15) - open/close 48279 (time 1746132584.22 total 443.97 last 112.85) - open/close 49498 (time 1746132594.23 total 453.98 last 121.84) - open/close 50000 (time 1746132599.14 total 458.89 last 102.30) - open/close 51175 (time 1746132609.14 total 468.89 last 117.40) - open/close 52249 (time 1746132619.15 total 478.90 last 107.34) - open/close 53410 (time 1746132629.16 total 488.91 last 115.99) - open/close 54590 (time 1746132639.16 total 498.91 last 117.93) - open/close 55888 (time 1746132649.17 total 508.92 last 129.73) - open/close 57246 (time 1746132659.17 total 518.92 last 135.79) - open/close 58224 (time 1746132669.18 total 528.93 last 97.70) - open/close 59214 (time 1746132679.18 total 538.94 last 98.96) - open/close 60000 (time 1746132686.89 total 546.64 last 101.98) - open/close 61176 (time 1746132696.90 total 556.65 last 117.55) - open/close 62073 (time 1746132706.90 total 566.65 last 89.68) - open/close 63085 (time 1746132716.90 total 576.65 last 101.20) - open/close 64124 (time 1746132726.90 total 586.65 last 103.86) total: 64768 open/close in 593.20 seconds: 109.18 ops/second - /unlink 1514 (time 1746132745.54 total 10.01 last 151.25) - /unlink 3241 (time 1746132755.54 total 20.01 last 172.70) - /unlink 5110 (time 1746132765.54 total 30.01 last 186.86) - /unlink 6929 (time 1746132775.54 total 40.01 last 181.85) - /unlink 8416 (time 1746132785.54 total 50.01 last 148.69) - /unlink 10000 (time 1746132793.68 total 58.15 last 194.64) - /unlink 12065 (time 1746132803.69 total 68.16 last 206.40) - /unlink 13673 (time 1746132813.69 total 78.16 last 160.70) - /unlink 15424 (time 1746132823.70 total 88.17 last 175.05) - /unlink 17341 (time 1746132833.70 total 98.17 last 191.67) - /unlink 19296 (time 1746132843.70 total 108.17 last 195.45) - /unlink 20000 (time 1746132847.71 total 112.19 last 175.35) - /unlink 21872 (time 1746132857.72 total 122.19 last 187.15) - /unlink 23866 (time 1746132867.72 total 132.19 last 199.36) - /unlink 25739 (time 1746132877.72 total 142.20 last 187.21) - /unlink 27418 (time 1746132887.73 total 152.20 last 167.84) - /unlink 29158 (time 1746132897.74 total 162.21 last 173.83) - /unlink 30000 (time 1746132901.53 total 166.01 last 221.81) - /unlink 32197 (time 1746132911.53 total 176.01 last 219.68) - /unlink 34290 (time 1746132921.54 total 186.01 last 209.29) - /unlink 36708 (time 1746132931.54 total 196.01 last 241.80) - /unlink 39104 (time 1746132941.54 total 206.01 last 239.43) - /unlink 40000 (time 1746132945.96 total 210.44 last 202.66) - /unlink 42132 (time 1746132955.96 total 220.44 last 213.19) - /unlink 44171 (time 1746132965.97 total 230.44 last 203.88) - /unlink 46828 (time 1746132975.97 total 240.44 last 265.67) - /unlink 48689 (time 1746132985.97 total 250.44 last 186.08) - /unlink 50000 (time 1746132992.07 total 256.54 last 214.90) - /unlink 52312 (time 1746133002.07 total 266.54 last 231.16) - /unlink 54907 (time 1746133012.07 total 276.54 last 259.49) - /unlink 57375 (time 1746133022.07 total 286.54 last 246.76) - /unlink 60000 (time 1746133031.76 total 296.23 last 271.02) - /unlink 61945 (time 1746133041.76 total 306.23 last 194.44) - /unlink 64058 (time 1746133051.77 total 316.24 last 211.18) total: 64768 /unlink in 319.48 seconds: 202.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1300 (time 1746133083.17 total 10.00 last 129.99) - open/close 2655 (time 1746133093.17 total 20.00 last 135.47) - open/close 4042 (time 1746133103.17 total 30.01 last 138.63) - open/close 5225 (time 1746133113.18 total 40.01 last 118.28) - open/close 6267 (time 1746133123.18 total 50.01 last 104.18) - open/close 7595 (time 1746133133.18 total 60.02 last 132.75) - open/close 8703 (time 1746133143.19 total 70.02 last 110.71) - open/close 9869 (time 1746133153.19 total 80.03 last 116.56) - open/close 10986 (time 1746133163.20 total 90.03 last 111.64) - open/close 12083 (time 1746133173.20 total 100.04 last 109.67) - open/close 13165 (time 1746133183.20 total 110.04 last 108.20) - open/close 14256 (time 1746133193.22 total 120.05 last 108.95) - open/close 15570 (time 1746133203.22 total 130.05 last 131.37) - open/close 16909 (time 1746133213.22 total 140.06 last 133.82) - open/close 18084 (time 1746133223.23 total 150.06 last 117.47) - open/close 19363 (time 1746133233.23 total 160.07 last 127.79) - open/close 20000 (time 1746133239.37 total 166.21 last 103.74) - open/close 21220 (time 1746133249.38 total 176.21 last 121.93) - open/close 22512 (time 1746133259.39 total 186.22 last 129.12) - open/close 23819 (time 1746133269.39 total 196.22 last 130.68) - open/close 24944 (time 1746133279.40 total 206.24 last 112.32) - open/close 26106 (time 1746133289.41 total 216.24 last 116.17) - open/close 27162 (time 1746133299.41 total 226.24 last 105.58) - open/close 28308 (time 1746133309.42 total 236.25 last 114.51) - open/close 29503 (time 1746133319.42 total 246.26 last 119.43) - open/close 30000 (time 1746133323.74 total 250.57 last 115.16) - open/close 31094 (time 1746133333.74 total 260.58 last 109.34) - open/close 32346 (time 1746133343.75 total 270.58 last 125.13) - open/close 33606 (time 1746133353.77 total 280.61 last 125.74) - open/close 34784 (time 1746133363.77 total 290.61 last 117.76) - open/close 35997 (time 1746133373.77 total 300.61 last 121.30) - open/close 37498 (time 1746133383.78 total 310.61 last 150.07) - open/close 38994 (time 1746133393.78 total 320.61 last 149.58) - open/close 40000 (time 1746133402.20 total 329.03 last 119.47) - open/close 41388 (time 1746133412.20 total 339.04 last 138.72) - open/close 42721 (time 1746133422.21 total 349.04 last 133.27) - open/close 44082 (time 1746133432.21 total 359.04 last 136.05) - open/close 45438 (time 1746133442.22 total 369.05 last 135.52) - open/close 46690 (time 1746133452.23 total 379.07 last 125.01) - open/close 47881 (time 1746133462.23 total 389.07 last 119.08) - open/close 49020 (time 1746133472.24 total 399.07 last 113.87) - open/close 50000 (time 1746133480.78 total 407.62 last 114.63) - open/close 51038 (time 1746133490.79 total 417.62 last 103.79) - open/close 52123 (time 1746133500.79 total 427.62 last 108.48) - open/close 53230 (time 1746133510.79 total 437.63 last 110.64) - open/close 54612 (time 1746133520.80 total 447.63 last 138.12) - open/close 55790 (time 1746133530.81 total 457.64 last 117.68) - open/close 56849 (time 1746133540.82 total 467.66 last 105.74) - open/close 58081 (time 1746133550.83 total 477.67 last 123.09) - open/close 59476 (time 1746133560.84 total 487.67 last 139.44) - open/close 60000 (time 1746133565.02 total 491.85 last 125.27) - open/close 61248 (time 1746133575.03 total 501.86 last 124.73) - open/close 62524 (time 1746133585.03 total 511.86 last 127.54) - open/close 63914 (time 1746133595.03 total 521.87 last 138.93) total: 64768 open/close in 527.73 seconds: 122.73 ops/second - /unlink 2043 (time 1746133612.47 total 10.00 last 204.25) - /unlink 4388 (time 1746133622.47 total 20.01 last 234.41) - /unlink 6567 (time 1746133632.47 total 30.01 last 217.84) - /unlink 8720 (time 1746133642.48 total 40.01 last 215.26) - /unlink 10000 (time 1746133648.88 total 46.41 last 199.95) - /unlink 11847 (time 1746133658.88 total 56.42 last 184.60) - /unlink 14114 (time 1746133668.88 total 66.42 last 226.69) - /unlink 16380 (time 1746133678.89 total 76.42 last 226.57) - /unlink 18635 (time 1746133688.89 total 86.42 last 225.45) - /unlink 20000 (time 1746133695.58 total 93.12 last 203.87) - /unlink 22085 (time 1746133705.59 total 103.12 last 208.39) - /unlink 23893 (time 1746133715.59 total 113.12 last 180.78) - /unlink 26005 (time 1746133725.59 total 123.13 last 211.15) - /unlink 28463 (time 1746133735.60 total 133.13 last 245.67) - /unlink 30000 (time 1746133743.56 total 141.10 last 192.94) - /unlink 32065 (time 1746133753.56 total 151.10 last 206.49) - /unlink 34070 (time 1746133763.56 total 161.10 last 200.48) - /unlink 36486 (time 1746133773.57 total 171.11 last 241.44) - /unlink 38738 (time 1746133783.57 total 181.11 last 225.18) - /unlink 40000 (time 1746133789.63 total 187.17 last 208.26) - /unlink 42363 (time 1746133799.63 total 197.17 last 236.30) - /unlink 44247 (time 1746133809.64 total 207.17 last 188.33) - /unlink 45955 (time 1746133819.64 total 217.17 last 170.77) - /unlink 47656 (time 1746133829.64 total 227.17 last 170.05) - /unlink 49407 (time 1746133839.64 total 237.17 last 175.09) - /unlink 50000 (time 1746133842.58 total 240.11 last 201.71) - /unlink 51844 (time 1746133852.59 total 250.12 last 184.31) - /unlink 53639 (time 1746133862.59 total 260.13 last 179.33) - /unlink 55736 (time 1746133872.59 total 270.13 last 209.69) - /unlink 58248 (time 1746133882.60 total 280.13 last 251.15) - /unlink 60000 (time 1746133890.73 total 288.27 last 215.30) - /unlink 61938 (time 1746133900.75 total 298.28 last 193.55) - /unlink 64237 (time 1746133910.76 total 308.29 last 229.73) total: 64768 /unlink in 310.79 seconds: 208.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2