== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:31:52 (1746156712) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) oleg407-server: oleg407-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg407-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg407-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg407-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1152 (time 1746156778.33 total 10.00 last 115.14) - open/close 2176 (time 1746156788.34 total 20.01 last 102.35) - open/close 3179 (time 1746156798.34 total 30.01 last 100.27) - open/close 4357 (time 1746156808.34 total 40.01 last 117.80) - open/close 5554 (time 1746156818.35 total 50.02 last 119.62) - open/close 6837 (time 1746156828.35 total 60.02 last 128.27) - open/close 8315 (time 1746156838.35 total 70.03 last 147.73) - open/close 9572 (time 1746156848.36 total 80.03 last 125.67) - open/close 10000 (time 1746156851.69 total 83.36 last 128.46) - open/close 11194 (time 1746156861.69 total 93.37 last 119.34) - open/close 12349 (time 1746156871.70 total 103.37 last 115.46) - open/close 13584 (time 1746156881.70 total 113.37 last 123.44) - open/close 14760 (time 1746156891.71 total 123.38 last 117.56) - open/close 16075 (time 1746156901.71 total 133.38 last 131.47) - open/close 17482 (time 1746156911.71 total 143.38 last 140.68) - open/close 19069 (time 1746156921.71 total 153.38 last 158.65) - open/close 20000 (time 1746156927.55 total 159.22 last 159.53) - open/close 21432 (time 1746156937.55 total 169.22 last 143.13) - open/close 22869 (time 1746156947.56 total 179.23 last 143.62) - open/close 24216 (time 1746156957.56 total 189.23 last 134.65) - open/close 25445 (time 1746156967.56 total 199.23 last 122.89) - open/close 26553 (time 1746156977.57 total 209.24 last 110.76) - open/close 27857 (time 1746156987.57 total 219.24 last 130.34) - open/close 29437 (time 1746156997.58 total 229.25 last 157.94) - open/close 30000 (time 1746157002.23 total 233.90 last 120.91) - open/close 31318 (time 1746157012.23 total 243.91 last 131.77) - open/close 32591 (time 1746157022.25 total 253.92 last 127.06) - open/close 33922 (time 1746157032.26 total 263.93 last 133.03) - open/close 35193 (time 1746157042.26 total 273.93 last 127.08) - open/close 36507 (time 1746157052.26 total 283.93 last 131.40) - open/close 37618 (time 1746157062.26 total 293.93 last 111.09) - open/close 38778 (time 1746157072.27 total 303.94 last 115.91) - open/close 40000 (time 1746157081.57 total 313.24 last 131.45) - open/close 41602 (time 1746157091.57 total 323.24 last 160.18) - open/close 43256 (time 1746157101.57 total 333.24 last 165.31) - open/close 44836 (time 1746157111.57 total 343.24 last 158.00) - open/close 46389 (time 1746157121.58 total 353.25 last 155.23) - open/close 47916 (time 1746157131.58 total 363.25 last 152.62) - open/close 49244 (time 1746157141.58 total 373.25 last 132.80) - open/close 50000 (time 1746157146.31 total 377.98 last 159.97) - open/close 51606 (time 1746157156.31 total 387.98 last 160.56) - open/close 53280 (time 1746157166.31 total 397.98 last 167.37) - open/close 54834 (time 1746157176.32 total 407.99 last 155.33) - open/close 56253 (time 1746157186.32 total 417.99 last 141.82) - open/close 57747 (time 1746157196.32 total 428.00 last 149.36) - open/close 59081 (time 1746157206.34 total 438.01 last 133.20) - open/close 60000 (time 1746157212.01 total 443.68 last 162.12) - open/close 61497 (time 1746157222.01 total 453.68 last 149.67) - open/close 63056 (time 1746157232.01 total 463.68 last 155.89) - open/close 64435 (time 1746157242.01 total 473.69 last 137.86) total: 64768 open/close in 476.08 seconds: 136.05 ops/second - /unlink 2633 (time 1746157255.79 total 10.00 last 263.28) - /unlink 5200 (time 1746157265.80 total 20.00 last 256.60) - /unlink 7707 (time 1746157275.80 total 30.01 last 250.65) - /unlink 10000 (time 1746157284.32 total 38.53 last 269.09) - /unlink 12701 (time 1746157294.32 total 48.53 last 270.09) - /unlink 14962 (time 1746157304.32 total 58.53 last 225.97) - /unlink 16596 (time 1746157314.33 total 68.54 last 163.31) - /unlink 18212 (time 1746157324.33 total 78.54 last 161.53) - /unlink 19821 (time 1746157334.34 total 88.55 last 160.83) - /unlink 21448 (time 1746157344.34 total 98.55 last 162.66) - /unlink 22948 (time 1746157354.35 total 108.55 last 149.93) - /unlink 24947 (time 1746157364.35 total 118.56 last 199.86) - /unlink 27375 (time 1746157374.35 total 128.56 last 242.78) - /unlink 29782 (time 1746157384.35 total 138.56 last 240.65) - /unlink 32537 (time 1746157394.35 total 148.56 last 275.49) - /unlink 34729 (time 1746157404.35 total 158.56 last 219.20) - /unlink 37014 (time 1746157414.35 total 168.56 last 228.45) - /unlink 39380 (time 1746157424.35 total 178.56 last 236.59) - /unlink 40000 (time 1746157426.69 total 180.90 last 265.26) - /unlink 42701 (time 1746157436.69 total 190.90 last 270.04) - /unlink 44892 (time 1746157446.70 total 200.91 last 219.03) - /unlink 47131 (time 1746157456.70 total 210.91 last 223.88) - /unlink 49733 (time 1746157466.70 total 220.91 last 260.07) - /unlink 52177 (time 1746157476.71 total 230.92 last 244.19) - /unlink 54350 (time 1746157486.72 total 240.93 last 217.15) - /unlink 56239 (time 1746157496.72 total 250.93 last 188.89) - /unlink 58263 (time 1746157506.72 total 260.93 last 202.34) - /unlink 60000 (time 1746157514.94 total 269.15 last 211.35) - /unlink 61930 (time 1746157524.95 total 279.16 last 192.87) - /unlink 63976 (time 1746157534.95 total 289.16 last 204.57) total: 64768 /unlink in 293.06 seconds: 221.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1293 (time 1746157567.83 total 10.00 last 129.24) - open/close 2511 (time 1746157577.84 total 20.01 last 121.70) - open/close 3664 (time 1746157587.84 total 30.02 last 115.25) - open/close 4974 (time 1746157597.84 total 40.02 last 130.96) - open/close 6444 (time 1746157607.85 total 50.02 last 146.94) - open/close 7680 (time 1746157617.85 total 60.03 last 123.53) - open/close 8807 (time 1746157627.86 total 70.03 last 112.66) - open/close 9972 (time 1746157637.86 total 80.04 last 116.47) - open/close 11160 (time 1746157647.87 total 90.04 last 118.74) - open/close 12345 (time 1746157657.87 total 100.04 last 118.50) - open/close 13550 (time 1746157667.87 total 110.05 last 120.41) - open/close 14801 (time 1746157677.88 total 120.05 last 125.03) - open/close 16023 (time 1746157687.88 total 130.06 last 122.15) - open/close 17166 (time 1746157697.89 total 140.06 last 114.25) - open/close 18392 (time 1746157707.89 total 150.06 last 122.59) - open/close 19643 (time 1746157717.89 total 160.06 last 125.10) - open/close 20000 (time 1746157720.92 total 163.10 last 117.54) - open/close 21200 (time 1746157730.93 total 173.10 last 119.95) - open/close 22335 (time 1746157740.94 total 183.11 last 113.41) - open/close 23553 (time 1746157750.94 total 193.11 last 121.79) - open/close 24734 (time 1746157760.94 total 203.12 last 118.06) - open/close 26032 (time 1746157770.94 total 213.12 last 129.78) - open/close 27417 (time 1746157780.95 total 223.12 last 138.45) - open/close 28698 (time 1746157790.95 total 233.13 last 128.06) - open/close 30000 (time 1746157800.80 total 242.97 last 132.24) - open/close 31219 (time 1746157810.80 total 252.97 last 121.88) - open/close 32952 (time 1746157820.80 total 262.97 last 173.29) - open/close 34565 (time 1746157830.80 total 272.98 last 161.25) - open/close 36229 (time 1746157840.80 total 282.98 last 166.36) - open/close 37605 (time 1746157850.81 total 292.98 last 137.57) - open/close 38832 (time 1746157860.81 total 302.99 last 122.65) - open/close 40000 (time 1746157869.26 total 311.44 last 138.17) - open/close 41625 (time 1746157879.27 total 321.44 last 162.42) - open/close 43435 (time 1746157889.27 total 331.45 last 180.93) - open/close 44900 (time 1746157899.27 total 341.45 last 146.48) - open/close 46524 (time 1746157909.28 total 351.46 last 162.32) - open/close 47838 (time 1746157919.28 total 361.46 last 131.38) - open/close 49415 (time 1746157929.28 total 371.46 last 157.69) - open/close 50000 (time 1746157932.74 total 374.92 last 169.17) - open/close 51303 (time 1746157942.74 total 384.92 last 130.26) - open/close 52518 (time 1746157952.75 total 394.92 last 121.44) - open/close 53750 (time 1746157962.76 total 404.93 last 123.08) - open/close 54962 (time 1746157972.76 total 414.94 last 121.15) - open/close 56105 (time 1746157982.77 total 424.95 last 114.18) - open/close 57370 (time 1746157992.77 total 434.95 last 126.50) - open/close 58589 (time 1746158002.79 total 444.97 last 121.69) - open/close 60000 (time 1746158011.21 total 453.39 last 167.50) - open/close 61689 (time 1746158021.21 total 463.39 last 168.90) - open/close 63446 (time 1746158031.22 total 473.39 last 175.64) total: 64768 open/close in 481.46 seconds: 134.52 ops/second - /unlink 3052 (time 1746158050.67 total 10.00 last 305.18) - /unlink 5804 (time 1746158060.67 total 20.00 last 275.19) - /unlink 8383 (time 1746158070.67 total 30.00 last 257.90) - /unlink 10000 (time 1746158078.48 total 37.80 last 207.21) - /unlink 12219 (time 1746158088.48 total 47.81 last 221.86) - /unlink 14291 (time 1746158098.48 total 57.81 last 207.11) - /unlink 16259 (time 1746158108.49 total 67.82 last 196.67) - /unlink 18375 (time 1746158118.49 total 77.82 last 211.53) - /unlink 20000 (time 1746158123.86 total 83.19 last 302.63) - /unlink 23108 (time 1746158133.87 total 93.19 last 310.78) - /unlink 25982 (time 1746158143.87 total 103.19 last 287.38) - /unlink 28431 (time 1746158153.87 total 113.19 last 244.89) - /unlink 30000 (time 1746158159.58 total 118.90 last 274.73) - /unlink 32294 (time 1746158169.58 total 128.90 last 229.37) - /unlink 34832 (time 1746158179.59 total 138.91 last 253.54) - /unlink 37141 (time 1746158189.59 total 148.91 last 230.90) - /unlink 39373 (time 1746158199.59 total 158.92 last 223.19) - /unlink 40000 (time 1746158201.89 total 161.22 last 272.56) - /unlink 42476 (time 1746158211.89 total 171.22 last 247.47) - /unlink 45548 (time 1746158221.90 total 181.22 last 307.14) - /unlink 48493 (time 1746158231.90 total 191.22 last 294.48) - /unlink 50000 (time 1746158238.39 total 197.71 last 232.26) - /unlink 51833 (time 1746158248.39 total 207.71 last 183.27) - /unlink 53822 (time 1746158258.39 total 217.72 last 198.82) - /unlink 56120 (time 1746158268.39 total 227.72 last 229.74) - /unlink 58687 (time 1746158278.39 total 237.72 last 256.67) - /unlink 60000 (time 1746158284.60 total 243.93 last 211.43) - /unlink 62201 (time 1746158294.61 total 253.93 last 220.10) - /unlink 64376 (time 1746158304.61 total 263.94 last 217.39) total: 64768 /unlink in 265.67 seconds: 243.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1484 (time 1746158334.56 total 10.00 last 148.38) - open/close 2983 (time 1746158344.56 total 20.00 last 149.85) - open/close 4459 (time 1746158354.57 total 30.01 last 147.49) - open/close 5717 (time 1746158364.57 total 40.02 last 125.74) - open/close 7057 (time 1746158374.57 total 50.02 last 133.96) - open/close 8444 (time 1746158384.58 total 60.02 last 138.67) - open/close 9818 (time 1746158394.58 total 70.02 last 137.39) - open/close 11062 (time 1746158404.58 total 80.03 last 124.35) - open/close 12349 (time 1746158414.58 total 90.03 last 128.66) - open/close 13838 (time 1746158424.59 total 100.03 last 148.83) - open/close 15262 (time 1746158434.59 total 110.03 last 142.40) - open/close 16508 (time 1746158444.59 total 120.04 last 124.58) - open/close 17837 (time 1746158454.59 total 130.04 last 132.90) - open/close 19229 (time 1746158464.59 total 140.04 last 139.16) - open/close 20000 (time 1746158470.16 total 145.60 last 138.54) - open/close 21505 (time 1746158480.17 total 155.61 last 150.40) - open/close 22958 (time 1746158490.17 total 165.61 last 145.28) - open/close 24173 (time 1746158500.17 total 175.61 last 121.47) - open/close 25400 (time 1746158510.18 total 185.62 last 122.60) - open/close 26631 (time 1746158520.18 total 195.63 last 123.04) - open/close 27893 (time 1746158530.18 total 205.63 last 126.19) - open/close 29169 (time 1746158540.19 total 215.63 last 127.56) - open/close 30000 (time 1746158546.64 total 222.08 last 128.81) - open/close 31390 (time 1746158556.64 total 232.09 last 138.94) - open/close 32705 (time 1746158566.64 total 242.09 last 131.46) - open/close 34201 (time 1746158576.65 total 252.10 last 149.51) - open/close 35579 (time 1746158586.65 total 262.10 last 137.74) - open/close 36936 (time 1746158596.66 total 272.10 last 135.65) - open/close 38434 (time 1746158606.66 total 282.10 last 149.79) - open/close 39930 (time 1746158616.66 total 292.11 last 149.56) - open/close 41616 (time 1746158626.67 total 302.12 last 168.40) - open/close 43186 (time 1746158636.68 total 312.12 last 156.98) - open/close 44603 (time 1746158646.68 total 322.13 last 141.62) - open/close 45898 (time 1746158656.68 total 332.13 last 129.47) - open/close 47173 (time 1746158666.69 total 342.13 last 127.45) - open/close 48444 (time 1746158676.69 total 352.14 last 127.06) - open/close 49785 (time 1746158686.70 total 362.15 last 133.98) - open/close 51257 (time 1746158696.70 total 372.15 last 147.18) - open/close 52725 (time 1746158706.73 total 382.18 last 146.38) - open/close 54048 (time 1746158716.73 total 392.18 last 132.28) - open/close 55287 (time 1746158726.73 total 402.18 last 123.88) - open/close 56476 (time 1746158736.74 total 412.19 last 118.82) - open/close 57641 (time 1746158746.74 total 422.19 last 116.48) - open/close 58867 (time 1746158756.75 total 432.19 last 122.55) - open/close 60000 (time 1746158766.70 total 442.15 last 113.79) - open/close 61223 (time 1746158776.71 total 452.15 last 122.24) - open/close 62598 (time 1746158786.71 total 462.15 last 137.49) - open/close 64069 (time 1746158796.72 total 472.17 last 146.93) total: 64768 open/close in 477.61 seconds: 135.61 ops/second - /unlink 2351 (time 1746158813.76 total 10.00 last 235.00) - /unlink 4432 (time 1746158823.76 total 20.01 last 208.02) - /unlink 6805 (time 1746158833.77 total 30.01 last 237.21) - /unlink 9064 (time 1746158843.77 total 40.01 last 225.85) - /unlink 10000 (time 1746158848.44 total 44.69 last 200.32) - /unlink 12081 (time 1746158858.44 total 54.69 last 208.09) - /unlink 14131 (time 1746158868.45 total 64.69 last 204.87) - /unlink 16529 (time 1746158878.46 total 74.70 last 239.61) - /unlink 18611 (time 1746158888.46 total 84.70 last 208.17) - /unlink 20000 (time 1746158894.03 total 90.27 last 249.37) - /unlink 22300 (time 1746158904.03 total 100.28 last 229.90) - /unlink 25362 (time 1746158914.04 total 110.28 last 306.08) - /unlink 28541 (time 1746158924.04 total 120.28 last 317.90) - /unlink 30000 (time 1746158929.40 total 125.64 last 272.24) - /unlink 32980 (time 1746158939.40 total 135.64 last 297.88) - /unlink 35719 (time 1746158949.40 total 145.65 last 273.87) - /unlink 38112 (time 1746158959.40 total 155.65 last 239.24) - /unlink 40000 (time 1746158967.02 total 163.27 last 247.84) - /unlink 42481 (time 1746158977.02 total 173.27 last 248.04) - /unlink 45499 (time 1746158987.03 total 183.27 last 301.61) - /unlink 48258 (time 1746158997.03 total 193.28 last 275.86) - /unlink 50000 (time 1746159003.69 total 199.93 last 261.80) - /unlink 52812 (time 1746159013.69 total 209.93 last 281.11) - /unlink 55631 (time 1746159023.69 total 219.94 last 281.84) - /unlink 58788 (time 1746159033.70 total 229.94 last 315.50) - /unlink 60000 (time 1746159036.65 total 232.90 last 410.12) - /unlink 64300 (time 1746159046.65 total 242.90 last 429.92) total: 64768 /unlink in 244.18 seconds: 265.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2203 (time 1746159074.88 total 10.00 last 220.28) - open/close 3725 (time 1746159084.88 total 20.00 last 152.16) - open/close 5441 (time 1746159094.89 total 30.01 last 171.48) - open/close 7503 (time 1746159104.89 total 40.01 last 206.16) - open/close 9102 (time 1746159114.89 total 50.01 last 159.89) - open/close 10000 (time 1746159120.35 total 55.47 last 164.55) - open/close 11369 (time 1746159130.36 total 65.48 last 136.73) - open/close 12624 (time 1746159140.37 total 75.49 last 125.46) - open/close 13947 (time 1746159150.37 total 85.49 last 132.21) - open/close 15371 (time 1746159160.38 total 95.50 last 142.33) - open/close 16999 (time 1746159170.38 total 105.50 last 162.80) - open/close 18576 (time 1746159180.39 total 115.50 last 157.59) - open/close 19861 (time 1746159190.39 total 125.51 last 128.47) - open/close 21468 (time 1746159200.39 total 135.51 last 160.67) - open/close 23226 (time 1746159210.40 total 145.52 last 175.66) - open/close 24798 (time 1746159220.40 total 155.52 last 157.20) - open/close 26392 (time 1746159230.40 total 165.52 last 159.33) - open/close 27913 (time 1746159240.40 total 175.52 last 152.08) - open/close 29615 (time 1746159250.41 total 185.53 last 170.13) - open/close 30000 (time 1746159252.94 total 188.06 last 151.85) - open/close 31258 (time 1746159262.95 total 198.07 last 125.71) - open/close 32741 (time 1746159272.95 total 208.07 last 148.24) - open/close 34418 (time 1746159282.96 total 218.07 last 167.69) - open/close 36118 (time 1746159292.96 total 228.08 last 169.97) - open/close 38232 (time 1746159302.96 total 238.08 last 211.38) - open/close 40000 (time 1746159312.23 total 247.35 last 190.75) - open/close 41705 (time 1746159322.23 total 257.35 last 170.39) - open/close 43905 (time 1746159332.23 total 267.35 last 219.99) - open/close 46249 (time 1746159342.24 total 277.36 last 234.23) - open/close 48136 (time 1746159352.24 total 287.36 last 188.70) - open/close 49752 (time 1746159362.24 total 297.36 last 161.58) - open/close 51451 (time 1746159372.25 total 307.36 last 169.86) - open/close 53049 (time 1746159382.25 total 317.37 last 159.73) - open/close 54941 (time 1746159392.25 total 327.37 last 189.16) - open/close 56533 (time 1746159402.25 total 337.37 last 159.15) - open/close 57850 (time 1746159412.26 total 347.38 last 131.68) - open/close 59505 (time 1746159422.26 total 357.38 last 165.50) - open/close 60000 (time 1746159425.05 total 360.17 last 177.32) - open/close 61654 (time 1746159435.05 total 370.17 last 165.36) - open/close 63315 (time 1746159445.05 total 380.17 last 166.07) total: 64768 open/close in 388.69 seconds: 166.63 ops/second - /unlink 3360 (time 1746159464.70 total 10.00 last 335.97) - /unlink 6286 (time 1746159474.71 total 20.00 last 292.56) - /unlink 8802 (time 1746159484.71 total 30.01 last 251.47) - /unlink 10000 (time 1746159489.00 total 34.29 last 279.44) - /unlink 12931 (time 1746159499.00 total 44.29 last 293.09) - /unlink 15440 (time 1746159509.00 total 54.30 last 250.86) - /unlink 17906 (time 1746159519.00 total 64.30 last 246.55) - /unlink 20000 (time 1746159527.94 total 73.24 last 234.21) - /unlink 22402 (time 1746159537.94 total 83.24 last 240.20) - /unlink 24948 (time 1746159547.94 total 93.24 last 254.59) - /unlink 27369 (time 1746159557.95 total 103.24 last 242.05) - /unlink 30000 (time 1746159566.47 total 111.77 last 308.58) - /unlink 32548 (time 1746159576.47 total 121.77 last 254.80) - /unlink 34712 (time 1746159586.47 total 131.77 last 216.37) - /unlink 37836 (time 1746159596.47 total 141.77 last 312.38) - /unlink 40000 (time 1746159604.22 total 149.51 last 279.51) - /unlink 42613 (time 1746159614.22 total 159.51 last 261.27) - /unlink 45053 (time 1746159624.22 total 169.51 last 243.99) - /unlink 48849 (time 1746159634.22 total 179.52 last 379.55) - /unlink 50000 (time 1746159636.68 total 181.98 last 466.88) - /unlink 52983 (time 1746159646.69 total 191.98 last 298.23) - /unlink 56061 (time 1746159656.69 total 201.98 last 307.74) - /unlink 59265 (time 1746159666.69 total 211.99 last 320.31) - /unlink 60000 (time 1746159669.09 total 214.39 last 306.22) - /unlink 63330 (time 1746159679.09 total 224.39 last 332.93) total: 64768 /unlink in 227.41 seconds: 284.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1748 (time 1746159711.04 total 10.00 last 174.75) - open/close 3267 (time 1746159721.05 total 20.01 last 151.86) - open/close 5384 (time 1746159731.05 total 30.01 last 211.66) - open/close 6868 (time 1746159741.05 total 40.01 last 148.39) - open/close 8176 (time 1746159751.05 total 50.01 last 130.77) - open/close 10000 (time 1746159758.95 total 57.91 last 230.88) - open/close 11918 (time 1746159768.95 total 67.91 last 191.74) - open/close 13585 (time 1746159778.96 total 77.92 last 166.66) - open/close 15147 (time 1746159788.96 total 87.92 last 156.14) - open/close 17353 (time 1746159798.96 total 97.92 last 220.55) - open/close 19468 (time 1746159808.97 total 107.93 last 211.42) - open/close 20000 (time 1746159812.08 total 111.04 last 170.93) - open/close 21552 (time 1746159822.09 total 121.05 last 155.02) - open/close 23328 (time 1746159832.09 total 131.05 last 177.58) - open/close 24818 (time 1746159842.10 total 141.06 last 148.92) - open/close 26547 (time 1746159852.10 total 151.06 last 172.89) - open/close 28149 (time 1746159862.10 total 161.06 last 160.19) - open/close 29645 (time 1746159872.10 total 171.06 last 149.58) - open/close 30000 (time 1746159874.36 total 173.32 last 156.99) - open/close 31391 (time 1746159884.36 total 183.32 last 139.08) - open/close 32770 (time 1746159894.37 total 193.33 last 137.85) - open/close 33933 (time 1746159904.37 total 203.33 last 116.28) - open/close 35199 (time 1746159914.38 total 213.34 last 126.41) - open/close 36385 (time 1746159924.39 total 223.35 last 118.47) - open/close 37707 (time 1746159934.40 total 233.36 last 132.15) - open/close 39045 (time 1746159944.40 total 243.36 last 133.77) - open/close 40000 (time 1746159951.67 total 250.63 last 131.30) - open/close 41265 (time 1746159961.68 total 260.64 last 126.45) - open/close 42527 (time 1746159971.68 total 270.64 last 126.16) - open/close 43655 (time 1746159981.68 total 280.64 last 112.74) - open/close 44931 (time 1746159991.69 total 290.65 last 127.51) - open/close 46172 (time 1746160001.70 total 300.66 last 124.04) - open/close 47434 (time 1746160011.70 total 310.66 last 126.11) - open/close 48756 (time 1746160021.71 total 320.67 last 132.10) - open/close 49967 (time 1746160031.71 total 330.67 last 121.06) - open/close 51214 (time 1746160041.72 total 340.68 last 124.65) - open/close 52328 (time 1746160051.72 total 350.68 last 111.33) - open/close 53492 (time 1746160061.73 total 360.69 last 116.38) - open/close 54688 (time 1746160071.73 total 370.69 last 119.54) - open/close 55817 (time 1746160081.74 total 380.70 last 112.82) - open/close 57042 (time 1746160091.74 total 390.70 last 122.45) - open/close 58290 (time 1746160101.75 total 400.71 last 124.75) - open/close 59629 (time 1746160111.75 total 410.71 last 133.89) - open/close 60000 (time 1746160115.00 total 413.96 last 113.99) - open/close 61227 (time 1746160125.00 total 423.96 last 122.70) - open/close 62426 (time 1746160135.00 total 433.96 last 119.90) - open/close 63794 (time 1746160145.01 total 443.97 last 136.71) total: 64768 open/close in 450.92 seconds: 143.63 ops/second - /unlink 2158 (time 1746160163.42 total 10.00 last 215.70) - /unlink 4192 (time 1746160173.42 total 20.01 last 203.35) - /unlink 5918 (time 1746160183.42 total 30.01 last 172.58) - /unlink 7911 (time 1746160193.42 total 40.01 last 199.22) - /unlink 9853 (time 1746160203.42 total 50.01 last 194.19) - /unlink 12184 (time 1746160213.42 total 60.01 last 233.10) - /unlink 14249 (time 1746160223.43 total 70.01 last 206.47) - /unlink 16363 (time 1746160233.43 total 80.02 last 211.37) - /unlink 18208 (time 1746160243.43 total 90.02 last 184.42) - /unlink 20000 (time 1746160252.24 total 98.83 last 203.51) - /unlink 22758 (time 1746160262.24 total 108.83 last 275.73) - /unlink 25231 (time 1746160272.24 total 118.83 last 247.29) - /unlink 28138 (time 1746160282.24 total 128.83 last 290.64) - /unlink 30000 (time 1746160290.29 total 136.88 last 231.39) - /unlink 32421 (time 1746160300.29 total 146.88 last 242.09) - /unlink 34730 (time 1746160310.29 total 156.88 last 230.85) - /unlink 36813 (time 1746160320.29 total 166.88 last 208.29) - /unlink 38971 (time 1746160330.29 total 176.88 last 215.78) - /unlink 40000 (time 1746160334.69 total 181.28 last 234.07) - /unlink 42945 (time 1746160344.69 total 191.28 last 294.48) - /unlink 45445 (time 1746160354.69 total 201.28 last 249.94) - /unlink 47995 (time 1746160364.69 total 211.28 last 254.94) - /unlink 50000 (time 1746160372.28 total 218.86 last 264.42) - /unlink 53122 (time 1746160382.28 total 228.87 last 312.15) - /unlink 57366 (time 1746160392.28 total 238.87 last 424.30) - /unlink 59753 (time 1746160402.29 total 248.88 last 238.54) - /unlink 62183 (time 1746160412.29 total 258.88 last 242.86) total: 64768 /unlink in 266.83 seconds: 242.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2