== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:00:21 (1746126021) Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2623 (time 1746126060.52 total 10.00 last 262.28) - open/close 5242 (time 1746126070.52 total 20.00 last 261.83) - open/close 7627 (time 1746126080.52 total 30.01 last 238.42) - open/close 9024 (time 1746126090.53 total 40.02 last 139.56) - open/close 9767 (time 1746126100.55 total 50.03 last 74.18) - open/close 10000 (time 1746126104.24 total 53.72 last 63.13) - open/close 10701 (time 1746126114.25 total 63.73 last 70.03) - open/close 11400 (time 1746126124.25 total 73.74 last 69.86) - open/close 12249 (time 1746126134.26 total 83.74 last 84.87) - open/close 13269 (time 1746126144.26 total 93.75 last 101.95) - open/close 14338 (time 1746126154.27 total 103.75 last 106.83) - open/close 15491 (time 1746126164.27 total 113.76 last 115.26) - open/close 16746 (time 1746126174.28 total 123.76 last 125.47) - open/close 17906 (time 1746126184.28 total 133.77 last 115.94) - open/close 19151 (time 1746126194.28 total 143.77 last 124.47) - open/close 20000 (time 1746126199.39 total 148.87 last 166.40) - open/close 21543 (time 1746126209.39 total 158.87 last 154.24) - open/close 23185 (time 1746126219.40 total 168.88 last 164.06) - open/close 24527 (time 1746126229.41 total 178.89 last 134.09) - open/close 25840 (time 1746126239.41 total 188.89 last 131.25) - open/close 27403 (time 1746126249.42 total 198.90 last 156.19) - open/close 28680 (time 1746126259.43 total 208.92 last 127.49) - open/close 29842 (time 1746126269.44 total 218.92 last 116.14) - open/close 30953 (time 1746126279.46 total 228.95 last 110.84) - open/close 32019 (time 1746126289.47 total 238.96 last 106.51) - open/close 33081 (time 1746126299.47 total 248.96 last 106.16) - open/close 34398 (time 1746126309.49 total 258.97 last 131.52) - open/close 35646 (time 1746126319.49 total 268.98 last 124.73) - open/close 36963 (time 1746126329.50 total 278.98 last 131.62) - open/close 38279 (time 1746126339.50 total 288.99 last 131.54) - open/close 39686 (time 1746126349.50 total 298.99 last 140.70) - open/close 40000 (time 1746126351.60 total 301.09 last 149.70) - open/close 41388 (time 1746126361.60 total 311.09 last 138.76) - open/close 42977 (time 1746126371.61 total 321.09 last 158.84) - open/close 44543 (time 1746126381.62 total 331.10 last 156.42) - open/close 45906 (time 1746126391.62 total 341.11 last 136.25) - open/close 47537 (time 1746126401.64 total 351.12 last 162.84) - open/close 49495 (time 1746126411.64 total 361.13 last 195.76) - open/close 50000 (time 1746126414.70 total 364.18 last 165.17) - open/close 51323 (time 1746126424.71 total 374.19 last 132.19) - open/close 52828 (time 1746126434.71 total 384.19 last 150.48) - open/close 54465 (time 1746126444.71 total 394.20 last 163.66) - open/close 56061 (time 1746126454.71 total 404.20 last 159.58) - open/close 57591 (time 1746126464.72 total 414.20 last 152.95) - open/close 58939 (time 1746126474.72 total 424.20 last 134.76) - open/close 60000 (time 1746126482.64 total 432.12 last 134.01) - open/close 61457 (time 1746126492.64 total 442.12 last 145.67) - open/close 62674 (time 1746126502.64 total 452.13 last 121.65) - open/close 63836 (time 1746126512.64 total 462.13 last 116.18) total: 64768 open/close in 470.58 seconds: 137.64 ops/second - /unlink 2138 (time 1746126532.85 total 10.00 last 213.74) - /unlink 4175 (time 1746126542.86 total 20.01 last 203.58) - /unlink 6253 (time 1746126552.86 total 30.01 last 207.71) - /unlink 8378 (time 1746126562.86 total 40.02 last 212.42) - /unlink 10000 (time 1746126569.02 total 46.17 last 263.52) - /unlink 12551 (time 1746126579.02 total 56.17 last 255.05) - /unlink 14615 (time 1746126589.02 total 66.18 last 206.35) - /unlink 17228 (time 1746126599.03 total 76.18 last 261.21) - /unlink 19639 (time 1746126609.03 total 86.18 last 241.10) - /unlink 22408 (time 1746126619.03 total 96.18 last 276.90) - /unlink 24830 (time 1746126629.03 total 106.18 last 242.13) - /unlink 27213 (time 1746126639.03 total 116.18 last 238.29) - /unlink 29970 (time 1746126649.03 total 126.18 last 275.67) - /unlink 32698 (time 1746126659.03 total 136.18 last 272.76) - /unlink 35535 (time 1746126669.04 total 146.19 last 283.64) - /unlink 38291 (time 1746126679.04 total 156.19 last 275.57) - /unlink 40000 (time 1746126686.05 total 163.20 last 243.76) - /unlink 42494 (time 1746126696.05 total 173.20 last 249.38) - /unlink 45242 (time 1746126706.05 total 183.20 last 274.75) - /unlink 48128 (time 1746126716.05 total 193.20 last 288.56) - /unlink 50000 (time 1746126723.13 total 200.29 last 264.28) - /unlink 52550 (time 1746126733.14 total 210.29 last 254.98) - /unlink 54992 (time 1746126743.14 total 220.29 last 244.18) - /unlink 57519 (time 1746126753.14 total 230.29 last 252.63) - /unlink 60000 (time 1746126763.09 total 240.24 last 249.35) - /unlink 62105 (time 1746126773.10 total 250.25 last 210.37) - /unlink 63910 (time 1746126783.10 total 260.25 last 180.47) total: 64768 /unlink in 264.10 seconds: 245.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1241 (time 1746126813.96 total 10.01 last 124.00) - open/close 2466 (time 1746126823.96 total 20.01 last 122.49) - open/close 3779 (time 1746126833.96 total 30.01 last 131.29) - open/close 5403 (time 1746126843.96 total 40.01 last 162.37) - open/close 6784 (time 1746126853.96 total 50.01 last 138.07) - open/close 8025 (time 1746126863.98 total 60.03 last 123.88) - open/close 9370 (time 1746126874.00 total 70.04 last 134.34) - open/close 10000 (time 1746126879.13 total 75.18 last 122.74) - open/close 11309 (time 1746126889.13 total 85.18 last 130.87) - open/close 12741 (time 1746126899.14 total 95.19 last 143.08) - open/close 14357 (time 1746126909.14 total 105.19 last 161.52) - open/close 15860 (time 1746126919.14 total 115.19 last 150.27) - open/close 17610 (time 1746126929.15 total 125.20 last 174.85) - open/close 19466 (time 1746126939.16 total 135.20 last 185.55) - open/close 20000 (time 1746126942.26 total 138.31 last 172.22) - open/close 21634 (time 1746126952.26 total 148.31 last 163.37) - open/close 23230 (time 1746126962.26 total 158.31 last 159.58) - open/close 24841 (time 1746126972.27 total 168.32 last 160.94) - open/close 26346 (time 1746126982.27 total 178.32 last 150.43) - open/close 27673 (time 1746126992.28 total 188.33 last 132.64) - open/close 29025 (time 1746127002.29 total 198.34 last 135.08) - open/close 30000 (time 1746127009.70 total 205.74 last 131.60) - open/close 31409 (time 1746127019.70 total 215.75 last 140.86) - open/close 32716 (time 1746127029.70 total 225.75 last 130.65) - open/close 34021 (time 1746127039.70 total 235.75 last 130.46) - open/close 35319 (time 1746127049.71 total 245.76 last 129.75) - open/close 36485 (time 1746127059.71 total 255.76 last 116.57) - open/close 37796 (time 1746127069.71 total 265.76 last 131.06) - open/close 39339 (time 1746127079.72 total 275.76 last 154.28) - open/close 40000 (time 1746127084.11 total 280.16 last 150.23) - open/close 41488 (time 1746127094.13 total 290.18 last 148.62) - open/close 42789 (time 1746127104.13 total 300.18 last 130.08) - open/close 44305 (time 1746127114.13 total 310.18 last 151.58) - open/close 45646 (time 1746127124.13 total 320.18 last 134.09) - open/close 47063 (time 1746127134.13 total 330.18 last 141.66) - open/close 48392 (time 1746127144.14 total 340.18 last 132.86) - open/close 49800 (time 1746127154.14 total 350.19 last 140.77) - open/close 51271 (time 1746127164.14 total 360.19 last 147.09) - open/close 52768 (time 1746127174.14 total 370.19 last 149.62) - open/close 54174 (time 1746127184.15 total 380.19 last 140.58) - open/close 55746 (time 1746127194.15 total 390.20 last 157.19) - open/close 57571 (time 1746127204.15 total 400.20 last 182.50) - open/close 59229 (time 1746127214.15 total 410.20 last 165.78) - open/close 60000 (time 1746127219.24 total 415.29 last 151.33) - open/close 61585 (time 1746127229.25 total 425.30 last 158.39) - open/close 63090 (time 1746127239.25 total 435.30 last 150.45) total: 64768 open/close in 443.51 seconds: 146.04 ops/second - /unlink 3549 (time 1746127258.62 total 10.00 last 354.86) - /unlink 6536 (time 1746127268.62 total 20.00 last 298.64) - /unlink 9760 (time 1746127278.62 total 30.01 last 322.26) - /unlink 12901 (time 1746127288.62 total 40.01 last 314.09) - /unlink 15448 (time 1746127298.63 total 50.01 last 254.58) - /unlink 17811 (time 1746127308.63 total 60.01 last 236.27) - /unlink 20000 (time 1746127315.88 total 67.27 last 301.76) - /unlink 22689 (time 1746127325.88 total 77.27 last 268.90) - /unlink 25346 (time 1746127335.89 total 87.27 last 265.66) - /unlink 28221 (time 1746127345.89 total 97.27 last 287.45) - /unlink 30000 (time 1746127353.31 total 104.69 last 239.67) - /unlink 32532 (time 1746127363.31 total 114.70 last 253.16) - /unlink 34996 (time 1746127373.31 total 124.70 last 246.36) - /unlink 37612 (time 1746127383.32 total 134.70 last 261.46) - /unlink 40000 (time 1746127392.04 total 143.42 last 273.95) - /unlink 42956 (time 1746127402.04 total 153.42 last 295.50) - /unlink 45693 (time 1746127412.04 total 163.42 last 273.69) - /unlink 48305 (time 1746127422.04 total 173.42 last 261.19) - /unlink 50000 (time 1746127428.58 total 179.97 last 259.06) - /unlink 52213 (time 1746127438.59 total 189.97 last 221.20) - /unlink 54128 (time 1746127448.59 total 199.97 last 191.42) - /unlink 56301 (time 1746127458.59 total 209.98 last 217.27) - /unlink 58635 (time 1746127468.59 total 219.98 last 233.39) - /unlink 60000 (time 1746127474.19 total 225.58 last 243.78) - /unlink 62492 (time 1746127484.19 total 235.58 last 249.17) total: 64768 /unlink in 244.54 seconds: 264.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1631 (time 1746127520.80 total 10.01 last 162.97) - open/close 3204 (time 1746127530.81 total 20.01 last 157.26) - open/close 5181 (time 1746127540.81 total 30.02 last 197.60) - open/close 6552 (time 1746127550.81 total 40.02 last 137.07) - open/close 7855 (time 1746127560.82 total 50.02 last 130.21) - open/close 9182 (time 1746127570.82 total 60.03 last 132.66) - open/close 10000 (time 1746127578.07 total 67.28 last 112.83) - open/close 11330 (time 1746127588.08 total 77.28 last 132.96) - open/close 12663 (time 1746127598.08 total 87.28 last 133.27) - open/close 13934 (time 1746127608.08 total 97.28 last 127.09) - open/close 15292 (time 1746127618.08 total 107.29 last 135.77) - open/close 16659 (time 1746127628.08 total 117.29 last 136.68) - open/close 17968 (time 1746127638.08 total 127.29 last 130.90) - open/close 19259 (time 1746127648.09 total 137.29 last 129.07) - open/close 20000 (time 1746127653.98 total 143.18 last 125.78) - open/close 21288 (time 1746127663.99 total 153.19 last 128.67) - open/close 22535 (time 1746127673.99 total 163.19 last 124.68) - open/close 23725 (time 1746127683.99 total 173.20 last 118.96) - open/close 24892 (time 1746127694.00 total 183.20 last 116.62) - open/close 26285 (time 1746127704.01 total 193.21 last 139.20) - open/close 28136 (time 1746127714.01 total 203.21 last 185.09) - open/close 29564 (time 1746127724.01 total 213.21 last 142.73) - open/close 30000 (time 1746127726.93 total 216.13 last 149.57) - open/close 31576 (time 1746127736.93 total 226.13 last 157.56) - open/close 33301 (time 1746127746.93 total 236.13 last 172.45) - open/close 35066 (time 1746127756.93 total 246.14 last 176.44) - open/close 36605 (time 1746127766.94 total 256.14 last 153.79) - open/close 37987 (time 1746127776.95 total 266.15 last 138.14) - open/close 39376 (time 1746127786.95 total 276.15 last 138.85) - open/close 40000 (time 1746127792.07 total 281.27 last 121.90) - open/close 41070 (time 1746127802.07 total 291.28 last 106.93) - open/close 42297 (time 1746127812.08 total 301.28 last 122.67) - open/close 43442 (time 1746127822.08 total 311.29 last 114.44) - open/close 44658 (time 1746127832.10 total 321.31 last 121.35) - open/close 45818 (time 1746127842.11 total 331.31 last 115.90) - open/close 47123 (time 1746127852.11 total 341.32 last 130.46) - open/close 48268 (time 1746127862.12 total 351.32 last 114.46) - open/close 49660 (time 1746127872.12 total 361.33 last 139.10) - open/close 51145 (time 1746127882.13 total 371.33 last 148.46) - open/close 52493 (time 1746127892.14 total 381.34 last 134.67) - open/close 53806 (time 1746127902.14 total 391.35 last 131.21) - open/close 55165 (time 1746127912.15 total 401.35 last 135.87) - open/close 56487 (time 1746127922.15 total 411.35 last 132.14) - open/close 57756 (time 1746127932.16 total 421.36 last 126.83) - open/close 58915 (time 1746127942.16 total 431.36 last 115.85) - open/close 60000 (time 1746127950.52 total 439.72 last 129.82) - open/close 61294 (time 1746127960.52 total 449.72 last 129.40) - open/close 62558 (time 1746127970.52 total 459.73 last 126.33) - open/close 63970 (time 1746127980.53 total 469.73 last 141.15) total: 64768 open/close in 474.65 seconds: 136.45 ops/second - /unlink 2102 (time 1746127996.83 total 10.00 last 210.13) - /unlink 4108 (time 1746128006.84 total 20.01 last 200.52) - /unlink 6123 (time 1746128016.84 total 30.01 last 201.38) - /unlink 8134 (time 1746128026.85 total 40.02 last 201.00) - /unlink 10000 (time 1746128035.98 total 49.15 last 204.36) - /unlink 11936 (time 1746128045.98 total 59.15 last 193.54) - /unlink 13903 (time 1746128055.98 total 69.16 last 196.65) - /unlink 15994 (time 1746128065.99 total 79.16 last 209.06) - /unlink 18210 (time 1746128075.99 total 89.16 last 221.48) - /unlink 20000 (time 1746128084.51 total 97.68 last 210.16) - /unlink 22015 (time 1746128094.51 total 107.68 last 201.40) - /unlink 23655 (time 1746128104.51 total 117.69 last 163.99) - /unlink 25264 (time 1746128114.52 total 127.69 last 160.78) - /unlink 26810 (time 1746128124.52 total 137.69 last 154.58) - /unlink 28442 (time 1746128134.53 total 147.70 last 163.10) - /unlink 30000 (time 1746128143.81 total 156.98 last 167.90) - /unlink 32052 (time 1746128153.81 total 166.98 last 205.16) - /unlink 34209 (time 1746128163.81 total 176.98 last 215.64) - /unlink 36124 (time 1746128173.82 total 186.99 last 191.42) - /unlink 38533 (time 1746128183.82 total 196.99 last 240.81) - /unlink 40000 (time 1746128189.52 total 202.70 last 257.21) - /unlink 41977 (time 1746128199.53 total 212.70 last 197.63) - /unlink 43893 (time 1746128209.53 total 222.70 last 191.58) - /unlink 45927 (time 1746128219.53 total 232.70 last 203.35) - /unlink 48643 (time 1746128229.53 total 242.71 last 271.50) - /unlink 50000 (time 1746128234.23 total 247.40 last 289.21) - /unlink 53183 (time 1746128244.23 total 257.40 last 318.19) - /unlink 56783 (time 1746128254.23 total 267.40 last 359.94) - /unlink 60000 (time 1746128264.04 total 277.21 last 328.00) - /unlink 63361 (time 1746128274.04 total 287.21 last 335.97) total: 64768 /unlink in 292.60 seconds: 221.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1863 (time 1746128310.80 total 10.01 last 186.20) - open/close 3363 (time 1746128320.80 total 20.01 last 149.92) - open/close 5110 (time 1746128330.81 total 30.01 last 174.65) - open/close 6762 (time 1746128340.81 total 40.01 last 165.20) - open/close 8235 (time 1746128350.81 total 50.02 last 147.27) - open/close 9672 (time 1746128360.81 total 60.02 last 143.63) - open/close 10000 (time 1746128363.60 total 62.81 last 117.72) - open/close 11422 (time 1746128373.61 total 72.81 last 142.10) - open/close 12646 (time 1746128383.61 total 82.82 last 122.36) - open/close 13785 (time 1746128393.61 total 92.82 last 113.89) - open/close 14918 (time 1746128403.62 total 102.82 last 113.24) - open/close 16133 (time 1746128413.62 total 112.83 last 121.41) - open/close 17343 (time 1746128423.63 total 122.84 last 120.95) - open/close 18512 (time 1746128433.63 total 132.84 last 116.86) - open/close 19699 (time 1746128443.63 total 142.84 last 118.67) - open/close 20000 (time 1746128446.07 total 145.28 last 123.59) - open/close 21083 (time 1746128456.07 total 155.28 last 108.27) - open/close 22123 (time 1746128466.08 total 165.29 last 103.86) - open/close 23143 (time 1746128476.09 total 175.29 last 101.99) - open/close 24303 (time 1746128486.09 total 185.30 last 115.92) - open/close 25317 (time 1746128496.10 total 195.31 last 101.30) - open/close 26462 (time 1746128506.13 total 205.34 last 114.22) - open/close 27446 (time 1746128516.13 total 215.34 last 98.38) - open/close 28400 (time 1746128526.14 total 225.34 last 95.33) - open/close 29428 (time 1746128536.14 total 235.35 last 102.76) - open/close 30000 (time 1746128541.61 total 240.82 last 104.51) - open/close 31086 (time 1746128551.62 total 250.83 last 108.55) - open/close 32177 (time 1746128561.63 total 260.83 last 109.03) - open/close 33211 (time 1746128571.63 total 270.84 last 103.37) - open/close 34252 (time 1746128581.64 total 280.85 last 103.98) - open/close 35336 (time 1746128591.65 total 290.86 last 108.32) - open/close 36252 (time 1746128601.65 total 300.86 last 91.57) - open/close 37194 (time 1746128611.66 total 310.87 last 94.07) - open/close 38130 (time 1746128621.67 total 320.87 last 93.59) - open/close 39070 (time 1746128631.67 total 330.88 last 93.94) - open/close 39980 (time 1746128641.68 total 340.89 last 90.94) - open/close 40986 (time 1746128651.69 total 350.90 last 100.49) - open/close 41913 (time 1746128661.69 total 360.90 last 92.68) - open/close 42851 (time 1746128671.69 total 370.90 last 93.79) - open/close 43812 (time 1746128681.70 total 380.90 last 96.05) - open/close 44761 (time 1746128691.70 total 390.90 last 94.89) - open/close 45693 (time 1746128701.72 total 400.93 last 92.99) - open/close 46624 (time 1746128711.73 total 410.94 last 92.99) - open/close 47615 (time 1746128721.74 total 420.94 last 99.04) - open/close 48584 (time 1746128731.75 total 430.96 last 96.74) - open/close 49481 (time 1746128741.76 total 440.97 last 89.61) - open/close 50000 (time 1746128747.73 total 446.94 last 87.00) - open/close 50956 (time 1746128757.74 total 456.95 last 95.45) - open/close 52050 (time 1746128767.75 total 466.95 last 109.38) - open/close 53222 (time 1746128777.76 total 476.96 last 117.09) - open/close 54316 (time 1746128787.76 total 486.97 last 109.35) - open/close 55415 (time 1746128797.76 total 496.97 last 109.86) - open/close 56637 (time 1746128807.77 total 506.97 last 122.18) - open/close 57864 (time 1746128817.77 total 516.97 last 122.70) - open/close 58884 (time 1746128827.77 total 526.98 last 101.91) - open/close 60000 (time 1746128837.02 total 536.23 last 120.68) - open/close 61179 (time 1746128847.03 total 546.23 last 117.85) - open/close 62313 (time 1746128857.03 total 556.23 last 113.39) - open/close 63534 (time 1746128867.03 total 566.24 last 122.06) - open/close 64647 (time 1746128877.04 total 576.24 last 111.23) total: 64768 open/close in 577.27 seconds: 112.20 ops/second - /unlink 2067 (time 1746128889.88 total 10.00 last 206.69) - /unlink 3718 (time 1746128899.89 total 20.00 last 165.06) - /unlink 5422 (time 1746128909.89 total 30.00 last 170.38) - /unlink 7166 (time 1746128919.89 total 40.01 last 174.39) - /unlink 8886 (time 1746128929.90 total 50.01 last 171.87) - /unlink 10000 (time 1746128936.44 total 56.55 last 170.29) - /unlink 11767 (time 1746128946.44 total 66.56 last 176.64) - /unlink 13558 (time 1746128956.44 total 76.56 last 179.08) - /unlink 15160 (time 1746128966.45 total 86.56 last 160.12) - /unlink 16801 (time 1746128976.45 total 96.56 last 164.08) - /unlink 18807 (time 1746128986.45 total 106.56 last 200.60) - /unlink 20000 (time 1746128992.57 total 112.69 last 194.80) - /unlink 21932 (time 1746129002.57 total 122.69 last 193.17) - /unlink 24027 (time 1746129012.58 total 132.70 last 209.40) - /unlink 26008 (time 1746129022.58 total 142.70 last 198.06) - /unlink 27791 (time 1746129032.59 total 152.71 last 178.11) - /unlink 29607 (time 1746129042.59 total 162.71 last 181.57) - /unlink 30000 (time 1746129044.63 total 164.75 last 192.79) - /unlink 31821 (time 1746129054.64 total 174.75 last 182.02) - /unlink 33920 (time 1746129064.64 total 184.76 last 209.81) - /unlink 35820 (time 1746129074.64 total 194.76 last 189.99) - /unlink 37850 (time 1746129084.64 total 204.76 last 202.93) - /unlink 39787 (time 1746129094.65 total 214.76 last 193.64) - /unlink 41780 (time 1746129104.65 total 224.77 last 199.17) - /unlink 43649 (time 1746129114.65 total 234.77 last 186.90) - /unlink 45319 (time 1746129124.66 total 244.77 last 166.96) - /unlink 47116 (time 1746129134.66 total 254.78 last 179.64) - /unlink 49046 (time 1746129144.66 total 264.78 last 192.97) - /unlink 50000 (time 1746129148.72 total 268.84 last 235.06) - /unlink 52406 (time 1746129158.73 total 278.84 last 240.47) - /unlink 54875 (time 1746129168.73 total 288.85 last 246.83) - /unlink 57296 (time 1746129178.74 total 298.85 last 241.94) - /unlink 59134 (time 1746129188.75 total 308.86 last 183.60) - /unlink 60000 (time 1746129192.91 total 313.03 last 208.02) - /unlink 62307 (time 1746129202.91 total 323.03 last 230.65) total: 64768 /unlink in 330.85 seconds: 195.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1476 (time 1746129240.91 total 10.00 last 147.58) - open/close 2982 (time 1746129250.92 total 20.00 last 150.55) - open/close 4468 (time 1746129260.92 total 30.01 last 148.50) - open/close 5804 (time 1746129270.93 total 40.01 last 133.58) - open/close 7131 (time 1746129280.93 total 50.02 last 132.67) - open/close 8676 (time 1746129290.93 total 60.02 last 154.43) - open/close 10000 (time 1746129300.66 total 69.75 last 136.12) - open/close 11521 (time 1746129310.66 total 79.75 last 152.09) - open/close 13063 (time 1746129320.66 total 89.75 last 154.15) - open/close 14653 (time 1746129330.67 total 99.76 last 158.92) - open/close 16196 (time 1746129340.67 total 109.76 last 154.22) - open/close 17819 (time 1746129350.68 total 119.77 last 162.23) - open/close 19371 (time 1746129360.69 total 129.77 last 155.10) - open/close 20000 (time 1746129365.18 total 134.26 last 140.05) - open/close 21215 (time 1746129375.18 total 144.27 last 121.42) - open/close 22722 (time 1746129385.19 total 154.28 last 150.61) - open/close 23996 (time 1746129395.20 total 164.28 last 127.31) - open/close 25293 (time 1746129405.20 total 174.28 last 129.69) - open/close 26610 (time 1746129415.20 total 184.29 last 131.65) - open/close 27878 (time 1746129425.20 total 194.29 last 126.79) - open/close 29071 (time 1746129435.21 total 204.29 last 119.24) - open/close 30000 (time 1746129442.88 total 211.97 last 120.98) - open/close 31213 (time 1746129452.89 total 221.98 last 121.23) - open/close 32688 (time 1746129462.90 total 231.99 last 147.36) - open/close 33852 (time 1746129472.91 total 242.00 last 116.24) - open/close 35187 (time 1746129482.91 total 252.00 last 133.49) - open/close 36573 (time 1746129492.92 total 262.01 last 138.53) - open/close 38030 (time 1746129502.93 total 272.02 last 145.53) - open/close 39212 (time 1746129512.93 total 282.02 last 118.16) - open/close 40000 (time 1746129519.34 total 288.43 last 122.96) - open/close 41344 (time 1746129529.35 total 298.44 last 134.28) - open/close 42980 (time 1746129539.35 total 308.44 last 163.57) - open/close 44483 (time 1746129549.36 total 318.45 last 150.20) - open/close 45780 (time 1746129559.36 total 328.45 last 129.70) - open/close 46979 (time 1746129569.36 total 338.45 last 119.89) - open/close 48131 (time 1746129579.36 total 348.45 last 115.18) - open/close 49326 (time 1746129589.36 total 358.45 last 119.48) - open/close 50000 (time 1746129594.22 total 363.31 last 138.76) - open/close 51279 (time 1746129604.22 total 373.31 last 127.86) - open/close 52348 (time 1746129614.23 total 383.32 last 106.84) - open/close 53525 (time 1746129624.23 total 393.32 last 117.69) - open/close 55361 (time 1746129634.24 total 403.32 last 183.49) - open/close 56861 (time 1746129644.24 total 413.33 last 149.91) - open/close 58295 (time 1746129654.25 total 423.34 last 143.31) - open/close 59628 (time 1746129664.25 total 433.34 last 133.24) - open/close 60000 (time 1746129666.61 total 435.69 last 158.13) - open/close 61590 (time 1746129676.61 total 445.70 last 158.97) - open/close 63115 (time 1746129686.62 total 455.70 last 152.38) - open/close 64640 (time 1746129696.62 total 465.71 last 152.39) total: 64768 open/close in 466.48 seconds: 138.85 ops/second - /unlink 2677 (time 1746129708.94 total 10.00 last 267.60) - /unlink 5400 (time 1746129718.94 total 20.00 last 272.29) - /unlink 8517 (time 1746129728.94 total 30.01 last 311.67) - /unlink 10000 (time 1746129734.23 total 35.30 last 280.21) - /unlink 12710 (time 1746129744.23 total 45.30 last 270.94) - /unlink 15154 (time 1746129754.24 total 55.30 last 244.32) - /unlink 17509 (time 1746129764.24 total 65.30 last 235.49) - /unlink 20000 (time 1746129773.28 total 74.34 last 275.54) - /unlink 22097 (time 1746129783.28 total 84.35 last 209.62) - /unlink 24559 (time 1746129793.29 total 94.35 last 246.04) - /unlink 26688 (time 1746129803.29 total 104.36 last 212.84) - /unlink 28801 (time 1746129813.29 total 114.36 last 211.27) - /unlink 30000 (time 1746129818.62 total 119.69 last 224.89) - /unlink 32552 (time 1746129828.62 total 129.69 last 255.16) - /unlink 35607 (time 1746129838.63 total 139.69 last 305.43) - /unlink 37944 (time 1746129848.63 total 149.69 last 233.69) - /unlink 40000 (time 1746129857.59 total 158.66 last 229.41) - /unlink 43267 (time 1746129867.59 total 168.66 last 326.64) - /unlink 46375 (time 1746129877.59 total 178.66 last 310.79) - /unlink 49635 (time 1746129887.59 total 188.66 last 325.94) - /unlink 52967 (time 1746129897.59 total 198.66 last 333.16) - /unlink 56660 (time 1746129907.60 total 208.67 last 369.16) - /unlink 59300 (time 1746129917.60 total 218.67 last 263.99) - /unlink 60000 (time 1746129919.95 total 221.02 last 297.70) - /unlink 62695 (time 1746129929.95 total 231.02 last 269.42) - /unlink 64671 (time 1746129939.95 total 241.02 last 197.58) total: 64768 /unlink in 241.44 seconds: 268.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2