== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:38:11 (1746131891) Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: MODOPTS_LIBCFS= oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg214-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg214-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1458 (time 1746131951.96 total 10.00 last 145.76) - open/close 3201 (time 1746131961.96 total 20.01 last 174.24) - open/close 4684 (time 1746131971.98 total 30.02 last 148.10) - open/close 5979 (time 1746131981.98 total 40.03 last 129.40) - open/close 7250 (time 1746131991.99 total 50.04 last 126.99) - open/close 8963 (time 1746132001.99 total 60.04 last 171.26) - open/close 10000 (time 1746132009.46 total 67.50 last 138.92) - open/close 11533 (time 1746132019.46 total 77.50 last 153.29) - open/close 12901 (time 1746132029.46 total 87.51 last 136.75) - open/close 14384 (time 1746132039.47 total 97.51 last 148.26) - open/close 16295 (time 1746132049.47 total 107.51 last 191.04) - open/close 17801 (time 1746132059.48 total 117.52 last 150.50) - open/close 19697 (time 1746132069.48 total 127.52 last 189.59) - open/close 21622 (time 1746132079.48 total 137.52 last 192.40) - open/close 23358 (time 1746132089.49 total 147.53 last 173.51) - open/close 25173 (time 1746132099.49 total 157.54 last 181.40) - open/close 27500 (time 1746132109.49 total 167.54 last 232.63) - open/close 30000 (time 1746132117.95 total 175.99 last 295.67) - open/close 33366 (time 1746132127.95 total 185.99 last 336.58) - open/close 36153 (time 1746132137.95 total 195.99 last 278.69) - open/close 38861 (time 1746132147.96 total 206.00 last 270.62) - open/close 40000 (time 1746132152.65 total 210.69 last 242.88) - open/close 43289 (time 1746132162.65 total 220.69 last 328.82) - open/close 46175 (time 1746132172.66 total 230.70 last 288.44) - open/close 48072 (time 1746132182.66 total 240.70 last 189.69) - open/close 50000 (time 1746132192.38 total 250.42 last 198.28) - open/close 51371 (time 1746132202.39 total 260.43 last 137.02) - open/close 53077 (time 1746132212.39 total 270.43 last 170.55) - open/close 55319 (time 1746132222.39 total 280.43 last 224.13) - open/close 57678 (time 1746132232.39 total 290.44 last 235.88) - open/close 60000 (time 1746132241.84 total 299.88 last 245.88) - open/close 62633 (time 1746132251.84 total 309.88 last 263.30) total: 64768 open/close in 315.76 seconds: 205.12 ops/second - /unlink 5422 (time 1746132268.74 total 10.00 last 542.14) - /unlink 9450 (time 1746132278.76 total 20.02 last 401.87) - /unlink 14489 (time 1746132288.77 total 30.03 last 503.72) - /unlink 17891 (time 1746132298.77 total 40.03 last 340.15) - /unlink 20000 (time 1746132305.64 total 46.90 last 307.03) - /unlink 22296 (time 1746132315.64 total 56.90 last 229.57) - /unlink 25748 (time 1746132325.64 total 66.90 last 345.18) - /unlink 29750 (time 1746132335.64 total 76.90 last 400.17) - /unlink 33127 (time 1746132345.64 total 86.90 last 337.68) - /unlink 36729 (time 1746132355.64 total 96.90 last 360.16) - /unlink 39809 (time 1746132365.64 total 106.90 last 307.99) - /unlink 43705 (time 1746132375.65 total 116.91 last 389.40) - /unlink 48421 (time 1746132385.65 total 126.91 last 471.57) - /unlink 50000 (time 1746132389.56 total 130.82 last 403.85) - /unlink 53293 (time 1746132399.56 total 140.82 last 329.23) - /unlink 58565 (time 1746132409.56 total 150.82 last 527.17) - /unlink 60000 (time 1746132413.27 total 154.52 last 387.47) total: 64768 /unlink in 162.52 seconds: 398.53 ops/second Waiting for MDT destroys to complete - open/close 2421 (time 1746132442.83 total 10.00 last 242.01) - open/close 4133 (time 1746132452.83 total 20.01 last 171.08) - open/close 6829 (time 1746132462.83 total 30.01 last 269.59) - open/close 9418 (time 1746132472.84 total 40.01 last 258.82) - open/close 10000 (time 1746132475.45 total 42.63 last 222.93) - open/close 12989 (time 1746132485.45 total 52.63 last 298.83) - open/close 14725 (time 1746132495.46 total 62.63 last 173.50) - open/close 16260 (time 1746132505.46 total 72.64 last 153.44) - open/close 18177 (time 1746132515.46 total 82.64 last 191.70) - open/close 20000 (time 1746132523.00 total 90.18 last 241.83) - open/close 21675 (time 1746132533.01 total 100.18 last 167.38) - open/close 24340 (time 1746132543.01 total 110.19 last 266.42) - open/close 26391 (time 1746132553.01 total 120.19 last 205.03) - open/close 28750 (time 1746132563.01 total 130.19 last 235.84) - open/close 30000 (time 1746132567.60 total 134.77 last 272.77) - open/close 33308 (time 1746132577.60 total 144.78 last 330.69) - open/close 35102 (time 1746132587.60 total 154.78 last 179.37) - open/close 37085 (time 1746132597.60 total 164.78 last 198.29) - open/close 39063 (time 1746132607.61 total 174.78 last 197.72) - open/close 40000 (time 1746132612.84 total 180.02 last 178.90) - open/close 42068 (time 1746132622.84 total 190.02 last 206.79) - open/close 43892 (time 1746132632.85 total 200.03 last 182.35) - open/close 46021 (time 1746132642.85 total 210.03 last 212.84) - open/close 48179 (time 1746132652.85 total 220.03 last 215.77) - open/close 50000 (time 1746132661.52 total 228.70 last 209.98) - open/close 52265 (time 1746132671.52 total 238.70 last 226.48) - open/close 54166 (time 1746132681.53 total 248.70 last 190.07) - open/close 56322 (time 1746132691.53 total 258.71 last 215.56) - open/close 58359 (time 1746132701.53 total 268.71 last 203.69) - open/close 60000 (time 1746132708.69 total 275.87 last 229.12) - open/close 61810 (time 1746132718.69 total 285.87 last 180.91) - open/close 63531 (time 1746132728.70 total 295.87 last 172.08) total: 64768 open/close in 304.52 seconds: 212.69 ops/second - /unlink 2665 (time 1746132748.72 total 10.00 last 266.40) - /unlink 5114 (time 1746132758.72 total 20.01 last 244.85) - /unlink 7897 (time 1746132768.72 total 30.01 last 278.28) - /unlink 10000 (time 1746132775.52 total 36.81 last 309.34) - /unlink 12768 (time 1746132785.52 total 46.81 last 276.76) - /unlink 14968 (time 1746132795.52 total 56.81 last 219.94) - /unlink 17572 (time 1746132805.53 total 66.81 last 260.26) - /unlink 19945 (time 1746132815.53 total 76.81 last 237.30) - /unlink 22355 (time 1746132825.53 total 86.82 last 240.95) - /unlink 24889 (time 1746132835.53 total 96.82 last 253.34) - /unlink 27010 (time 1746132845.53 total 106.82 last 212.09) - /unlink 29494 (time 1746132855.54 total 116.82 last 248.36) - /unlink 32647 (time 1746132865.54 total 126.82 last 315.30) - /unlink 35142 (time 1746132875.54 total 136.83 last 249.27) - /unlink 38531 (time 1746132885.55 total 146.83 last 338.85) - /unlink 40000 (time 1746132888.20 total 149.49 last 552.95) - /unlink 42749 (time 1746132898.20 total 159.49 last 274.88) - /unlink 46791 (time 1746132908.21 total 169.49 last 404.08) - /unlink 49669 (time 1746132918.21 total 179.49 last 287.78) - /unlink 51901 (time 1746132928.21 total 189.50 last 223.15) - /unlink 55780 (time 1746132938.23 total 199.52 last 386.97) - /unlink 60000 (time 1746132946.48 total 207.77 last 511.65) - /unlink 63038 (time 1746132956.48 total 217.77 last 303.76) total: 64768 /unlink in 222.02 seconds: 291.73 ops/second Waiting for MDT destroys to complete - open/close 1383 (time 1746132979.05 total 10.00 last 138.28) - open/close 3137 (time 1746132989.05 total 20.00 last 175.39) - open/close 4468 (time 1746132999.06 total 30.01 last 133.01) - open/close 5668 (time 1746133009.06 total 40.01 last 119.99) - open/close 7054 (time 1746133019.06 total 50.01 last 138.55) - open/close 8267 (time 1746133029.07 total 60.02 last 121.17) - open/close 9360 (time 1746133039.08 total 70.03 last 109.24) - open/close 10000 (time 1746133044.82 total 75.77 last 111.55) - open/close 10957 (time 1746133054.82 total 85.77 last 95.68) - open/close 12053 (time 1746133064.82 total 95.77 last 109.54) - open/close 13148 (time 1746133074.83 total 105.78 last 109.47) - open/close 14242 (time 1746133084.83 total 115.78 last 109.35) - open/close 15291 (time 1746133094.84 total 125.79 last 104.82) - open/close 16367 (time 1746133104.84 total 135.79 last 107.55) - open/close 17638 (time 1746133114.85 total 145.80 last 126.99) - open/close 18745 (time 1746133124.85 total 155.80 last 110.69) - open/close 19940 (time 1746133134.86 total 165.81 last 119.44) - open/close 21010 (time 1746133144.87 total 175.82 last 106.91) - open/close 22180 (time 1746133154.87 total 185.82 last 116.96) - open/close 23372 (time 1746133164.88 total 195.83 last 119.08) - open/close 24491 (time 1746133174.88 total 205.83 last 111.87) - open/close 25621 (time 1746133184.89 total 215.84 last 112.88) - open/close 26874 (time 1746133194.90 total 225.85 last 125.26) - open/close 28327 (time 1746133204.91 total 235.86 last 145.16) - open/close 29540 (time 1746133214.91 total 245.86 last 121.26) - open/close 30000 (time 1746133219.15 total 250.10 last 108.54) - open/close 31017 (time 1746133229.15 total 260.10 last 101.69) - open/close 32143 (time 1746133239.15 total 270.10 last 112.58) - open/close 33278 (time 1746133249.15 total 280.10 last 113.49) - open/close 34325 (time 1746133259.16 total 290.11 last 104.57) - open/close 35553 (time 1746133269.16 total 300.12 last 122.78) - open/close 36913 (time 1746133279.17 total 310.12 last 135.95) - open/close 38279 (time 1746133289.17 total 320.12 last 136.55) - open/close 39501 (time 1746133299.17 total 330.12 last 122.19) - open/close 40000 (time 1746133303.15 total 334.10 last 125.39) - open/close 41688 (time 1746133313.15 total 344.10 last 168.79) - open/close 43146 (time 1746133323.16 total 354.11 last 145.65) - open/close 44196 (time 1746133333.17 total 364.12 last 104.95) - open/close 45335 (time 1746133343.17 total 374.12 last 113.89) - open/close 46677 (time 1746133353.17 total 384.12 last 134.18) - open/close 48282 (time 1746133363.17 total 394.13 last 160.42) - open/close 49596 (time 1746133373.18 total 404.13 last 131.39) - open/close 50000 (time 1746133375.90 total 406.85 last 148.18) - open/close 51421 (time 1746133385.91 total 416.86 last 141.96) - open/close 53310 (time 1746133395.91 total 426.86 last 188.87) - open/close 54974 (time 1746133405.92 total 436.87 last 166.23) - open/close 56256 (time 1746133415.93 total 446.88 last 128.11) - open/close 57674 (time 1746133425.94 total 456.89 last 141.73) - open/close 59228 (time 1746133435.94 total 466.90 last 155.25) - open/close 60000 (time 1746133440.21 total 471.16 last 180.96) - open/close 61445 (time 1746133450.22 total 481.17 last 144.42) - open/close 62739 (time 1746133460.22 total 491.17 last 129.35) - open/close 64000 (time 1746133470.23 total 501.18 last 126.01) total: 64768 open/close in 507.54 seconds: 127.61 ops/second - /unlink 2301 (time 1746133488.18 total 10.00 last 230.01) - /unlink 4400 (time 1746133498.18 total 20.00 last 209.87) - /unlink 6520 (time 1746133508.19 total 30.01 last 211.93) - /unlink 8580 (time 1746133518.19 total 40.01 last 205.90) - /unlink 10000 (time 1746133525.08 total 46.90 last 206.08) - /unlink 11805 (time 1746133535.09 total 56.91 last 180.46) - /unlink 13618 (time 1746133545.09 total 66.91 last 181.25) - /unlink 15639 (time 1746133555.09 total 76.91 last 202.05) - /unlink 17620 (time 1746133565.09 total 86.91 last 198.06) - /unlink 19860 (time 1746133575.10 total 96.92 last 223.96) - /unlink 22286 (time 1746133585.10 total 106.92 last 242.48) - /unlink 24221 (time 1746133595.10 total 116.92 last 193.47) - /unlink 25856 (time 1746133605.10 total 126.92 last 163.48) - /unlink 27591 (time 1746133615.10 total 136.92 last 173.48) - /unlink 29505 (time 1746133625.11 total 146.93 last 191.37) - /unlink 30000 (time 1746133628.48 total 150.30 last 146.74) - /unlink 31669 (time 1746133638.48 total 160.30 last 166.88) - /unlink 33839 (time 1746133648.48 total 170.30 last 216.98) - /unlink 35946 (time 1746133658.48 total 180.30 last 210.68) - /unlink 37858 (time 1746133668.48 total 190.30 last 191.20) - /unlink 39647 (time 1746133678.48 total 200.30 last 178.89) - /unlink 41885 (time 1746133688.49 total 210.31 last 223.76) - /unlink 43800 (time 1746133698.49 total 220.31 last 191.45) - /unlink 46084 (time 1746133708.49 total 230.31 last 228.38) - /unlink 48384 (time 1746133718.49 total 240.31 last 229.90) - /unlink 50000 (time 1746133726.94 total 248.76 last 191.30) - /unlink 52103 (time 1746133736.94 total 258.76 last 210.27) - /unlink 55625 (time 1746133746.95 total 268.76 last 352.08) - /unlink 57554 (time 1746133756.95 total 278.77 last 192.87) - /unlink 59177 (time 1746133766.95 total 288.77 last 162.28) - /unlink 60000 (time 1746133771.39 total 293.21 last 185.16) - /unlink 62337 (time 1746133781.40 total 303.22 last 233.58) - /unlink 63965 (time 1746133791.40 total 313.22 last 162.78) total: 64768 /unlink in 318.16 seconds: 203.57 ops/second Waiting for MDT destroys to complete - open/close 880 (time 1746133821.40 total 10.01 last 87.94) - open/close 1793 (time 1746133831.40 total 20.01 last 91.29) - open/close 2786 (time 1746133841.42 total 30.02 last 99.16) - open/close 3797 (time 1746133851.42 total 40.02 last 101.09) - open/close 4674 (time 1746133861.44 total 50.04 last 87.51) - open/close 5537 (time 1746133871.44 total 60.05 last 86.26) - open/close 6651 (time 1746133881.45 total 70.05 last 111.38) - open/close 7631 (time 1746133891.46 total 80.06 last 97.91) - open/close 8587 (time 1746133901.46 total 90.07 last 95.52) - open/close 9752 (time 1746133911.47 total 100.07 last 116.44) - open/close 10857 (time 1746133921.49 total 110.09 last 110.32) - open/close 11826 (time 1746133931.49 total 120.09 last 96.86) - open/close 12874 (time 1746133941.50 total 130.10 last 104.71) - open/close 13991 (time 1746133951.50 total 140.11 last 111.63) - open/close 14894 (time 1746133961.51 total 150.11 last 90.27) - open/close 15890 (time 1746133971.51 total 160.12 last 99.56) - open/close 16980 (time 1746133981.52 total 170.12 last 108.94) - open/close 17953 (time 1746133991.54 total 180.15 last 97.05) - open/close 18853 (time 1746134001.55 total 190.15 last 89.95) - open/close 19989 (time 1746134011.55 total 200.15 last 113.58) - open/close 21040 (time 1746134021.56 total 210.17 last 104.96) - open/close 21914 (time 1746134031.57 total 220.18 last 87.29) - open/close 23000 (time 1746134041.58 total 230.18 last 108.58) - open/close 24108 (time 1746134051.58 total 240.18 last 110.76) - open/close 25022 (time 1746134061.58 total 250.19 last 91.36) - open/close 26055 (time 1746134071.59 total 260.19 last 103.28) - open/close 27078 (time 1746134081.59 total 270.19 last 102.27) - open/close 28205 (time 1746134091.59 total 280.20 last 112.66) - open/close 29122 (time 1746134101.60 total 290.21 last 91.60) - open/close 30000 (time 1746134110.07 total 298.67 last 103.71) - open/close 31384 (time 1746134120.08 total 308.68 last 138.26) - open/close 32413 (time 1746134130.08 total 318.69 last 102.87) - open/close 33287 (time 1746134140.09 total 328.69 last 87.34) - open/close 34222 (time 1746134150.10 total 338.70 last 93.40) - open/close 35433 (time 1746134160.10 total 348.71 last 121.05) - open/close 36279 (time 1746134170.12 total 358.73 last 84.45) - open/close 37244 (time 1746134180.12 total 368.73 last 96.49) - open/close 38240 (time 1746134190.13 total 378.73 last 99.56) - open/close 39405 (time 1746134200.13 total 388.73 last 116.49) - open/close 40000 (time 1746134207.35 total 395.95 last 82.42) - open/close 40929 (time 1746134217.35 total 405.95 last 92.89) - open/close 41840 (time 1746134227.36 total 415.97 last 90.98) - open/close 42874 (time 1746134237.37 total 425.97 last 103.36) - open/close 43858 (time 1746134247.37 total 435.97 last 98.35) - open/close 44741 (time 1746134257.39 total 445.99 last 88.17) - open/close 45673 (time 1746134267.39 total 455.99 last 93.15) - open/close 46740 (time 1746134277.39 total 466.00 last 106.70) - open/close 47781 (time 1746134287.39 total 476.00 last 104.09) - open/close 48649 (time 1746134297.41 total 486.01 last 86.66) - open/close 49545 (time 1746134307.42 total 496.02 last 89.49) - open/close 50000 (time 1746134312.16 total 500.77 last 95.95) - open/close 51045 (time 1746134322.17 total 510.78 last 104.39) - open/close 52024 (time 1746134332.19 total 520.79 last 97.73) - open/close 52889 (time 1746134342.19 total 530.79 last 86.49) - open/close 53743 (time 1746134352.20 total 540.80 last 85.33) - open/close 54778 (time 1746134362.20 total 550.80 last 103.50) - open/close 55640 (time 1746134372.21 total 560.81 last 86.13) - open/close 56499 (time 1746134382.21 total 570.82 last 85.83) - open/close 57380 (time 1746134392.23 total 580.83 last 88.00) - open/close 58373 (time 1746134402.23 total 590.83 last 99.29) - open/close 59421 (time 1746134412.24 total 600.84 last 104.70) - open/close 60000 (time 1746134419.25 total 607.86 last 82.50) - open/close 60824 (time 1746134429.26 total 617.86 last 82.37) - open/close 61831 (time 1746134439.26 total 627.87 last 100.65) - open/close 63208 (time 1746134449.26 total 637.87 last 137.69) - open/close 64062 (time 1746134459.27 total 647.87 last 85.36) total: 64768 open/close in 656.21 seconds: 98.70 ops/second - /unlink 1025 (time 1746134479.69 total 10.00 last 102.49) - /unlink 2118 (time 1746134489.70 total 20.01 last 109.17) - /unlink 3082 (time 1746134499.71 total 30.02 last 96.36) - /unlink 4098 (time 1746134509.71 total 40.02 last 101.59) - /unlink 5319 (time 1746134519.71 total 50.02 last 122.09) - /unlink 6264 (time 1746134529.71 total 60.02 last 94.50) - /unlink 7215 (time 1746134539.71 total 70.02 last 95.06) - /unlink 8602 (time 1746134549.72 total 80.03 last 138.63) - /unlink 9625 (time 1746134559.72 total 90.03 last 102.30) - /unlink 10000 (time 1746134564.29 total 94.60 last 82.02) - /unlink 10894 (time 1746134574.29 total 104.60 last 89.39) - /unlink 12122 (time 1746134584.29 total 114.61 last 122.74) - /unlink 13259 (time 1746134594.31 total 124.62 last 113.50) - /unlink 14131 (time 1746134604.32 total 134.63 last 87.15) - /unlink 14926 (time 1746134614.32 total 144.64 last 79.45) - /unlink 16130 (time 1746134624.33 total 154.64 last 120.34) - /unlink 17072 (time 1746134634.34 total 164.65 last 94.13) - /unlink 17943 (time 1746134644.35 total 174.66 last 86.99) - /unlink 19114 (time 1746134654.35 total 184.67 last 117.02) - /unlink 20000 (time 1746134663.44 total 193.75 last 97.49) - /unlink 20952 (time 1746134673.45 total 203.76 last 95.17) - /unlink 21911 (time 1746134683.45 total 213.76 last 95.86) - /unlink 23137 (time 1746134693.46 total 223.77 last 122.49) - /unlink 24011 (time 1746134703.46 total 233.77 last 87.39) - /unlink 24979 (time 1746134713.47 total 243.78 last 96.71) - /unlink 25952 (time 1746134723.48 total 253.79 last 97.22) - /unlink 27178 (time 1746134733.48 total 263.79 last 122.56) - /unlink 28153 (time 1746134743.49 total 273.80 last 97.39) - /unlink 29095 (time 1746134753.50 total 283.81 last 94.15) - /unlink 30000 (time 1746134760.41 total 290.72 last 130.95) - /unlink 31151 (time 1746134770.42 total 300.73 last 115.02) - /unlink 31983 (time 1746134780.43 total 310.74 last 83.13) - /unlink 32969 (time 1746134790.43 total 320.75 last 98.52) - /unlink 34224 (time 1746134800.44 total 330.75 last 125.45) - /unlink 35125 (time 1746134810.44 total 340.75 last 90.06) - /unlink 36034 (time 1746134820.45 total 350.76 last 90.87) - /unlink 37189 (time 1746134830.45 total 360.76 last 115.45) - /unlink 38244 (time 1746134840.46 total 370.77 last 105.41) - /unlink 39087 (time 1746134850.47 total 380.78 last 84.18) - /unlink 40000 (time 1746134860.16 total 390.47 last 94.24) - /unlink 41141 (time 1746134870.17 total 400.48 last 114.03) - /unlink 42049 (time 1746134880.17 total 410.48 last 90.77) - /unlink 42908 (time 1746134890.17 total 420.48 last 85.87) - /unlink 43896 (time 1746134900.17 total 430.49 last 98.78) - /unlink 45028 (time 1746134910.18 total 440.49 last 113.13) - /unlink 46030 (time 1746134920.18 total 450.49 last 100.19) - /unlink 46911 (time 1746134930.20 total 460.51 last 87.96) - /unlink 48000 (time 1746134940.20 total 470.51 last 108.84) - /unlink 49117 (time 1746134950.22 total 480.53 last 111.55) - /unlink 50000 (time 1746134959.70 total 490.01 last 93.10) - /unlink 50956 (time 1746134969.71 total 500.02 last 95.56) - /unlink 51973 (time 1746134979.71 total 510.02 last 101.63) - /unlink 52996 (time 1746134989.72 total 520.03 last 102.23) - /unlink 53890 (time 1746134999.73 total 530.04 last 89.32) - /unlink 54953 (time 1746135009.73 total 540.04 last 106.24) - /unlink 56199 (time 1746135019.74 total 550.05 last 124.55) - /unlink 57083 (time 1746135029.74 total 560.05 last 88.39) - /unlink 57904 (time 1746135039.75 total 570.06 last 82.03) - /unlink 58795 (time 1746135049.75 total 580.06 last 89.06) - /unlink 59858 (time 1746135059.76 total 590.07 last 106.24) - /unlink 60881 (time 1746135069.76 total 600.08 last 102.23) - /unlink 61721 (time 1746135079.78 total 610.09 last 83.90) - /unlink 62631 (time 1746135089.78 total 620.09 last 90.98) - /unlink 63633 (time 1746135099.78 total 630.09 last 100.16) - /unlink 64681 (time 1746135109.79 total 640.10 last 104.75) total: 64768 /unlink in 640.88 seconds: 101.06 ops/second Waiting for MDT destroys to complete - open/close 880 (time 1746135134.22 total 10.01 last 87.94) - open/close 1772 (time 1746135144.24 total 20.03 last 89.03) - open/close 2977 (time 1746135154.24 total 30.03 last 120.49) - open/close 3931 (time 1746135164.25 total 40.03 last 95.34) - open/close 4883 (time 1746135174.26 total 50.04 last 95.12) - open/close 5894 (time 1746135184.26 total 60.04 last 101.06) - open/close 7058 (time 1746135194.27 total 70.05 last 116.35) - open/close 7965 (time 1746135204.27 total 80.05 last 90.69) - open/close 8820 (time 1746135214.29 total 90.07 last 85.33) - open/close 9769 (time 1746135224.29 total 100.07 last 94.88) - open/close 10000 (time 1746135226.46 total 102.24 last 106.39) - open/close 11197 (time 1746135236.47 total 112.26 last 119.55) - open/close 12115 (time 1746135246.48 total 122.27 last 91.71) - open/close 12956 (time 1746135256.49 total 132.27 last 84.03) - open/close 13815 (time 1746135266.51 total 142.29 last 85.76) - open/close 14758 (time 1746135276.51 total 152.30 last 94.24) - open/close 15704 (time 1746135286.53 total 162.31 last 94.44) - open/close 16455 (time 1746135296.54 total 172.32 last 75.07) - open/close 17268 (time 1746135306.55 total 182.33 last 81.20) - open/close 18158 (time 1746135316.55 total 192.33 last 88.98) - open/close 19101 (time 1746135326.56 total 202.34 last 94.22) - open/close 20000 (time 1746135336.54 total 212.33 last 90.03) - open/close 20707 (time 1746135346.55 total 222.33 last 70.68) - open/close 21444 (time 1746135356.55 total 232.33 last 73.66) - open/close 22194 (time 1746135366.55 total 242.34 last 74.98) - open/close 22961 (time 1746135376.55 total 252.34 last 76.70) - open/close 23879 (time 1746135386.56 total 262.34 last 91.77) - open/close 25002 (time 1746135396.56 total 272.34 last 112.29) - open/close 25828 (time 1746135406.57 total 282.35 last 82.53) - open/close 26706 (time 1746135416.58 total 292.36 last 87.71) - open/close 27562 (time 1746135426.58 total 302.37 last 85.55) - open/close 28625 (time 1746135436.59 total 312.37 last 106.28) - open/close 29533 (time 1746135446.60 total 322.38 last 90.69) - open/close 30000 (time 1746135452.40 total 328.19 last 80.41) - open/close 30843 (time 1746135462.41 total 338.19 last 84.23) - open/close 31770 (time 1746135472.42 total 348.20 last 92.62) - open/close 33201 (time 1746135482.43 total 358.21 last 143.05) - open/close 34024 (time 1746135492.43 total 368.21 last 82.24) - open/close 34923 (time 1746135502.43 total 378.22 last 89.89) - open/close 35818 (time 1746135512.44 total 388.22 last 89.47) - open/close 37040 (time 1746135522.44 total 398.22 last 122.18) - open/close 38013 (time 1746135532.44 total 408.22 last 97.26) - open/close 38952 (time 1746135542.45 total 418.24 last 93.80) - open/close 39955 (time 1746135552.45 total 428.24 last 100.28) - open/close 41229 (time 1746135562.46 total 438.24 last 127.37) - open/close 42161 (time 1746135572.46 total 448.24 last 93.16) - open/close 43073 (time 1746135582.48 total 458.27 last 91.00) - open/close 44027 (time 1746135592.49 total 468.27 last 95.38) - open/close 45307 (time 1746135602.49 total 478.27 last 127.93) - open/close 46213 (time 1746135612.50 total 488.28 last 90.53) - open/close 47331 (time 1746135622.52 total 498.30 last 111.60) - open/close 48386 (time 1746135632.52 total 508.30 last 105.48) - open/close 49498 (time 1746135642.52 total 518.30 last 111.16) - open/close 50000 (time 1746135647.36 total 523.15 last 103.66) - open/close 50887 (time 1746135657.37 total 533.16 last 88.62) - open/close 51938 (time 1746135667.38 total 543.16 last 105.03) - open/close 53022 (time 1746135677.38 total 553.17 last 108.36) - open/close 53925 (time 1746135687.39 total 563.17 last 90.28) - open/close 54788 (time 1746135697.40 total 573.18 last 86.21) - open/close 55667 (time 1746135707.40 total 583.18 last 87.86) - open/close 56607 (time 1746135717.41 total 593.19 last 93.96) - open/close 57655 (time 1746135727.41 total 603.19 last 104.78) - open/close 58547 (time 1746135737.41 total 613.19 last 89.16) - open/close 59327 (time 1746135747.41 total 623.20 last 77.98) - open/close 60000 (time 1746135754.88 total 630.66 last 90.14) - open/close 61074 (time 1746135764.89 total 640.67 last 107.35) - open/close 62021 (time 1746135774.89 total 650.67 last 94.66) - open/close 62844 (time 1746135784.89 total 660.68 last 82.26) - open/close 63713 (time 1746135794.90 total 670.69 last 86.83) total: 64768 open/close in 680.34 seconds: 95.20 ops/second - /unlink 918 (time 1746135816.18 total 10.00 last 91.77) - /unlink 1857 (time 1746135826.18 total 20.00 last 93.89) - /unlink 2814 (time 1746135836.19 total 30.01 last 95.63) - /unlink 3979 (time 1746135846.19 total 40.02 last 116.43) - /unlink 4981 (time 1746135856.19 total 50.02 last 100.19) - /unlink 6124 (time 1746135866.20 total 60.02 last 114.27) - /unlink 7431 (time 1746135876.20 total 70.03 last 130.64) - /unlink 8470 (time 1746135886.20 total 80.03 last 103.87) - /unlink 9510 (time 1746135896.21 total 90.03 last 103.95) - /unlink 10000 (time 1746135900.00 total 93.82 last 129.34) - /unlink 11436 (time 1746135910.00 total 103.83 last 143.54) - /unlink 12702 (time 1746135920.01 total 113.83 last 126.54) - /unlink 13958 (time 1746135930.01 total 123.83 last 125.60) - /unlink 15020 (time 1746135940.01 total 133.84 last 106.14) - /unlink 16269 (time 1746135950.02 total 143.84 last 124.85) - /unlink 17544 (time 1746135960.02 total 153.84 last 127.49) - /unlink 18801 (time 1746135970.02 total 163.85 last 125.61) - /unlink 20000 (time 1746135978.72 total 172.54 last 137.88) - /unlink 21387 (time 1746135988.73 total 182.55 last 138.57) - /unlink 22770 (time 1746135998.73 total 192.56 last 138.25) - /unlink 24184 (time 1746136008.74 total 202.56 last 141.37) - /unlink 25575 (time 1746136018.74 total 212.56 last 139.03) - /unlink 26881 (time 1746136028.78 total 222.60 last 130.12) - /unlink 28108 (time 1746136038.78 total 232.60 last 122.67) - /unlink 29463 (time 1746136048.79 total 242.61 last 135.38) - /unlink 30000 (time 1746136052.51 total 246.33 last 144.44) - /unlink 32157 (time 1746136062.51 total 256.33 last 215.67) - /unlink 34616 (time 1746136072.51 total 266.33 last 245.89) - /unlink 36885 (time 1746136082.51 total 276.34 last 226.76) - /unlink 39600 (time 1746136092.52 total 286.34 last 271.45) - /unlink 42459 (time 1746136102.52 total 296.34 last 285.84) - /unlink 44843 (time 1746136112.52 total 306.34 last 238.37) - /unlink 47222 (time 1746136122.52 total 316.34 last 237.87) - /unlink 49794 (time 1746136132.52 total 326.35 last 257.18) - /unlink 52391 (time 1746136142.52 total 336.35 last 259.66) - /unlink 55444 (time 1746136152.52 total 346.35 last 305.29) - /unlink 58784 (time 1746136162.52 total 356.35 last 333.99) - /unlink 60000 (time 1746136166.33 total 360.15 last 319.43) - /unlink 62735 (time 1746136176.33 total 370.16 last 273.43) total: 64768 /unlink in 378.32 seconds: 171.20 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2