== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:32:36 (1746145956) Stopping clients: oleg418-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg418-client.virtnet /mnt/lustre2 (opts:-f) oleg418-server: oleg418-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing load_modules_local oleg418-server: Loading modules from /home/green/git/lustre-release/lustre oleg418-server: detected 4 online CPUs by sysfs oleg418-server: MODOPTS_LIBCFS= oleg418-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock 192.168.204.118@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1310 (time 1746146019.58 total 10.01 last 130.88) - open/close 2529 (time 1746146029.59 total 20.02 last 121.79) - open/close 3876 (time 1746146039.60 total 30.03 last 134.59) - open/close 5208 (time 1746146049.60 total 40.03 last 133.18) - open/close 6603 (time 1746146059.60 total 50.03 last 139.49) - open/close 8046 (time 1746146069.60 total 60.03 last 144.27) - open/close 9479 (time 1746146079.61 total 70.04 last 143.20) - open/close 10000 (time 1746146082.86 total 73.29 last 159.97) - open/close 11397 (time 1746146092.87 total 83.30 last 139.67) - open/close 12908 (time 1746146102.87 total 93.30 last 151.06) - open/close 14421 (time 1746146112.87 total 103.30 last 151.25) - open/close 15956 (time 1746146122.88 total 113.31 last 153.43) - open/close 17095 (time 1746146132.88 total 123.31 last 113.89) - open/close 18306 (time 1746146142.88 total 133.31 last 121.09) - open/close 19589 (time 1746146152.88 total 143.31 last 128.29) - open/close 20000 (time 1746146156.10 total 146.53 last 127.51) - open/close 21410 (time 1746146166.11 total 156.54 last 140.95) - open/close 23188 (time 1746146176.11 total 166.54 last 177.73) - open/close 24814 (time 1746146186.11 total 176.54 last 162.54) - open/close 26302 (time 1746146196.12 total 186.55 last 148.68) - open/close 27654 (time 1746146206.13 total 196.56 last 135.15) - open/close 29145 (time 1746146216.13 total 206.56 last 149.05) - open/close 30000 (time 1746146220.72 total 211.15 last 186.19) - open/close 31261 (time 1746146230.73 total 221.16 last 126.05) - open/close 32729 (time 1746146240.73 total 231.16 last 146.78) - open/close 34015 (time 1746146250.73 total 241.16 last 128.54) - open/close 35220 (time 1746146260.74 total 251.17 last 120.43) - open/close 36425 (time 1746146270.74 total 261.17 last 120.49) - open/close 37724 (time 1746146280.74 total 271.17 last 129.89) - open/close 39328 (time 1746146290.74 total 281.17 last 160.38) - open/close 40000 (time 1746146295.17 total 285.60 last 151.57) - open/close 41484 (time 1746146305.18 total 295.61 last 148.38) - open/close 42911 (time 1746146315.18 total 305.61 last 142.69) - open/close 44440 (time 1746146325.18 total 315.61 last 152.89) - open/close 45568 (time 1746146335.19 total 325.62 last 112.68) - open/close 46843 (time 1746146345.19 total 335.62 last 127.48) - open/close 48061 (time 1746146355.19 total 345.62 last 121.77) - open/close 49298 (time 1746146365.19 total 355.62 last 123.65) - open/close 50000 (time 1746146370.94 total 361.37 last 122.11) - open/close 51337 (time 1746146380.95 total 371.38 last 133.59) - open/close 52541 (time 1746146390.96 total 381.39 last 120.33) - open/close 53913 (time 1746146400.96 total 391.39 last 137.12) - open/close 55347 (time 1746146410.97 total 401.40 last 143.36) - open/close 56702 (time 1746146420.97 total 411.40 last 135.44) - open/close 58121 (time 1746146430.98 total 421.41 last 141.75) - open/close 59383 (time 1746146440.99 total 431.42 last 126.12) - open/close 60000 (time 1746146445.65 total 436.08 last 132.32) - open/close 61538 (time 1746146455.66 total 446.09 last 153.66) - open/close 62927 (time 1746146465.67 total 456.10 last 138.81) - open/close 64580 (time 1746146475.67 total 466.10 last 165.25) total: 64768 open/close in 467.34 seconds: 138.59 ops/second - /unlink 2252 (time 1746146488.37 total 10.00 last 225.17) - /unlink 4717 (time 1746146498.37 total 20.01 last 246.39) - /unlink 7055 (time 1746146508.37 total 30.01 last 233.75) - /unlink 9168 (time 1746146518.38 total 40.01 last 211.21) - /unlink 10000 (time 1746146522.37 total 44.00 last 208.73) - /unlink 12461 (time 1746146532.37 total 54.00 last 246.01) - /unlink 15033 (time 1746146542.37 total 64.00 last 257.14) - /unlink 17752 (time 1746146552.37 total 74.00 last 271.87) - /unlink 20000 (time 1746146559.67 total 81.31 last 307.94) - /unlink 22941 (time 1746146569.67 total 91.31 last 294.05) - /unlink 25837 (time 1746146579.68 total 101.31 last 289.55) - /unlink 29251 (time 1746146589.68 total 111.31 last 341.35) - /unlink 30000 (time 1746146592.47 total 114.10 last 268.35) - /unlink 32838 (time 1746146602.47 total 124.10 last 283.74) - /unlink 35887 (time 1746146612.47 total 134.10 last 304.84) - /unlink 38756 (time 1746146622.47 total 144.11 last 286.87) - /unlink 40000 (time 1746146626.36 total 147.99 last 320.09) - /unlink 43273 (time 1746146636.36 total 157.99 last 327.28) - /unlink 46336 (time 1746146646.36 total 167.99 last 306.28) - /unlink 50000 (time 1746146656.20 total 177.83 last 372.51) - /unlink 53082 (time 1746146666.20 total 187.83 last 308.19) - /unlink 56073 (time 1746146676.20 total 197.83 last 298.99) - /unlink 58912 (time 1746146686.20 total 207.84 last 283.82) - /unlink 60000 (time 1746146690.60 total 212.24 last 247.17) - /unlink 62799 (time 1746146700.61 total 222.24 last 279.89) total: 64768 /unlink in 229.44 seconds: 282.28 ops/second Waiting for MDT destroys to complete - open/close 1893 (time 1746146727.36 total 10.00 last 189.27) - open/close 3596 (time 1746146737.37 total 20.01 last 170.21) - open/close 5226 (time 1746146747.37 total 30.01 last 162.93) - open/close 6906 (time 1746146757.37 total 40.01 last 167.98) - open/close 8464 (time 1746146767.38 total 50.01 last 155.77) - open/close 10000 (time 1746146775.33 total 57.97 last 193.16) - open/close 11874 (time 1746146785.33 total 67.97 last 187.40) - open/close 13497 (time 1746146795.33 total 77.97 last 162.27) - open/close 15126 (time 1746146805.33 total 87.97 last 162.83) - open/close 16380 (time 1746146815.34 total 97.97 last 125.38) - open/close 17629 (time 1746146825.34 total 107.98 last 124.82) - open/close 18952 (time 1746146835.35 total 117.99 last 132.23) - open/close 20000 (time 1746146843.13 total 125.77 last 134.66) - open/close 21476 (time 1746146853.13 total 135.77 last 147.59) - open/close 23030 (time 1746146863.13 total 145.77 last 155.38) - open/close 24366 (time 1746146873.13 total 155.77 last 133.59) - open/close 25593 (time 1746146883.13 total 165.77 last 122.68) - open/close 26856 (time 1746146893.14 total 175.78 last 126.27) - open/close 28368 (time 1746146903.14 total 185.78 last 151.14) - open/close 29606 (time 1746146913.14 total 195.78 last 123.76) - open/close 30000 (time 1746146916.39 total 199.03 last 121.31) - open/close 31211 (time 1746146926.40 total 209.04 last 121.00) - open/close 32582 (time 1746146936.40 total 219.04 last 137.03) - open/close 33862 (time 1746146946.41 total 229.05 last 127.96) - open/close 35348 (time 1746146956.41 total 239.05 last 148.58) - open/close 36459 (time 1746146966.42 total 249.06 last 111.02) - open/close 37598 (time 1746146976.42 total 259.06 last 113.84) - open/close 38674 (time 1746146986.43 total 269.07 last 107.52) - open/close 39932 (time 1746146996.44 total 279.08 last 125.69) - open/close 41163 (time 1746147006.44 total 289.08 last 123.05) - open/close 42622 (time 1746147016.45 total 299.09 last 145.84) - open/close 43941 (time 1746147026.45 total 309.09 last 131.78) - open/close 45242 (time 1746147036.46 total 319.10 last 130.03) - open/close 46589 (time 1746147046.46 total 329.10 last 134.69) - open/close 47904 (time 1746147056.47 total 339.11 last 131.38) - open/close 49178 (time 1746147066.48 total 349.12 last 127.30) - open/close 50000 (time 1746147072.01 total 354.65 last 148.69) - open/close 51368 (time 1746147082.01 total 364.65 last 136.79) - open/close 52880 (time 1746147092.01 total 374.65 last 151.15) - open/close 54304 (time 1746147102.02 total 384.65 last 142.34) - open/close 55550 (time 1746147112.02 total 394.66 last 124.55) - open/close 56960 (time 1746147122.02 total 404.66 last 140.95) - open/close 58311 (time 1746147132.03 total 414.67 last 135.04) - open/close 59680 (time 1746147142.03 total 424.67 last 136.85) - open/close 60000 (time 1746147144.60 total 427.24 last 124.50) - open/close 61465 (time 1746147154.60 total 437.24 last 146.44) - open/close 62772 (time 1746147164.61 total 447.25 last 130.58) - open/close 64007 (time 1746147174.62 total 457.26 last 123.46) total: 64768 open/close in 463.05 seconds: 139.87 ops/second - /unlink 2162 (time 1746147192.13 total 10.01 last 216.04) - /unlink 4357 (time 1746147202.13 total 20.01 last 219.44) - /unlink 6612 (time 1746147212.14 total 30.02 last 225.37) - /unlink 8633 (time 1746147222.14 total 40.02 last 202.04) - /unlink 10000 (time 1746147228.37 total 46.25 last 219.40) - /unlink 12062 (time 1746147238.38 total 56.25 last 206.15) - /unlink 14043 (time 1746147248.38 total 66.25 last 198.06) - /unlink 16364 (time 1746147258.38 total 76.26 last 232.03) - /unlink 18693 (time 1746147268.38 total 86.26 last 232.85) - /unlink 20000 (time 1746147273.19 total 91.07 last 271.60) - /unlink 22358 (time 1746147283.20 total 101.07 last 235.74) - /unlink 24557 (time 1746147293.20 total 111.08 last 219.85) - /unlink 26917 (time 1746147303.20 total 121.08 last 236.00) - /unlink 29494 (time 1746147313.21 total 131.08 last 257.52) - /unlink 30000 (time 1746147315.51 total 133.38 last 220.07) - /unlink 32414 (time 1746147325.51 total 143.39 last 241.29) - /unlink 34888 (time 1746147335.52 total 153.39 last 247.26) - /unlink 37501 (time 1746147345.52 total 163.40 last 261.20) - /unlink 39904 (time 1746147355.52 total 173.40 last 240.28) - /unlink 42247 (time 1746147365.52 total 183.40 last 234.28) - /unlink 44525 (time 1746147375.52 total 193.40 last 227.79) - /unlink 46936 (time 1746147385.53 total 203.41 last 240.92) - /unlink 49266 (time 1746147395.53 total 213.41 last 232.99) - /unlink 50000 (time 1746147398.76 total 216.64 last 227.18) - /unlink 52485 (time 1746147408.77 total 226.64 last 248.38) - /unlink 55644 (time 1746147418.77 total 236.64 last 315.85) - /unlink 58344 (time 1746147428.77 total 246.65 last 269.90) - /unlink 60000 (time 1746147434.64 total 252.52 last 282.12) - /unlink 62841 (time 1746147444.65 total 262.52 last 283.91) total: 64768 /unlink in 270.67 seconds: 239.29 ops/second Waiting for MDT destroys to complete - open/close 1549 (time 1746147474.06 total 10.00 last 154.89) - open/close 3058 (time 1746147484.06 total 20.01 last 150.81) - open/close 4554 (time 1746147494.06 total 30.01 last 149.60) - open/close 6070 (time 1746147504.07 total 40.01 last 151.54) - open/close 7690 (time 1746147514.07 total 50.02 last 161.91) - open/close 9613 (time 1746147524.08 total 60.02 last 192.23) - open/close 10000 (time 1746147526.21 total 62.16 last 181.26) - open/close 11467 (time 1746147536.21 total 72.16 last 146.69) - open/close 12919 (time 1746147546.21 total 82.16 last 145.19) - open/close 14484 (time 1746147556.22 total 92.17 last 156.35) - open/close 15974 (time 1746147566.23 total 102.17 last 148.91) - open/close 17357 (time 1746147576.24 total 112.18 last 138.21) - open/close 18712 (time 1746147586.24 total 122.18 last 135.46) - open/close 20000 (time 1746147595.25 total 131.19 last 142.94) - open/close 21317 (time 1746147605.25 total 141.20 last 131.67) - open/close 22728 (time 1746147615.25 total 151.20 last 141.09) - open/close 23914 (time 1746147625.26 total 161.20 last 118.50) - open/close 24855 (time 1746147635.29 total 171.23 last 93.88) - open/close 25802 (time 1746147645.29 total 181.24 last 94.62) - open/close 26989 (time 1746147655.30 total 191.24 last 118.63) - open/close 28183 (time 1746147665.30 total 201.24 last 119.39) - open/close 29541 (time 1746147675.30 total 211.25 last 135.74) - open/close 30000 (time 1746147678.91 total 214.85 last 127.27) - open/close 31616 (time 1746147688.91 total 224.86 last 161.54) - open/close 33170 (time 1746147698.92 total 234.86 last 155.38) - open/close 34600 (time 1746147708.92 total 244.87 last 142.89) - open/close 36033 (time 1746147718.93 total 254.87 last 143.27) - open/close 37725 (time 1746147728.93 total 264.87 last 169.19) - open/close 39348 (time 1746147738.93 total 274.87 last 162.28) - open/close 40000 (time 1746147742.40 total 278.34 last 187.71) - open/close 41694 (time 1746147752.40 total 288.35 last 169.35) - open/close 43132 (time 1746147762.41 total 298.35 last 143.77) - open/close 44667 (time 1746147772.41 total 308.35 last 153.49) - open/close 46129 (time 1746147782.41 total 318.35 last 146.19) - open/close 47620 (time 1746147792.41 total 328.36 last 149.00) - open/close 49066 (time 1746147802.42 total 338.36 last 144.54) - open/close 50000 (time 1746147809.11 total 345.05 last 139.57) - open/close 51645 (time 1746147819.11 total 355.06 last 164.43) - open/close 53309 (time 1746147829.12 total 365.06 last 166.34) - open/close 55245 (time 1746147839.12 total 375.06 last 193.55) - open/close 57007 (time 1746147849.12 total 385.06 last 176.17) - open/close 59037 (time 1746147859.13 total 395.07 last 202.93) - open/close 60000 (time 1746147865.34 total 401.28 last 155.01) - open/close 61442 (time 1746147875.34 total 411.29 last 144.12) - open/close 62940 (time 1746147885.35 total 421.29 last 149.76) - open/close 64420 (time 1746147895.36 total 431.30 last 147.87) total: 64768 open/close in 433.45 seconds: 149.43 ops/second - /unlink 2749 (time 1746147909.00 total 10.00 last 274.78) - /unlink 6031 (time 1746147919.00 total 20.01 last 328.14) - /unlink 8506 (time 1746147929.00 total 30.01 last 247.41) - /unlink 10000 (time 1746147935.86 total 36.87 last 217.92) - /unlink 12444 (time 1746147945.86 total 46.87 last 244.39) - /unlink 14649 (time 1746147955.87 total 56.87 last 220.34) - /unlink 17088 (time 1746147965.87 total 66.87 last 243.89) - /unlink 19700 (time 1746147975.87 total 76.88 last 261.15) - /unlink 22666 (time 1746147985.90 total 86.90 last 295.80) - /unlink 25053 (time 1746147995.90 total 96.90 last 238.68) - /unlink 27441 (time 1746148005.91 total 106.91 last 238.58) - /unlink 29622 (time 1746148015.91 total 116.92 last 218.04) - /unlink 31971 (time 1746148025.91 total 126.92 last 234.85) - /unlink 34234 (time 1746148035.91 total 136.92 last 226.21) - /unlink 36639 (time 1746148045.92 total 146.93 last 240.34) - /unlink 39079 (time 1746148055.92 total 156.93 last 243.92) - /unlink 40000 (time 1746148059.86 total 160.86 last 234.18) - /unlink 42424 (time 1746148069.86 total 170.87 last 242.29) - /unlink 44746 (time 1746148079.87 total 180.88 last 231.93) - /unlink 46989 (time 1746148089.87 total 190.88 last 224.29) - /unlink 49470 (time 1746148099.87 total 200.88 last 248.09) - /unlink 50000 (time 1746148102.40 total 203.41 last 209.95) - /unlink 52486 (time 1746148112.40 total 213.41 last 248.55) - /unlink 54986 (time 1746148122.40 total 223.41 last 249.95) - /unlink 57854 (time 1746148132.40 total 233.41 last 286.80) - /unlink 60000 (time 1746148139.44 total 240.45 last 304.82) - /unlink 62846 (time 1746148149.45 total 250.45 last 284.55) total: 64768 /unlink in 256.60 seconds: 252.41 ops/second Waiting for MDT destroys to complete - open/close 2106 (time 1746148169.99 total 10.01 last 210.48) - open/close 3923 (time 1746148179.99 total 20.01 last 181.68) - open/close 6092 (time 1746148189.99 total 30.01 last 216.88) - open/close 8061 (time 1746148199.99 total 40.01 last 196.84) - open/close 9874 (time 1746148210.00 total 50.02 last 181.15) - open/close 11952 (time 1746148220.01 total 60.02 last 207.71) - open/close 13429 (time 1746148230.01 total 70.03 last 147.63) - open/close 14853 (time 1746148240.02 total 80.04 last 142.28) - open/close 16287 (time 1746148250.02 total 90.04 last 143.35) - open/close 17627 (time 1746148260.03 total 100.05 last 133.89) - open/close 19079 (time 1746148270.03 total 110.05 last 145.19) - open/close 20000 (time 1746148275.97 total 115.99 last 155.11) - open/close 21643 (time 1746148285.97 total 125.99 last 164.29) - open/close 23146 (time 1746148295.97 total 135.99 last 150.25) - open/close 24324 (time 1746148305.98 total 145.99 last 117.78) - open/close 25621 (time 1746148315.99 total 156.01 last 129.51) - open/close 26817 (time 1746148326.00 total 166.01 last 119.54) - open/close 28007 (time 1746148336.00 total 176.02 last 118.91) - open/close 29208 (time 1746148346.01 total 186.02 last 120.06) - open/close 30000 (time 1746148352.07 total 192.09 last 130.64) - open/close 31420 (time 1746148362.07 total 202.09 last 141.94) - open/close 32944 (time 1746148372.08 total 212.09 last 152.38) - open/close 34220 (time 1746148382.08 total 222.09 last 127.57) - open/close 35498 (time 1746148392.09 total 232.10 last 127.71) - open/close 36985 (time 1746148402.09 total 242.11 last 148.59) - open/close 38443 (time 1746148412.10 total 252.11 last 145.72) - open/close 39716 (time 1746148422.10 total 262.12 last 127.27) - open/close 40000 (time 1746148424.29 total 264.31 last 129.49) - open/close 41267 (time 1746148434.30 total 274.32 last 126.63) - open/close 42537 (time 1746148444.30 total 284.32 last 126.99) - open/close 43797 (time 1746148454.30 total 294.32 last 125.99) - open/close 45066 (time 1746148464.30 total 304.32 last 126.87) - open/close 46331 (time 1746148474.31 total 314.32 last 126.47) - open/close 47645 (time 1746148484.31 total 324.32 last 131.40) - open/close 49566 (time 1746148494.31 total 334.33 last 192.04) - open/close 50000 (time 1746148496.64 total 336.65 last 186.41) - open/close 51848 (time 1746148506.64 total 346.66 last 184.77) - open/close 54240 (time 1746148516.64 total 356.66 last 239.17) - open/close 56030 (time 1746148526.64 total 366.66 last 178.98) - open/close 57438 (time 1746148536.64 total 376.66 last 140.77) - open/close 58879 (time 1746148546.65 total 386.66 last 144.05) - open/close 60000 (time 1746148554.51 total 394.53 last 142.52) - open/close 61525 (time 1746148564.52 total 404.54 last 152.40) - open/close 62877 (time 1746148574.53 total 414.54 last 135.11) - open/close 64175 (time 1746148584.53 total 424.55 last 129.73) total: 64768 open/close in 429.35 seconds: 150.85 ops/second - /unlink 2723 (time 1746148600.90 total 10.00 last 272.26) - /unlink 4952 (time 1746148610.90 total 20.01 last 222.81) - /unlink 7215 (time 1746148620.90 total 30.01 last 226.21) - /unlink 9733 (time 1746148630.91 total 40.01 last 251.73) - /unlink 12396 (time 1746148640.91 total 50.01 last 266.24) - /unlink 15235 (time 1746148650.91 total 60.02 last 283.78) - /unlink 17852 (time 1746148660.92 total 70.02 last 261.65) - /unlink 20000 (time 1746148668.72 total 77.83 last 275.08) - /unlink 22934 (time 1746148678.72 total 87.83 last 293.39) - /unlink 25635 (time 1746148688.73 total 97.84 last 269.95) - /unlink 28428 (time 1746148698.73 total 107.84 last 279.18) - /unlink 30000 (time 1746148703.90 total 113.00 last 304.51) - /unlink 32770 (time 1746148713.90 total 123.00 last 276.99) - /unlink 35470 (time 1746148723.90 total 133.00 last 269.96) - /unlink 37945 (time 1746148733.90 total 143.00 last 247.48) - /unlink 40000 (time 1746148742.31 total 151.41 last 244.39) - /unlink 42489 (time 1746148752.31 total 161.41 last 248.89) - /unlink 45238 (time 1746148762.31 total 171.41 last 274.89) - /unlink 47769 (time 1746148772.31 total 181.41 last 253.07) - /unlink 50000 (time 1746148780.63 total 189.74 last 268.12) - /unlink 52575 (time 1746148790.64 total 199.74 last 257.34) - /unlink 55087 (time 1746148800.64 total 209.74 last 251.20) - /unlink 57311 (time 1746148810.64 total 219.75 last 222.31) - /unlink 59671 (time 1746148820.64 total 229.75 last 235.95) - /unlink 62340 (time 1746148830.64 total 239.75 last 266.89) total: 64768 /unlink in 249.53 seconds: 259.56 ops/second Waiting for MDT destroys to complete - open/close 1615 (time 1746148861.29 total 10.00 last 161.46) - open/close 3372 (time 1746148871.30 total 20.00 last 175.68) - open/close 4864 (time 1746148881.30 total 30.01 last 149.17) - open/close 6183 (time 1746148891.30 total 40.01 last 131.83) - open/close 7601 (time 1746148901.31 total 50.02 last 141.73) - open/close 9088 (time 1746148911.31 total 60.02 last 148.64) - open/close 10000 (time 1746148917.48 total 66.19 last 147.80) - open/close 11701 (time 1746148927.48 total 76.19 last 170.07) - open/close 13371 (time 1746148937.49 total 86.19 last 166.96) - open/close 15183 (time 1746148947.49 total 96.20 last 181.15) - open/close 16841 (time 1746148957.50 total 106.20 last 165.68) - open/close 18778 (time 1746148967.50 total 116.21 last 193.63) - open/close 20000 (time 1746148975.25 total 123.96 last 157.60) - open/close 21394 (time 1746148985.26 total 133.97 last 139.34) - open/close 22970 (time 1746148995.26 total 143.97 last 157.59) - open/close 24570 (time 1746149005.26 total 153.97 last 159.93) - open/close 26183 (time 1746149015.27 total 163.97 last 161.24) - open/close 28126 (time 1746149025.27 total 173.98 last 194.23) - open/close 29931 (time 1746149035.27 total 183.98 last 180.46) - open/close 31600 (time 1746149045.28 total 193.99 last 166.80) - open/close 33576 (time 1746149055.28 total 203.99 last 197.59) - open/close 35466 (time 1746149065.28 total 213.99 last 188.96) - open/close 37007 (time 1746149075.29 total 223.99 last 154.03) - open/close 38475 (time 1746149085.29 total 234.00 last 146.74) - open/close 40000 (time 1746149093.71 total 242.41 last 181.19) - open/close 41624 (time 1746149103.71 total 252.42 last 162.37) - open/close 43135 (time 1746149113.71 total 262.42 last 151.02) - open/close 44497 (time 1746149123.72 total 272.43 last 136.06) - open/close 46008 (time 1746149133.73 total 282.44 last 150.95) - open/close 47435 (time 1746149143.74 total 292.45 last 142.60) - open/close 48906 (time 1746149153.74 total 302.45 last 147.04) - open/close 50000 (time 1746149159.56 total 308.26 last 188.27) - open/close 51765 (time 1746149169.56 total 318.27 last 176.41) - open/close 53521 (time 1746149179.56 total 328.27 last 175.54) - open/close 55244 (time 1746149189.57 total 338.28 last 172.22) - open/close 57080 (time 1746149199.57 total 348.28 last 183.49) - open/close 58766 (time 1746149209.58 total 358.29 last 168.54) - open/close 60000 (time 1746149216.66 total 365.37 last 174.17) - open/close 61741 (time 1746149226.67 total 375.38 last 173.92) - open/close 63323 (time 1746149236.68 total 385.39 last 158.13) total: 64768 open/close in 394.91 seconds: 164.01 ops/second - /unlink 3134 (time 1746149257.78 total 10.00 last 313.29) - /unlink 6718 (time 1746149267.79 total 20.01 last 358.19) - /unlink 9801 (time 1746149277.79 total 30.02 last 308.07) - /unlink 12506 (time 1746149287.80 total 40.02 last 270.32) - /unlink 15558 (time 1746149297.81 total 50.03 last 305.08) - /unlink 18318 (time 1746149307.81 total 60.03 last 275.92) - /unlink 20000 (time 1746149312.89 total 65.11 last 330.91) - /unlink 23239 (time 1746149322.89 total 75.11 last 323.88) - /unlink 26181 (time 1746149332.90 total 85.12 last 294.05) - /unlink 28882 (time 1746149342.90 total 95.12 last 270.09) - /unlink 30000 (time 1746149346.32 total 98.54 last 326.66) - /unlink 33141 (time 1746149356.33 total 108.55 last 313.92) - /unlink 36436 (time 1746149366.33 total 118.55 last 329.42) - /unlink 39592 (time 1746149376.33 total 128.55 last 315.56) - /unlink 42188 (time 1746149386.33 total 138.55 last 259.57) - /unlink 44709 (time 1746149396.33 total 148.55 last 252.09) - /unlink 47747 (time 1746149406.33 total 158.56 last 303.70) - /unlink 50000 (time 1746149414.14 total 166.37 last 288.51) - /unlink 52664 (time 1746149424.14 total 176.37 last 266.38) - /unlink 55624 (time 1746149434.15 total 186.37 last 295.92) - /unlink 58259 (time 1746149444.15 total 196.37 last 263.47) - /unlink 60000 (time 1746149451.31 total 203.53 last 243.02) - /unlink 62958 (time 1746149461.32 total 213.54 last 295.66) total: 64768 /unlink in 220.69 seconds: 293.48 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules via unload_modules_local on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2