== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:22:19 (1746138139) Stopping clients: oleg360-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg360-client.virtnet /mnt/lustre2 (opts:-f) oleg360-server: oleg360-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing load_modules_local oleg360-server: Loading modules from /home/green/git/lustre-release/lustre oleg360-server: detected 4 online CPUs by sysfs oleg360-server: MODOPTS_LIBCFS= oleg360-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg360-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg360-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg360-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg360-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug -1 all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2990 (time 1746138188.33 total 10.00 last 298.97) - open/close 5293 (time 1746138198.33 total 20.01 last 230.20) - open/close 7676 (time 1746138208.34 total 30.01 last 238.22) - open/close 9608 (time 1746138218.34 total 40.02 last 193.06) - open/close 10000 (time 1746138220.63 total 42.30 last 171.61) - open/close 12380 (time 1746138230.63 total 52.30 last 237.99) - open/close 14667 (time 1746138240.63 total 62.30 last 228.66) - open/close 17052 (time 1746138250.63 total 72.31 last 238.42) - open/close 19135 (time 1746138260.64 total 82.31 last 208.27) - open/close 20000 (time 1746138263.87 total 85.54 last 267.39) - open/close 22003 (time 1746138273.87 total 95.54 last 200.26) - open/close 24537 (time 1746138283.88 total 105.55 last 253.33) - open/close 26387 (time 1746138293.88 total 115.55 last 184.98) - open/close 28422 (time 1746138303.88 total 125.55 last 203.42) - open/close 30000 (time 1746138311.52 total 133.19 last 206.65) - open/close 31739 (time 1746138321.52 total 143.19 last 173.82) - open/close 33452 (time 1746138331.53 total 153.20 last 171.11) - open/close 34984 (time 1746138341.53 total 163.21 last 153.19) - open/close 36614 (time 1746138351.54 total 173.21 last 162.97) - open/close 38336 (time 1746138361.54 total 183.21 last 172.17) - open/close 40000 (time 1746138370.74 total 192.42 last 180.75) - open/close 41852 (time 1746138380.74 total 202.42 last 185.17) - open/close 44198 (time 1746138390.74 total 212.42 last 234.59) - open/close 46725 (time 1746138400.75 total 222.42 last 252.62) - open/close 48863 (time 1746138410.75 total 232.42 last 213.79) - open/close 50000 (time 1746138417.97 total 239.64 last 157.46) - open/close 51629 (time 1746138427.97 total 249.64 last 162.90) - open/close 52890 (time 1746138437.97 total 259.64 last 126.06) - open/close 54646 (time 1746138447.97 total 269.64 last 175.59) - open/close 56025 (time 1746138457.97 total 279.65 last 137.88) - open/close 57495 (time 1746138467.98 total 289.65 last 146.92) - open/close 58877 (time 1746138477.98 total 299.65 last 138.16) - open/close 60000 (time 1746138486.62 total 308.29 last 130.09) - open/close 61282 (time 1746138496.62 total 318.30 last 128.10) - open/close 62747 (time 1746138506.63 total 328.30 last 146.45) - open/close 64139 (time 1746138516.63 total 338.30 last 139.17) total: 64768 open/close in 341.94 seconds: 189.41 ops/second - /unlink 4067 (time 1746138531.36 total 10.00 last 406.63) - /unlink 7273 (time 1746138541.36 total 20.00 last 320.56) - /unlink 10000 (time 1746138547.89 total 26.53 last 417.94) - /unlink 14834 (time 1746138557.89 total 36.53 last 483.36) - /unlink 19818 (time 1746138567.89 total 46.53 last 498.37) - /unlink 24491 (time 1746138577.89 total 56.53 last 467.29) - /unlink 28253 (time 1746138587.89 total 66.53 last 376.11) - /unlink 30000 (time 1746138592.15 total 70.79 last 410.59) - /unlink 33818 (time 1746138602.15 total 80.79 last 381.73) - /unlink 36537 (time 1746138612.15 total 90.79 last 271.85) - /unlink 39281 (time 1746138622.15 total 100.79 last 274.34) - /unlink 40000 (time 1746138625.47 total 104.11 last 216.57) - /unlink 42080 (time 1746138635.48 total 114.11 last 207.95) - /unlink 44517 (time 1746138645.48 total 124.12 last 243.67) - /unlink 46853 (time 1746138655.48 total 134.12 last 233.57) - /unlink 49547 (time 1746138665.48 total 144.12 last 269.37) - /unlink 52045 (time 1746138675.49 total 154.13 last 249.63) - /unlink 54241 (time 1746138685.49 total 164.13 last 219.40) - /unlink 56507 (time 1746138695.50 total 174.14 last 226.50) - /unlink 58847 (time 1746138705.50 total 184.14 last 233.98) - /unlink 60000 (time 1746138711.09 total 189.73 last 206.09) - /unlink 62427 (time 1746138721.09 total 199.73 last 242.70) - /unlink 64646 (time 1746138731.10 total 209.74 last 221.89) total: 64768 /unlink in 210.17 seconds: 308.17 ops/second Waiting for MDT destroys to complete - open/close 1312 (time 1746138752.74 total 10.00 last 131.14) - open/close 2509 (time 1746138762.75 total 20.01 last 119.59) - open/close 3889 (time 1746138772.75 total 30.01 last 137.99) - open/close 5322 (time 1746138782.76 total 40.02 last 143.21) - open/close 6745 (time 1746138792.76 total 50.02 last 142.30) - open/close 8010 (time 1746138802.77 total 60.03 last 126.36) - open/close 9144 (time 1746138812.78 total 70.04 last 113.32) - open/close 10000 (time 1746138819.06 total 76.32 last 136.18) - open/close 11257 (time 1746138829.07 total 86.33 last 125.66) - open/close 12605 (time 1746138839.07 total 96.33 last 134.75) - open/close 14013 (time 1746138849.07 total 106.33 last 140.79) - open/close 15388 (time 1746138859.07 total 116.33 last 137.46) - open/close 16846 (time 1746138869.07 total 126.34 last 145.79) - open/close 18168 (time 1746138879.08 total 136.34 last 132.16) - open/close 19596 (time 1746138889.08 total 146.35 last 142.70) - open/close 20000 (time 1746138891.48 total 148.74 last 168.78) - open/close 21465 (time 1746138901.48 total 158.74 last 146.47) - open/close 22810 (time 1746138911.49 total 168.75 last 134.40) - open/close 24125 (time 1746138921.50 total 178.76 last 131.35) - open/close 25383 (time 1746138931.50 total 188.77 last 125.72) - open/close 26814 (time 1746138941.51 total 198.77 last 143.01) - open/close 28153 (time 1746138951.51 total 208.77 last 133.89) - open/close 29451 (time 1746138961.52 total 218.78 last 129.75) - open/close 30000 (time 1746138965.95 total 223.21 last 123.75) - open/close 31388 (time 1746138975.96 total 233.22 last 138.73) - open/close 32646 (time 1746138985.96 total 243.22 last 125.80) - open/close 33914 (time 1746138995.96 total 253.22 last 126.80) - open/close 35218 (time 1746139005.96 total 263.22 last 130.35) - open/close 36393 (time 1746139015.96 total 273.23 last 117.47) - open/close 37533 (time 1746139025.97 total 283.23 last 113.98) - open/close 38752 (time 1746139035.98 total 293.24 last 121.71) - open/close 39952 (time 1746139045.99 total 303.25 last 119.94) - open/close 41129 (time 1746139055.99 total 313.25 last 117.69) - open/close 42328 (time 1746139065.99 total 323.25 last 119.90) - open/close 43464 (time 1746139076.00 total 333.26 last 113.48) - open/close 44544 (time 1746139086.00 total 343.26 last 107.96) - open/close 45649 (time 1746139096.01 total 353.27 last 110.46) - open/close 46788 (time 1746139106.01 total 363.27 last 113.87) - open/close 47894 (time 1746139116.01 total 373.27 last 110.59) - open/close 49018 (time 1746139126.02 total 383.28 last 112.33) - open/close 50000 (time 1746139132.94 total 390.20 last 141.76) - open/close 51238 (time 1746139142.95 total 400.21 last 123.76) - open/close 52434 (time 1746139152.96 total 410.22 last 119.49) - open/close 53789 (time 1746139162.96 total 420.22 last 135.48) - open/close 55065 (time 1746139172.96 total 430.22 last 127.57) - open/close 56423 (time 1746139182.97 total 440.23 last 135.63) - open/close 57567 (time 1746139192.98 total 450.24 last 114.32) - open/close 58874 (time 1746139202.99 total 460.25 last 130.61) - open/close 60000 (time 1746139212.07 total 469.33 last 123.93) - open/close 61237 (time 1746139222.07 total 479.33 last 123.70) - open/close 62458 (time 1746139232.08 total 489.34 last 121.97) - open/close 63739 (time 1746139242.08 total 499.34 last 128.10) total: 64768 open/close in 507.27 seconds: 127.68 ops/second - /unlink 2692 (time 1746139261.67 total 10.00 last 269.15) - /unlink 4948 (time 1746139271.68 total 20.00 last 225.55) - /unlink 7023 (time 1746139281.68 total 30.00 last 207.50) - /unlink 9446 (time 1746139291.68 total 40.01 last 242.27) - /unlink 10000 (time 1746139293.91 total 42.24 last 248.00) - /unlink 12301 (time 1746139303.91 total 52.24 last 230.02) - /unlink 14208 (time 1746139313.92 total 62.25 last 190.64) - /unlink 16158 (time 1746139323.92 total 72.25 last 194.96) - /unlink 18128 (time 1746139333.93 total 82.25 last 196.89) - /unlink 20000 (time 1746139341.24 total 89.57 last 255.81) - /unlink 22695 (time 1746139351.25 total 99.58 last 269.38) - /unlink 25055 (time 1746139361.25 total 109.58 last 235.90) - /unlink 27174 (time 1746139371.26 total 119.58 last 211.81) - /unlink 29245 (time 1746139381.26 total 129.59 last 207.05) - /unlink 30000 (time 1746139384.48 total 132.80 last 234.69) - /unlink 32559 (time 1746139394.48 total 142.80 last 255.90) - /unlink 35374 (time 1746139404.48 total 152.81 last 281.41) - /unlink 37887 (time 1746139414.48 total 162.81 last 251.29) - /unlink 40000 (time 1746139424.13 total 172.46 last 219.01) - /unlink 42544 (time 1746139434.13 total 182.46 last 254.35) - /unlink 45248 (time 1746139444.13 total 192.46 last 270.40) - /unlink 47926 (time 1746139454.13 total 202.46 last 267.76) - /unlink 50000 (time 1746139462.64 total 210.96 last 243.84) - /unlink 52489 (time 1746139472.64 total 220.97 last 248.85) - /unlink 55066 (time 1746139482.64 total 230.97 last 257.65) - /unlink 57955 (time 1746139492.65 total 240.97 last 288.76) - /unlink 60000 (time 1746139500.21 total 248.54 last 270.36) - /unlink 63014 (time 1746139510.21 total 258.54 last 301.30) total: 64768 /unlink in 264.64 seconds: 244.74 ops/second Waiting for MDT destroys to complete - open/close 1583 (time 1746139538.51 total 10.00 last 158.22) - open/close 3279 (time 1746139548.51 total 20.01 last 169.59) - open/close 4966 (time 1746139558.51 total 30.01 last 168.70) - open/close 6937 (time 1746139568.51 total 40.01 last 197.04) - open/close 8911 (time 1746139578.51 total 50.01 last 197.33) - open/close 10000 (time 1746139585.58 total 57.07 last 154.19) - open/close 11683 (time 1746139595.58 total 67.08 last 168.25) - open/close 13132 (time 1746139605.58 total 77.08 last 144.89) - open/close 14584 (time 1746139615.58 total 87.08 last 145.19) - open/close 15909 (time 1746139625.59 total 97.08 last 132.43) - open/close 17346 (time 1746139635.59 total 107.09 last 143.67) - open/close 18832 (time 1746139645.60 total 117.09 last 148.48) - open/close 20000 (time 1746139655.26 total 126.76 last 120.85) - open/close 21273 (time 1746139665.26 total 136.76 last 127.30) - open/close 22535 (time 1746139675.27 total 146.77 last 126.13) - open/close 23818 (time 1746139685.27 total 156.77 last 128.28) - open/close 25102 (time 1746139695.28 total 166.77 last 128.31) - open/close 26326 (time 1746139705.28 total 176.78 last 122.36) - open/close 27565 (time 1746139715.29 total 186.78 last 123.83) - open/close 28838 (time 1746139725.29 total 196.79 last 127.26) - open/close 30000 (time 1746139734.74 total 206.24 last 122.92) - open/close 31604 (time 1746139744.74 total 216.24 last 160.39) - open/close 32912 (time 1746139754.74 total 226.24 last 130.78) - open/close 34153 (time 1746139764.75 total 236.24 last 124.07) - open/close 35563 (time 1746139774.75 total 246.24 last 140.99) - open/close 36815 (time 1746139784.75 total 256.25 last 125.19) - open/close 38051 (time 1746139794.75 total 266.25 last 123.59) - open/close 39350 (time 1746139804.75 total 276.25 last 129.85) - open/close 40000 (time 1746139809.33 total 280.83 last 142.07) - open/close 41264 (time 1746139819.33 total 290.83 last 126.32) - open/close 42678 (time 1746139829.33 total 300.83 last 141.38) - open/close 44048 (time 1746139839.33 total 310.83 last 137.00) - open/close 45706 (time 1746139849.34 total 320.83 last 165.79) - open/close 47033 (time 1746139859.34 total 330.84 last 132.66) - open/close 48430 (time 1746139869.35 total 340.84 last 139.60) - open/close 50000 (time 1746139878.01 total 349.51 last 181.14) - open/close 51632 (time 1746139888.02 total 359.51 last 163.16) - open/close 52960 (time 1746139898.02 total 369.52 last 132.73) - open/close 54405 (time 1746139908.03 total 379.52 last 144.43) - open/close 55903 (time 1746139918.03 total 389.53 last 149.75) - open/close 57599 (time 1746139928.03 total 399.53 last 169.54) - open/close 58955 (time 1746139938.03 total 409.53 last 135.58) - open/close 60000 (time 1746139945.51 total 417.00 last 139.86) - open/close 61244 (time 1746139955.51 total 427.01 last 124.33) - open/close 62674 (time 1746139965.51 total 437.01 last 142.99) - open/close 63887 (time 1746139975.52 total 447.01 last 121.26) total: 64768 open/close in 454.87 seconds: 142.39 ops/second - /unlink 2172 (time 1746139995.17 total 10.00 last 217.19) - /unlink 4187 (time 1746140005.17 total 20.00 last 201.42) - /unlink 6129 (time 1746140015.19 total 30.02 last 193.82) - /unlink 8366 (time 1746140025.20 total 40.03 last 223.58) - /unlink 10000 (time 1746140033.59 total 48.42 last 194.75) - /unlink 12071 (time 1746140043.59 total 58.42 last 207.04) - /unlink 14183 (time 1746140053.59 total 68.42 last 211.20) - /unlink 15930 (time 1746140063.59 total 78.43 last 174.65) - /unlink 18331 (time 1746140073.60 total 88.43 last 240.07) - /unlink 20000 (time 1746140080.46 total 95.29 last 243.31) - /unlink 21837 (time 1746140090.46 total 105.29 last 183.59) - /unlink 23981 (time 1746140100.50 total 115.33 last 213.51) - /unlink 26075 (time 1746140110.51 total 125.34 last 209.33) - /unlink 28054 (time 1746140120.51 total 135.34 last 197.76) - /unlink 30000 (time 1746140130.20 total 145.03 last 200.86) - /unlink 31980 (time 1746140140.21 total 155.04 last 197.90) - /unlink 33944 (time 1746140150.21 total 165.04 last 196.38) - /unlink 35880 (time 1746140160.21 total 175.04 last 193.54) - /unlink 37765 (time 1746140170.22 total 185.05 last 188.42) - /unlink 39914 (time 1746140180.22 total 195.05 last 214.80) - /unlink 42041 (time 1746140190.22 total 205.05 last 212.68) - /unlink 44039 (time 1746140200.25 total 215.08 last 199.26) - /unlink 46143 (time 1746140210.25 total 225.08 last 210.36) - /unlink 48411 (time 1746140220.26 total 235.09 last 226.66) - /unlink 50000 (time 1746140227.63 total 242.46 last 215.54) - /unlink 51953 (time 1746140237.63 total 252.46 last 195.28) - /unlink 54323 (time 1746140247.63 total 262.46 last 236.98) - /unlink 56437 (time 1746140257.63 total 272.46 last 211.39) - /unlink 58644 (time 1746140267.63 total 282.46 last 220.68) - /unlink 60000 (time 1746140273.50 total 288.33 last 231.26) - /unlink 62163 (time 1746140283.50 total 298.33 last 216.24) - /unlink 64541 (time 1746140293.50 total 308.33 last 237.76) total: 64768 /unlink in 309.20 seconds: 209.47 ops/second Waiting for MDT destroys to complete - open/close 1370 (time 1746140320.35 total 10.00 last 136.95) - open/close 2723 (time 1746140330.35 total 20.00 last 135.30) - open/close 4105 (time 1746140340.36 total 30.01 last 138.14) - open/close 5520 (time 1746140350.36 total 40.01 last 141.44) - open/close 6846 (time 1746140360.37 total 50.02 last 132.52) - open/close 8326 (time 1746140370.37 total 60.02 last 147.94) - open/close 9612 (time 1746140380.37 total 70.02 last 128.59) - open/close 10000 (time 1746140383.28 total 72.94 last 133.14) - open/close 11364 (time 1746140393.29 total 82.95 last 136.30) - open/close 12741 (time 1746140403.30 total 92.95 last 137.57) - open/close 14117 (time 1746140413.31 total 102.96 last 137.52) - open/close 15494 (time 1746140423.31 total 112.96 last 137.64) - open/close 17026 (time 1746140433.31 total 122.97 last 153.18) - open/close 18385 (time 1746140443.32 total 132.97 last 135.81) - open/close 19720 (time 1746140453.32 total 142.97 last 133.48) - open/close 21230 (time 1746140463.32 total 152.98 last 150.98) - open/close 22648 (time 1746140473.33 total 162.98 last 141.73) - open/close 24031 (time 1746140483.33 total 172.98 last 138.26) - open/close 25443 (time 1746140493.33 total 182.99 last 141.15) - open/close 26812 (time 1746140503.33 total 192.99 last 136.88) - open/close 28223 (time 1746140513.34 total 202.99 last 141.01) - open/close 29572 (time 1746140523.34 total 212.99 last 134.90) - open/close 30000 (time 1746140526.74 total 216.39 last 126.05) - open/close 31268 (time 1746140536.74 total 226.39 last 126.80) - open/close 32582 (time 1746140546.74 total 236.39 last 131.35) - open/close 33737 (time 1746140556.75 total 246.41 last 115.35) - open/close 34857 (time 1746140566.76 total 256.41 last 111.96) - open/close 36096 (time 1746140576.76 total 266.41 last 123.86) - open/close 37442 (time 1746140586.76 total 276.41 last 134.59) - open/close 38947 (time 1746140596.76 total 286.42 last 150.46) - open/close 40000 (time 1746140603.92 total 293.58 last 147.07) - open/close 41278 (time 1746140613.93 total 303.59 last 127.69) - open/close 42753 (time 1746140623.94 total 313.59 last 147.45) - open/close 44128 (time 1746140633.94 total 323.60 last 137.37) - open/close 45391 (time 1746140643.95 total 333.60 last 126.27) - open/close 47267 (time 1746140653.95 total 343.60 last 187.55) - open/close 48833 (time 1746140663.96 total 353.61 last 156.52) - open/close 50000 (time 1746140671.49 total 361.14 last 154.91) - open/close 51477 (time 1746140681.49 total 371.14 last 147.69) - open/close 52823 (time 1746140691.49 total 381.14 last 134.60) - open/close 54354 (time 1746140701.50 total 391.15 last 153.00) - open/close 55938 (time 1746140711.50 total 401.15 last 158.39) - open/close 57682 (time 1746140721.50 total 411.15 last 174.34) - open/close 59167 (time 1746140731.50 total 421.16 last 148.46) - open/close 60000 (time 1746140737.77 total 427.42 last 132.93) - open/close 61368 (time 1746140747.77 total 437.43 last 136.77) - open/close 62602 (time 1746140757.77 total 447.43 last 123.39) - open/close 63829 (time 1746140767.77 total 457.43 last 122.68) total: 64768 open/close in 464.12 seconds: 139.55 ops/second - /unlink 2127 (time 1746140786.26 total 10.00 last 212.68) - /unlink 4344 (time 1746140796.26 total 20.00 last 221.63) - /unlink 6582 (time 1746140806.26 total 30.00 last 223.79) - /unlink 9089 (time 1746140816.27 total 40.01 last 250.55) - /unlink 10000 (time 1746140819.94 total 43.68 last 248.22) - /unlink 12644 (time 1746140829.94 total 53.68 last 264.28) - /unlink 14930 (time 1746140839.94 total 63.68 last 228.60) - /unlink 17458 (time 1746140849.95 total 73.69 last 252.62) - /unlink 19806 (time 1746140859.95 total 83.69 last 234.77) - /unlink 22146 (time 1746140869.96 total 93.70 last 233.91) - /unlink 24523 (time 1746140879.96 total 103.70 last 237.64) - /unlink 26779 (time 1746140889.96 total 113.70 last 225.51) - /unlink 29288 (time 1746140899.96 total 123.70 last 250.90) - /unlink 30000 (time 1746140902.50 total 126.24 last 281.13) - /unlink 32468 (time 1746140912.50 total 136.24 last 246.76) - /unlink 35125 (time 1746140922.50 total 146.24 last 265.55) - /unlink 38042 (time 1746140932.50 total 156.25 last 291.66) - /unlink 40000 (time 1746140940.66 total 164.40 last 240.12) - /unlink 42801 (time 1746140950.66 total 174.40 last 280.04) - /unlink 45661 (time 1746140960.66 total 184.40 last 285.97) - /unlink 48037 (time 1746140970.66 total 194.40 last 237.56) - /unlink 50000 (time 1746140978.19 total 201.93 last 260.82) - /unlink 52278 (time 1746140988.19 total 211.93 last 227.73) - /unlink 54622 (time 1746140998.20 total 221.94 last 234.36) - /unlink 56937 (time 1746141008.22 total 231.96 last 230.87) - /unlink 59536 (time 1746141018.23 total 241.97 last 259.76) - /unlink 62141 (time 1746141028.23 total 251.97 last 260.47) - /unlink 64374 (time 1746141038.23 total 261.97 last 223.23) total: 64768 /unlink in 263.55 seconds: 245.75 ops/second Waiting for MDT destroys to complete - open/close 1239 (time 1746141060.39 total 10.01 last 123.72) - open/close 2381 (time 1746141070.39 total 20.02 last 114.17) - open/close 3608 (time 1746141080.39 total 30.02 last 122.69) - open/close 4855 (time 1746141090.40 total 40.02 last 124.64) - open/close 5971 (time 1746141100.40 total 50.03 last 111.56) - open/close 7195 (time 1746141110.40 total 60.03 last 122.39) - open/close 8405 (time 1746141120.40 total 70.03 last 121.00) - open/close 9587 (time 1746141130.41 total 80.03 last 118.13) - open/close 10000 (time 1746141134.35 total 83.97 last 104.85) - open/close 11400 (time 1746141144.35 total 93.98 last 139.93) - open/close 12842 (time 1746141154.36 total 103.98 last 144.16) - open/close 14228 (time 1746141164.36 total 113.98 last 138.59) - open/close 15783 (time 1746141174.36 total 123.98 last 155.44) - open/close 17157 (time 1746141184.37 total 133.99 last 137.28) - open/close 18495 (time 1746141194.37 total 144.00 last 133.74) - open/close 19980 (time 1746141204.38 total 154.00 last 148.44) - open/close 21376 (time 1746141214.38 total 164.00 last 139.58) - open/close 22748 (time 1746141224.39 total 174.01 last 137.05) - open/close 24275 (time 1746141234.40 total 184.02 last 152.65) - open/close 25761 (time 1746141244.40 total 194.02 last 148.59) - open/close 27322 (time 1746141254.41 total 204.03 last 155.94) - open/close 28708 (time 1746141264.42 total 214.04 last 138.46) - open/close 30000 (time 1746141274.28 total 223.90 last 131.04) - open/close 31462 (time 1746141284.28 total 233.90 last 146.14) - open/close 32886 (time 1746141294.28 total 243.90 last 142.39) - open/close 34353 (time 1746141304.29 total 253.91 last 146.62) - open/close 35851 (time 1746141314.29 total 263.92 last 149.72) - open/close 37222 (time 1746141324.29 total 273.92 last 137.09) - open/close 38587 (time 1746141334.30 total 283.92 last 136.38) - open/close 40000 (time 1746141343.70 total 293.33 last 150.28) - open/close 41434 (time 1746141353.71 total 303.33 last 143.33) - open/close 42754 (time 1746141363.71 total 313.34 last 131.93) - open/close 44316 (time 1746141373.72 total 323.34 last 156.18) - open/close 45712 (time 1746141383.72 total 333.35 last 139.49) - open/close 47088 (time 1746141393.73 total 343.35 last 137.51) - open/close 48516 (time 1746141403.74 total 353.36 last 142.69) - open/close 50000 (time 1746141412.62 total 362.24 last 167.14) - open/close 51552 (time 1746141422.62 total 372.24 last 155.20) - open/close 53326 (time 1746141432.62 total 382.24 last 177.40) - open/close 55213 (time 1746141442.62 total 392.25 last 188.60) - open/close 56851 (time 1746141452.62 total 402.25 last 163.80) - open/close 58473 (time 1746141462.62 total 412.25 last 162.19) - open/close 60000 (time 1746141471.20 total 420.82 last 178.11) - open/close 61875 (time 1746141481.20 total 430.83 last 187.35) - open/close 63692 (time 1746141491.21 total 440.83 last 181.66) total: 64768 open/close in 446.16 seconds: 145.17 ops/second - /unlink 3751 (time 1746141507.67 total 10.00 last 374.95) - /unlink 7437 (time 1746141517.67 total 20.00 last 368.59) - /unlink 10000 (time 1746141525.37 total 27.71 last 332.78) - /unlink 13669 (time 1746141535.37 total 37.71 last 366.84) - /unlink 17590 (time 1746141545.38 total 47.71 last 391.93) - /unlink 20000 (time 1746141551.94 total 54.27 last 367.44) - /unlink 23911 (time 1746141561.94 total 64.27 last 391.06) - /unlink 27743 (time 1746141571.94 total 74.27 last 383.11) - /unlink 30000 (time 1746141578.78 total 81.11 last 330.08) - /unlink 33024 (time 1746141588.78 total 91.11 last 302.39) - /unlink 36138 (time 1746141598.78 total 101.11 last 311.40) - /unlink 39597 (time 1746141608.78 total 111.11 last 345.89) - /unlink 42942 (time 1746141618.78 total 121.11 last 334.47) - /unlink 46064 (time 1746141628.78 total 131.12 last 312.13) - /unlink 49455 (time 1746141638.78 total 141.12 last 339.09) - /unlink 54917 (time 1746141648.78 total 151.12 last 546.11) - /unlink 58531 (time 1746141658.78 total 161.12 last 361.35) - /unlink 60000 (time 1746141663.22 total 165.55 last 331.53) - /unlink 63699 (time 1746141673.22 total 175.55 last 369.88) total: 64768 /unlink in 177.89 seconds: 364.09 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg360-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg360-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg360-server stop mds service on oleg360-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg360-server unloading modules via unload_modules_local on: 'oleg360-server' oleg360-server: oleg360-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-client: ssh exited with exit code 2 pdsh@oleg360-client: oleg360-server: ssh exited with exit code 2