== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:23:05 (1746145385) Stopping clients: oleg244-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg244-client.virtnet /mnt/lustre2 (opts:-f) oleg244-server: oleg244-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing load_modules_local oleg244-server: Loading modules from /home/green/git/lustre-release/lustre oleg244-server: detected 4 online CPUs by sysfs oleg244-server: MODOPTS_LIBCFS= oleg244-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg244-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg244-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg244-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1362 (time 1746145448.09 total 10.01 last 136.10) - open/close 2571 (time 1746145458.10 total 20.02 last 120.79) - open/close 3816 (time 1746145468.11 total 30.02 last 124.44) - open/close 5147 (time 1746145478.11 total 40.03 last 133.04) - open/close 6440 (time 1746145488.12 total 50.03 last 129.22) - open/close 7614 (time 1746145498.12 total 60.03 last 117.39) - open/close 8806 (time 1746145508.12 total 70.04 last 119.16) - open/close 9769 (time 1746145518.12 total 80.04 last 96.27) - open/close 10000 (time 1746145520.58 total 82.50 last 93.89) - open/close 10970 (time 1746145530.58 total 92.50 last 96.99) - open/close 11935 (time 1746145540.58 total 102.50 last 96.50) - open/close 12933 (time 1746145550.59 total 112.50 last 99.78) - open/close 14033 (time 1746145560.59 total 122.51 last 109.94) - open/close 15216 (time 1746145570.60 total 132.51 last 118.26) - open/close 16357 (time 1746145580.60 total 142.51 last 114.09) - open/close 17661 (time 1746145590.60 total 152.51 last 130.38) - open/close 18759 (time 1746145600.61 total 162.52 last 109.71) - open/close 19888 (time 1746145610.61 total 172.53 last 112.86) - open/close 21044 (time 1746145620.62 total 182.53 last 115.55) - open/close 22206 (time 1746145630.62 total 192.54 last 116.09) - open/close 23372 (time 1746145640.63 total 202.54 last 116.57) - open/close 24537 (time 1746145650.63 total 212.54 last 116.48) - open/close 25868 (time 1746145660.64 total 222.55 last 133.01) - open/close 26999 (time 1746145670.64 total 232.56 last 113.00) - open/close 28237 (time 1746145680.65 total 242.57 last 123.74) - open/close 29469 (time 1746145690.66 total 252.57 last 123.11) - open/close 30000 (time 1746145695.56 total 257.48 last 108.27) - open/close 31112 (time 1746145705.57 total 267.49 last 111.09) - open/close 32293 (time 1746145715.57 total 277.49 last 118.07) - open/close 33473 (time 1746145725.58 total 287.49 last 117.96) - open/close 34558 (time 1746145735.58 total 297.49 last 108.48) - open/close 35775 (time 1746145745.58 total 307.50 last 121.65) - open/close 36965 (time 1746145755.59 total 317.51 last 118.91) - open/close 38053 (time 1746145765.59 total 327.51 last 108.77) - open/close 39115 (time 1746145775.60 total 337.52 last 106.10) - open/close 40000 (time 1746145782.56 total 344.48 last 127.18) - open/close 41103 (time 1746145792.57 total 354.49 last 110.20) - open/close 42247 (time 1746145802.58 total 364.49 last 114.33) - open/close 43588 (time 1746145812.58 total 374.50 last 134.03) - open/close 44829 (time 1746145822.58 total 384.50 last 124.09) - open/close 46267 (time 1746145832.59 total 394.50 last 143.72) - open/close 47564 (time 1746145842.59 total 404.50 last 129.68) - open/close 48840 (time 1746145852.59 total 414.51 last 127.58) - open/close 50000 (time 1746145861.93 total 423.84 last 124.22) - open/close 51186 (time 1746145871.93 total 433.85 last 118.54) - open/close 52356 (time 1746145881.94 total 443.85 last 116.96) - open/close 53578 (time 1746145891.94 total 453.85 last 122.19) - open/close 54838 (time 1746145901.94 total 463.86 last 125.95) - open/close 56054 (time 1746145911.94 total 473.86 last 121.57) - open/close 57169 (time 1746145921.95 total 483.87 last 111.40) - open/close 58288 (time 1746145931.95 total 493.87 last 111.88) - open/close 59382 (time 1746145941.96 total 503.87 last 109.35) - open/close 60000 (time 1746145947.31 total 509.23 last 115.38) - open/close 61328 (time 1746145957.32 total 519.24 last 132.74) - open/close 62574 (time 1746145967.32 total 529.24 last 124.56) - open/close 63810 (time 1746145977.33 total 539.24 last 123.56) total: 64768 open/close in 547.01 seconds: 118.40 ops/second - /unlink 2172 (time 1746145997.01 total 10.00 last 217.14) - /unlink 4454 (time 1746146007.01 total 20.01 last 228.13) - /unlink 6794 (time 1746146017.02 total 30.01 last 233.89) - /unlink 8681 (time 1746146027.02 total 40.01 last 188.65) - /unlink 10000 (time 1746146034.10 total 47.09 last 186.43) - /unlink 12333 (time 1746146044.10 total 57.09 last 233.24) - /unlink 14605 (time 1746146054.11 total 67.10 last 227.05) - /unlink 16862 (time 1746146064.11 total 77.10 last 225.60) - /unlink 19283 (time 1746146074.11 total 87.10 last 242.09) - /unlink 20000 (time 1746146077.33 total 90.33 last 222.44) - /unlink 22370 (time 1746146087.34 total 100.33 last 236.91) - /unlink 24579 (time 1746146097.34 total 110.33 last 220.87) - /unlink 26581 (time 1746146107.34 total 120.33 last 200.18) - /unlink 28776 (time 1746146117.34 total 130.33 last 219.43) - /unlink 30000 (time 1746146122.23 total 135.22 last 250.34) - /unlink 32368 (time 1746146132.24 total 145.23 last 236.64) - /unlink 34699 (time 1746146142.24 total 155.23 last 233.02) - /unlink 36910 (time 1746146152.25 total 165.24 last 221.03) - /unlink 39074 (time 1746146162.25 total 175.24 last 216.32) - /unlink 40000 (time 1746146166.61 total 179.60 last 212.43) - /unlink 42240 (time 1746146176.61 total 189.60 last 223.96) - /unlink 44546 (time 1746146186.61 total 199.60 last 230.54) - /unlink 47153 (time 1746146196.61 total 209.61 last 260.68) - /unlink 50000 (time 1746146205.53 total 218.52 last 319.21) - /unlink 52627 (time 1746146215.53 total 228.53 last 262.66) - /unlink 55081 (time 1746146225.54 total 238.53 last 245.30) - /unlink 57214 (time 1746146235.55 total 248.54 last 213.11) - /unlink 59770 (time 1746146245.55 total 258.54 last 255.50) - /unlink 62128 (time 1746146255.55 total 268.54 last 235.78) - /unlink 64295 (time 1746146265.55 total 278.55 last 216.63) total: 64768 /unlink in 280.46 seconds: 230.93 ops/second Waiting for MDT destroys to complete - open/close 1335 (time 1746146290.20 total 10.00 last 133.45) - open/close 2894 (time 1746146300.20 total 20.01 last 155.86) - open/close 4281 (time 1746146310.20 total 30.01 last 138.69) - open/close 5917 (time 1746146320.21 total 40.01 last 163.55) - open/close 7273 (time 1746146330.21 total 50.01 last 135.60) - open/close 8692 (time 1746146340.21 total 60.01 last 141.85) - open/close 10000 (time 1746146348.78 total 68.59 last 152.55) - open/close 11453 (time 1746146358.79 total 78.59 last 145.25) - open/close 13013 (time 1746146368.79 total 88.59 last 155.97) - open/close 14332 (time 1746146378.79 total 98.60 last 131.85) - open/close 15685 (time 1746146388.80 total 108.60 last 135.25) - open/close 16986 (time 1746146398.80 total 118.61 last 130.02) - open/close 18279 (time 1746146408.81 total 128.61 last 129.23) - open/close 19555 (time 1746146418.81 total 138.62 last 127.58) - open/close 20000 (time 1746146422.10 total 141.90 last 135.36) - open/close 21339 (time 1746146432.11 total 151.92 last 133.68) - open/close 22759 (time 1746146442.16 total 161.96 last 141.41) - open/close 24143 (time 1746146452.18 total 171.99 last 138.01) - open/close 25485 (time 1746146462.22 total 182.03 last 133.68) - open/close 26793 (time 1746146472.24 total 192.04 last 130.61) - open/close 28109 (time 1746146482.26 total 202.07 last 131.30) - open/close 29508 (time 1746146492.26 total 212.07 last 139.87) - open/close 30000 (time 1746146495.21 total 215.02 last 166.77) - open/close 31424 (time 1746146505.23 total 225.03 last 142.22) - open/close 32637 (time 1746146515.23 total 235.03 last 121.26) - open/close 33998 (time 1746146525.23 total 245.04 last 136.06) - open/close 35148 (time 1746146535.24 total 255.04 last 114.91) - open/close 36263 (time 1746146545.24 total 265.05 last 111.45) - open/close 37378 (time 1746146555.25 total 275.05 last 111.46) - open/close 38548 (time 1746146565.25 total 285.06 last 116.96) - open/close 39686 (time 1746146575.26 total 295.06 last 113.75) - open/close 40000 (time 1746146577.78 total 297.59 last 124.26) - open/close 41104 (time 1746146587.80 total 307.60 last 110.24) - open/close 42447 (time 1746146597.80 total 317.61 last 134.25) - open/close 43577 (time 1746146607.81 total 327.61 last 112.94) - open/close 44702 (time 1746146617.81 total 337.62 last 112.46) - open/close 45862 (time 1746146627.81 total 347.62 last 115.96) - open/close 47233 (time 1746146637.82 total 357.62 last 137.04) - open/close 48493 (time 1746146647.82 total 367.63 last 125.95) - open/close 49727 (time 1746146657.83 total 377.63 last 123.34) - open/close 51056 (time 1746146667.83 total 387.63 last 132.89) - open/close 52281 (time 1746146677.83 total 397.64 last 122.43) - open/close 53509 (time 1746146687.83 total 407.64 last 122.80) - open/close 54727 (time 1746146697.84 total 417.64 last 121.73) - open/close 56001 (time 1746146707.84 total 427.65 last 127.39) - open/close 57177 (time 1746146717.85 total 437.65 last 117.53) - open/close 58405 (time 1746146727.85 total 447.66 last 122.75) - open/close 59887 (time 1746146737.85 total 457.66 last 148.19) - open/close 61168 (time 1746146747.85 total 467.66 last 128.08) - open/close 62488 (time 1746146757.85 total 477.66 last 131.99) - open/close 63731 (time 1746146767.86 total 487.66 last 124.25) total: 64768 open/close in 495.59 seconds: 130.69 ops/second - /unlink 2019 (time 1746146787.66 total 10.01 last 201.74) - /unlink 3853 (time 1746146797.67 total 20.02 last 183.19) - /unlink 5789 (time 1746146807.67 total 30.02 last 193.60) - /unlink 7595 (time 1746146817.67 total 40.02 last 180.58) - /unlink 9728 (time 1746146827.67 total 50.02 last 213.26) - /unlink 11838 (time 1746146837.68 total 60.03 last 210.89) - /unlink 14380 (time 1746146847.68 total 70.03 last 254.14) - /unlink 16613 (time 1746146857.68 total 80.03 last 223.25) - /unlink 18952 (time 1746146867.68 total 90.03 last 233.89) - /unlink 20000 (time 1746146871.54 total 93.89 last 271.66) - /unlink 22312 (time 1746146881.54 total 103.89 last 231.17) - /unlink 24750 (time 1746146891.54 total 113.89 last 243.76) - /unlink 27449 (time 1746146901.57 total 123.92 last 269.09) - /unlink 29581 (time 1746146911.57 total 133.92 last 213.20) - /unlink 32080 (time 1746146921.57 total 143.93 last 249.84) - /unlink 33986 (time 1746146931.58 total 153.93 last 190.54) - /unlink 36183 (time 1746146941.58 total 163.93 last 219.61) - /unlink 38269 (time 1746146951.58 total 173.93 last 208.57) - /unlink 40000 (time 1746146960.26 total 182.61 last 199.41) - /unlink 41949 (time 1746146970.27 total 192.62 last 194.78) - /unlink 43980 (time 1746146980.27 total 202.63 last 203.01) - /unlink 46163 (time 1746146990.28 total 212.63 last 218.27) - /unlink 48042 (time 1746147000.28 total 222.63 last 187.83) - /unlink 50000 (time 1746147010.25 total 232.60 last 196.32) - /unlink 52491 (time 1746147020.25 total 242.61 last 249.06) - /unlink 54700 (time 1746147030.26 total 252.61 last 220.85) - /unlink 56862 (time 1746147040.26 total 262.61 last 216.11) - /unlink 59070 (time 1746147050.26 total 272.62 last 220.71) - /unlink 60000 (time 1746147054.79 total 277.14 last 205.63) - /unlink 62229 (time 1746147064.79 total 287.14 last 222.76) - /unlink 64517 (time 1746147074.80 total 297.15 last 228.63) total: 64768 /unlink in 298.29 seconds: 217.13 ops/second Waiting for MDT destroys to complete - open/close 1353 (time 1746147096.68 total 10.00 last 135.24) - open/close 2681 (time 1746147106.69 total 20.01 last 132.79) - open/close 4109 (time 1746147116.70 total 30.02 last 142.63) - open/close 5290 (time 1746147126.70 total 40.02 last 118.09) - open/close 6544 (time 1746147136.70 total 50.02 last 125.40) - open/close 7927 (time 1746147146.70 total 60.02 last 138.29) - open/close 9195 (time 1746147156.70 total 70.02 last 126.78) - open/close 10000 (time 1746147162.82 total 76.14 last 131.59) - open/close 11271 (time 1746147172.82 total 86.14 last 127.07) - open/close 12573 (time 1746147182.83 total 96.15 last 130.13) - open/close 13787 (time 1746147192.83 total 106.15 last 121.33) - open/close 15189 (time 1746147202.83 total 116.15 last 140.17) - open/close 16502 (time 1746147212.84 total 126.16 last 131.28) - open/close 17928 (time 1746147222.84 total 136.16 last 142.46) - open/close 19245 (time 1746147232.85 total 146.17 last 131.63) - open/close 20000 (time 1746147238.99 total 152.31 last 122.95) - open/close 21198 (time 1746147249.00 total 162.31 last 119.75) - open/close 22462 (time 1746147259.00 total 172.32 last 126.36) - open/close 23877 (time 1746147269.00 total 182.32 last 141.44) - open/close 25051 (time 1746147279.01 total 192.33 last 117.34) - open/close 26090 (time 1746147289.01 total 202.33 last 103.86) - open/close 27329 (time 1746147299.01 total 212.33 last 123.87) - open/close 28483 (time 1746147309.02 total 222.34 last 115.38) - open/close 29692 (time 1746147319.02 total 232.34 last 120.90) - open/close 30000 (time 1746147321.37 total 234.69 last 131.05) - open/close 31355 (time 1746147331.37 total 244.69 last 135.50) - open/close 32664 (time 1746147341.37 total 254.69 last 130.80) - open/close 33848 (time 1746147351.39 total 264.71 last 118.25) - open/close 34970 (time 1746147361.40 total 274.72 last 112.11) - open/close 36125 (time 1746147371.41 total 284.73 last 115.38) - open/close 37323 (time 1746147381.41 total 294.73 last 119.73) - open/close 38532 (time 1746147391.42 total 304.74 last 120.85) - open/close 39755 (time 1746147401.42 total 314.74 last 122.24) - open/close 41028 (time 1746147411.43 total 324.75 last 127.18) - open/close 42277 (time 1746147421.44 total 334.76 last 124.82) - open/close 43462 (time 1746147431.44 total 344.76 last 118.42) - open/close 44773 (time 1746147441.45 total 354.77 last 130.99) - open/close 45873 (time 1746147451.45 total 364.77 last 109.98) - open/close 47115 (time 1746147461.46 total 374.78 last 124.15) - open/close 48432 (time 1746147471.46 total 384.78 last 131.59) - open/close 49577 (time 1746147481.47 total 394.79 last 114.49) - open/close 50000 (time 1746147485.06 total 398.38 last 117.52) - open/close 51226 (time 1746147495.07 total 408.39 last 122.56) - open/close 52809 (time 1746147505.07 total 418.39 last 158.26) - open/close 54108 (time 1746147515.08 total 428.40 last 129.81) - open/close 55574 (time 1746147525.08 total 438.40 last 146.56) - open/close 56799 (time 1746147535.09 total 448.41 last 122.43) - open/close 58036 (time 1746147545.09 total 458.41 last 123.63) - open/close 59529 (time 1746147555.09 total 468.41 last 149.26) - open/close 60000 (time 1746147558.94 total 472.26 last 122.54) - open/close 61247 (time 1746147568.94 total 482.26 last 124.61) - open/close 62720 (time 1746147578.95 total 492.27 last 147.24) - open/close 64040 (time 1746147588.96 total 502.28 last 131.83) total: 64768 open/close in 507.96 seconds: 127.51 ops/second - /unlink 2520 (time 1746147606.30 total 10.00 last 251.89) - /unlink 4956 (time 1746147616.30 total 20.01 last 243.52) - /unlink 7496 (time 1746147626.30 total 30.01 last 253.93) - /unlink 9360 (time 1746147636.31 total 40.02 last 186.22) - /unlink 10000 (time 1746147639.37 total 43.08 last 209.41) - /unlink 12137 (time 1746147649.37 total 53.08 last 213.66) - /unlink 14242 (time 1746147659.37 total 63.08 last 210.49) - /unlink 16453 (time 1746147669.38 total 73.09 last 220.83) - /unlink 18574 (time 1746147679.39 total 83.09 last 212.04) - /unlink 20000 (time 1746147685.55 total 89.25 last 231.49) - /unlink 22123 (time 1746147695.55 total 99.26 last 212.27) - /unlink 24221 (time 1746147705.55 total 109.26 last 209.78) - /unlink 26542 (time 1746147715.57 total 119.27 last 231.74) - /unlink 28523 (time 1746147725.57 total 129.27 last 198.07) - /unlink 30000 (time 1746147732.44 total 136.15 last 214.92) - /unlink 32518 (time 1746147742.44 total 146.15 last 251.67) - /unlink 34797 (time 1746147752.45 total 156.15 last 227.88) - /unlink 37105 (time 1746147762.45 total 166.15 last 230.77) - /unlink 39279 (time 1746147772.45 total 176.16 last 217.29) - /unlink 40000 (time 1746147775.63 total 179.34 last 226.73) - /unlink 42379 (time 1746147785.63 total 189.34 last 237.82) - /unlink 44562 (time 1746147795.64 total 199.34 last 218.26) - /unlink 46834 (time 1746147805.64 total 209.35 last 227.08) - /unlink 49913 (time 1746147815.64 total 219.35 last 307.85) - /unlink 53209 (time 1746147825.65 total 229.35 last 329.54) - /unlink 56099 (time 1746147835.65 total 239.35 last 288.98) - /unlink 58382 (time 1746147845.65 total 249.35 last 228.30) - /unlink 60000 (time 1746147852.99 total 256.69 last 220.46) - /unlink 62363 (time 1746147862.99 total 266.69 last 236.29) - /unlink 64493 (time 1746147872.99 total 276.69 last 212.95) total: 64768 /unlink in 277.88 seconds: 233.08 ops/second Waiting for MDT destroys to complete - open/close 1546 (time 1746147892.60 total 10.00 last 154.57) - open/close 3014 (time 1746147902.60 total 20.00 last 146.79) - open/close 4641 (time 1746147912.61 total 30.01 last 162.62) - open/close 6101 (time 1746147922.61 total 40.02 last 145.87) - open/close 7543 (time 1746147932.62 total 50.02 last 144.15) - open/close 9216 (time 1746147942.62 total 60.02 last 167.25) - open/close 10000 (time 1746147948.38 total 65.78 last 136.18) - open/close 11767 (time 1746147958.38 total 75.78 last 176.66) - open/close 13226 (time 1746147968.38 total 85.79 last 145.85) - open/close 14624 (time 1746147978.39 total 95.79 last 139.75) - open/close 15941 (time 1746147988.39 total 105.79 last 131.65) - open/close 17233 (time 1746147998.39 total 115.80 last 129.14) - open/close 18757 (time 1746148008.40 total 125.80 last 152.38) - open/close 20000 (time 1746148016.96 total 134.36 last 145.15) - open/close 21629 (time 1746148026.96 total 144.37 last 162.84) - open/close 23255 (time 1746148036.97 total 154.37 last 162.56) - open/close 25399 (time 1746148046.97 total 164.37 last 214.31) - open/close 27776 (time 1746148056.97 total 174.37 last 237.70) - open/close 29463 (time 1746148066.97 total 184.38 last 168.64) - open/close 30000 (time 1746148069.79 total 187.19 last 190.70) - open/close 31551 (time 1746148079.79 total 197.19 last 155.09) - open/close 32906 (time 1746148089.79 total 207.20 last 135.45) - open/close 34470 (time 1746148099.80 total 217.20 last 156.36) - open/close 35856 (time 1746148109.80 total 227.20 last 138.54) - open/close 37252 (time 1746148119.81 total 237.21 last 139.51) - open/close 38840 (time 1746148129.81 total 247.21 last 158.79) - open/close 40000 (time 1746148135.86 total 253.26 last 191.72) - open/close 41454 (time 1746148145.86 total 263.26 last 145.38) - open/close 42793 (time 1746148155.86 total 273.26 last 133.88) - open/close 44289 (time 1746148165.87 total 283.27 last 149.52) - open/close 45476 (time 1746148175.87 total 293.28 last 118.63) - open/close 46646 (time 1746148185.87 total 303.28 last 116.99) - open/close 47928 (time 1746148195.88 total 313.28 last 128.14) - open/close 49411 (time 1746148205.88 total 323.28 last 148.30) - open/close 50000 (time 1746148209.70 total 327.11 last 153.99) - open/close 51564 (time 1746148219.71 total 337.11 last 156.35) - open/close 53061 (time 1746148229.71 total 347.11 last 149.66) - open/close 54704 (time 1746148239.71 total 357.12 last 164.22) - open/close 56024 (time 1746148249.72 total 367.12 last 131.92) - open/close 57439 (time 1746148259.72 total 377.12 last 141.49) - open/close 58709 (time 1746148269.73 total 387.13 last 126.93) - open/close 60000 (time 1746148278.49 total 395.89 last 147.34) - open/close 61623 (time 1746148288.49 total 405.89 last 162.28) - open/close 63221 (time 1746148298.49 total 415.90 last 159.72) total: 64768 open/close in 425.44 seconds: 152.24 ops/second - /unlink 2710 (time 1746148319.52 total 10.01 last 270.84) - /unlink 5373 (time 1746148329.52 total 20.01 last 266.28) - /unlink 8096 (time 1746148339.52 total 30.01 last 272.23) - /unlink 10000 (time 1746148345.95 total 36.43 last 296.42) - /unlink 13463 (time 1746148355.95 total 46.43 last 346.25) - /unlink 17120 (time 1746148365.95 total 56.44 last 365.62) - /unlink 19994 (time 1746148375.95 total 66.44 last 287.39) - /unlink 22944 (time 1746148385.95 total 76.44 last 294.94) - /unlink 25895 (time 1746148395.95 total 86.44 last 295.02) - /unlink 29302 (time 1746148405.96 total 96.44 last 340.67) - /unlink 30000 (time 1746148407.97 total 98.45 last 346.86) - /unlink 32446 (time 1746148417.98 total 108.46 last 244.40) - /unlink 35289 (time 1746148427.98 total 118.47 last 284.22) - /unlink 37611 (time 1746148437.98 total 128.47 last 232.19) - /unlink 40000 (time 1746148447.10 total 137.58 last 262.03) - /unlink 42457 (time 1746148457.10 total 147.58 last 245.69) - /unlink 44973 (time 1746148467.10 total 157.58 last 251.58) - /unlink 47581 (time 1746148477.10 total 167.59 last 260.69) - /unlink 49771 (time 1746148487.10 total 177.59 last 218.96) - /unlink 52513 (time 1746148497.10 total 187.59 last 274.18) - /unlink 55018 (time 1746148507.11 total 197.59 last 250.48) - /unlink 57654 (time 1746148517.11 total 207.59 last 263.55) - /unlink 60000 (time 1746148526.51 total 216.99 last 249.56) - /unlink 62466 (time 1746148536.51 total 227.00 last 246.59) total: 64768 /unlink in 235.29 seconds: 275.27 ops/second Waiting for MDT destroys to complete - open/close 1658 (time 1746148564.34 total 10.00 last 165.77) - open/close 3339 (time 1746148574.35 total 20.01 last 167.93) - open/close 4688 (time 1746148584.35 total 30.01 last 134.87) - open/close 6171 (time 1746148594.36 total 40.02 last 148.25) - open/close 7902 (time 1746148604.36 total 50.02 last 173.09) - open/close 9356 (time 1746148614.36 total 60.02 last 145.39) - open/close 10000 (time 1746148618.93 total 64.59 last 140.96) - open/close 11402 (time 1746148628.93 total 74.60 last 140.09) - open/close 12823 (time 1746148638.93 total 84.60 last 142.10) - open/close 14204 (time 1746148648.94 total 94.60 last 138.03) - open/close 15478 (time 1746148658.94 total 104.60 last 127.38) - open/close 16890 (time 1746148668.95 total 114.61 last 141.11) - open/close 18405 (time 1746148678.95 total 124.61 last 151.47) - open/close 19853 (time 1746148688.95 total 134.61 last 144.74) - open/close 21269 (time 1746148698.96 total 144.62 last 141.55) - open/close 22731 (time 1746148708.96 total 154.62 last 146.17) - open/close 24333 (time 1746148718.96 total 164.62 last 160.15) - open/close 25972 (time 1746148728.97 total 174.63 last 163.83) - open/close 27440 (time 1746148738.98 total 184.64 last 146.66) - open/close 28725 (time 1746148748.98 total 194.64 last 128.43) - open/close 29908 (time 1746148758.99 total 204.65 last 118.25) - open/close 31188 (time 1746148769.00 total 214.66 last 127.84) - open/close 32603 (time 1746148779.00 total 224.66 last 141.46) - open/close 34088 (time 1746148789.01 total 234.67 last 148.42) - open/close 35561 (time 1746148799.02 total 244.68 last 147.16) - open/close 37026 (time 1746148809.03 total 254.69 last 146.34) - open/close 38416 (time 1746148819.03 total 264.70 last 138.90) - open/close 39896 (time 1746148829.04 total 274.70 last 147.95) - open/close 41310 (time 1746148839.04 total 284.70 last 141.31) - open/close 42657 (time 1746148849.05 total 294.71 last 134.61) - open/close 43875 (time 1746148859.06 total 304.72 last 121.68) - open/close 45262 (time 1746148869.06 total 314.72 last 138.65) - open/close 46669 (time 1746148879.07 total 324.73 last 140.67) - open/close 47966 (time 1746148889.07 total 334.73 last 129.61) - open/close 49438 (time 1746148899.08 total 344.74 last 147.11) - open/close 50000 (time 1746148902.33 total 347.99 last 172.78) - open/close 51502 (time 1746148912.33 total 357.99 last 150.16) - open/close 52809 (time 1746148922.34 total 368.00 last 130.63) - open/close 54071 (time 1746148932.34 total 378.00 last 126.19) - open/close 55409 (time 1746148942.35 total 388.01 last 133.72) - open/close 56859 (time 1746148952.35 total 398.01 last 144.89) - open/close 58643 (time 1746148962.36 total 408.02 last 178.37) - open/close 60000 (time 1746148971.43 total 417.09 last 149.55) - open/close 61387 (time 1746148981.43 total 427.09 last 138.67) - open/close 62744 (time 1746148991.44 total 437.10 last 135.65) - open/close 64188 (time 1746149001.44 total 447.10 last 144.28) total: 64768 open/close in 451.01 seconds: 143.61 ops/second - /unlink 2358 (time 1746149017.22 total 10.00 last 235.74) - /unlink 4729 (time 1746149027.22 total 20.00 last 237.10) - /unlink 7526 (time 1746149037.22 total 30.00 last 279.68) - /unlink 10000 (time 1746149045.85 total 38.63 last 286.70) - /unlink 12810 (time 1746149055.85 total 48.63 last 280.93) - /unlink 15377 (time 1746149065.85 total 58.64 last 256.66) - /unlink 17955 (time 1746149075.85 total 68.64 last 257.76) - /unlink 20000 (time 1746149082.51 total 75.30 last 307.10) - /unlink 22442 (time 1746149092.51 total 85.30 last 244.17) - /unlink 24921 (time 1746149102.51 total 95.30 last 247.88) - /unlink 27342 (time 1746149112.52 total 105.30 last 242.04) - /unlink 30000 (time 1746149122.08 total 114.86 last 277.96) - /unlink 32399 (time 1746149132.08 total 124.87 last 239.84) - /unlink 34776 (time 1746149142.08 total 134.87 last 237.67) - /unlink 37425 (time 1746149152.08 total 144.87 last 264.87) - /unlink 39811 (time 1746149162.09 total 154.87 last 238.57) - /unlink 42187 (time 1746149172.09 total 164.87 last 237.52) - /unlink 44713 (time 1746149182.09 total 174.88 last 252.53) - /unlink 47203 (time 1746149192.09 total 184.88 last 249.00) - /unlink 49523 (time 1746149202.10 total 194.88 last 231.89) - /unlink 52045 (time 1746149212.10 total 204.88 last 252.19) - /unlink 55044 (time 1746149222.10 total 214.88 last 299.90) - /unlink 58021 (time 1746149232.10 total 224.88 last 297.67) - /unlink 60000 (time 1746149240.23 total 233.01 last 243.46) - /unlink 62400 (time 1746149250.23 total 243.01 last 239.96) - /unlink 64767 (time 1746149260.23 total 253.01 last 236.70) total: 64768 /unlink in 253.01 seconds: 255.99 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg244-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg244-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server stop mds service on oleg244-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg244-server stop mds service on oleg244-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg244-server unloading modules via unload_modules_local on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2