== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:07:10 (1746126430) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg419-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg419-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1289 (time 1746126506.55 total 10.01 last 128.74) - open/close 2577 (time 1746126516.55 total 20.01 last 128.78) - open/close 3844 (time 1746126526.55 total 30.02 last 126.62) - open/close 5138 (time 1746126536.56 total 40.02 last 129.40) - open/close 6418 (time 1746126546.56 total 50.02 last 127.96) - open/close 7600 (time 1746126556.57 total 60.03 last 118.10) - open/close 8719 (time 1746126566.57 total 70.03 last 111.89) - open/close 9918 (time 1746126576.57 total 80.04 last 119.83) - open/close 11060 (time 1746126586.58 total 90.04 last 114.14) - open/close 12240 (time 1746126596.59 total 100.06 last 117.86) - open/close 13602 (time 1746126606.59 total 110.06 last 136.14) - open/close 14779 (time 1746126616.60 total 120.06 last 117.66) - open/close 16046 (time 1746126626.60 total 130.07 last 126.68) - open/close 17328 (time 1746126636.60 total 140.07 last 128.16) - open/close 18559 (time 1746126646.61 total 150.07 last 123.04) - open/close 20000 (time 1746126656.02 total 159.48 last 153.14) - open/close 21520 (time 1746126666.02 total 169.49 last 151.96) - open/close 22929 (time 1746126676.02 total 179.49 last 140.83) - open/close 24331 (time 1746126686.03 total 189.49 last 140.16) - open/close 25770 (time 1746126696.03 total 199.49 last 143.88) - open/close 27321 (time 1746126706.03 total 209.50 last 155.05) - open/close 28852 (time 1746126716.03 total 219.50 last 153.09) - open/close 30000 (time 1746126722.45 total 225.91 last 178.97) - open/close 31545 (time 1746126732.45 total 235.92 last 154.45) - open/close 33143 (time 1746126742.46 total 245.92 last 159.73) - open/close 34698 (time 1746126752.46 total 255.92 last 155.46) - open/close 36229 (time 1746126762.47 total 265.93 last 152.95) - open/close 38133 (time 1746126772.47 total 275.94 last 190.32) - open/close 39872 (time 1746126782.47 total 285.94 last 173.85) - open/close 41941 (time 1746126792.48 total 295.94 last 206.87) - open/close 43294 (time 1746126802.48 total 305.94 last 135.29) - open/close 44800 (time 1746126812.48 total 315.95 last 150.55) - open/close 46563 (time 1746126822.48 total 325.95 last 176.25) - open/close 48377 (time 1746126832.49 total 335.95 last 181.33) - open/close 49909 (time 1746126842.50 total 345.96 last 153.07) - open/close 51516 (time 1746126852.50 total 355.96 last 160.68) - open/close 53491 (time 1746126862.50 total 365.96 last 197.45) - open/close 55399 (time 1746126872.50 total 375.97 last 190.76) - open/close 57148 (time 1746126882.50 total 385.97 last 174.88) - open/close 58655 (time 1746126892.50 total 395.97 last 150.68) - open/close 60000 (time 1746126901.74 total 405.20 last 145.67) - open/close 61485 (time 1746126911.74 total 415.20 last 148.49) - open/close 63019 (time 1746126921.75 total 425.21 last 153.25) total: 64768 open/close in 434.15 seconds: 149.18 ops/second - /unlink 4000 (time 1746126941.93 total 10.00 last 399.94) - /unlink 7972 (time 1746126951.93 total 20.00 last 397.13) - /unlink 10000 (time 1746126957.79 total 25.87 last 345.65) - /unlink 13066 (time 1746126967.80 total 35.87 last 306.56) - /unlink 15897 (time 1746126977.80 total 45.87 last 283.09) - /unlink 19151 (time 1746126987.80 total 55.88 last 325.23) - /unlink 20000 (time 1746126990.52 total 58.59 last 312.67) - /unlink 22992 (time 1746127000.52 total 68.59 last 299.16) - /unlink 25683 (time 1746127010.52 total 78.60 last 268.96) - /unlink 29081 (time 1746127020.52 total 88.60 last 339.78) - /unlink 30000 (time 1746127023.10 total 91.18 last 356.63) - /unlink 33842 (time 1746127033.10 total 101.18 last 384.18) - /unlink 37093 (time 1746127043.10 total 111.18 last 325.07) - /unlink 40000 (time 1746127051.78 total 119.85 last 335.04) - /unlink 43572 (time 1746127061.78 total 129.86 last 357.16) - /unlink 47164 (time 1746127071.78 total 139.86 last 359.08) - /unlink 50000 (time 1746127078.95 total 147.03 last 395.46) - /unlink 53972 (time 1746127088.96 total 157.03 last 397.14) - /unlink 57118 (time 1746127098.96 total 167.03 last 314.60) - /unlink 60000 (time 1746127105.77 total 173.85 last 422.84) - /unlink 64373 (time 1746127115.77 total 183.85 last 437.26) total: 64768 /unlink in 184.69 seconds: 350.68 ops/second Waiting for MDT destroys to complete - open/close 2216 (time 1746127135.32 total 10.00 last 221.52) - open/close 4481 (time 1746127145.32 total 20.01 last 226.39) - open/close 6652 (time 1746127155.32 total 30.01 last 217.06) - open/close 9052 (time 1746127165.32 total 40.01 last 239.98) - open/close 10000 (time 1746127169.68 total 44.37 last 217.52) - open/close 12203 (time 1746127179.68 total 54.37 last 220.28) - open/close 14307 (time 1746127189.69 total 64.37 last 210.32) - open/close 16341 (time 1746127199.69 total 74.38 last 203.28) - open/close 18141 (time 1746127209.69 total 84.38 last 180.00) - open/close 20000 (time 1746127219.14 total 93.83 last 196.79) - open/close 21738 (time 1746127229.14 total 103.83 last 173.71) - open/close 23735 (time 1746127239.14 total 113.83 last 199.67) - open/close 25658 (time 1746127249.15 total 123.84 last 192.26) - open/close 28182 (time 1746127259.15 total 133.84 last 252.30) - open/close 30000 (time 1746127267.14 total 141.83 last 227.56) - open/close 32046 (time 1746127277.14 total 151.83 last 204.60) - open/close 34155 (time 1746127287.14 total 161.83 last 210.84) - open/close 36029 (time 1746127297.14 total 171.83 last 187.38) - open/close 38343 (time 1746127307.15 total 181.84 last 231.34) - open/close 39943 (time 1746127317.15 total 191.84 last 159.95) - open/close 41240 (time 1746127327.15 total 201.84 last 129.65) - open/close 43055 (time 1746127337.16 total 211.84 last 181.47) - open/close 45039 (time 1746127347.16 total 221.84 last 198.39) - open/close 46801 (time 1746127357.16 total 231.85 last 176.17) - open/close 48919 (time 1746127367.16 total 241.85 last 211.77) - open/close 50000 (time 1746127371.78 total 246.47 last 234.09) - open/close 51753 (time 1746127381.78 total 256.47 last 175.25) - open/close 53659 (time 1746127391.78 total 266.47 last 190.55) - open/close 55294 (time 1746127401.79 total 276.47 last 163.44) - open/close 57070 (time 1746127411.79 total 286.48 last 177.51) - open/close 59247 (time 1746127421.80 total 296.49 last 217.56) - open/close 60000 (time 1746127425.49 total 300.18 last 203.87) - open/close 61644 (time 1746127435.49 total 310.18 last 164.37) - open/close 63418 (time 1746127445.50 total 320.18 last 177.36) total: 64768 open/close in 328.29 seconds: 197.29 ops/second - /unlink 2593 (time 1746127464.67 total 10.00 last 259.24) - /unlink 4559 (time 1746127474.68 total 20.01 last 196.54) - /unlink 6598 (time 1746127484.68 total 30.01 last 203.86) - /unlink 8143 (time 1746127494.68 total 40.01 last 154.47) - /unlink 9690 (time 1746127504.68 total 50.01 last 154.67) - /unlink 11791 (time 1746127514.69 total 60.01 last 210.05) - /unlink 13844 (time 1746127524.69 total 70.02 last 205.25) - /unlink 15211 (time 1746127534.69 total 80.02 last 136.68) - /unlink 16556 (time 1746127544.69 total 90.02 last 134.50) - /unlink 17851 (time 1746127554.70 total 100.02 last 129.41) - /unlink 19115 (time 1746127564.70 total 110.03 last 126.35) - /unlink 20000 (time 1746127571.83 total 117.16 last 124.16) - /unlink 21339 (time 1746127581.83 total 127.16 last 133.82) - /unlink 23669 (time 1746127591.84 total 137.17 last 232.83) - /unlink 25833 (time 1746127601.87 total 147.20 last 215.70) - /unlink 27708 (time 1746127611.88 total 157.20 last 187.49) - /unlink 29760 (time 1746127621.88 total 167.21 last 205.15) - /unlink 31730 (time 1746127631.88 total 177.21 last 196.95) - /unlink 33550 (time 1746127641.88 total 187.21 last 181.99) - /unlink 35419 (time 1746127651.88 total 197.21 last 186.88) - /unlink 37302 (time 1746127661.88 total 207.21 last 188.24) - /unlink 39356 (time 1746127671.89 total 217.22 last 205.28) - /unlink 40000 (time 1746127675.06 total 220.39 last 203.26) - /unlink 42083 (time 1746127685.06 total 230.39 last 208.25) - /unlink 44086 (time 1746127695.08 total 240.40 last 199.99) - /unlink 46054 (time 1746127705.08 total 250.41 last 196.76) - /unlink 48088 (time 1746127715.08 total 260.41 last 203.34) - /unlink 50000 (time 1746127724.85 total 270.18 last 195.78) - /unlink 52061 (time 1746127734.85 total 280.18 last 206.05) - /unlink 54106 (time 1746127744.85 total 290.18 last 204.48) - /unlink 56167 (time 1746127754.86 total 300.19 last 205.95) - /unlink 58174 (time 1746127764.86 total 310.19 last 200.62) - /unlink 60000 (time 1746127773.71 total 319.03 last 206.46) - /unlink 62276 (time 1746127783.71 total 329.04 last 227.57) - /unlink 64536 (time 1746127793.71 total 339.04 last 225.94) total: 64768 /unlink in 340.00 seconds: 190.50 ops/second Waiting for MDT destroys to complete - open/close 1186 (time 1746127818.21 total 10.01 last 118.53) - open/close 2356 (time 1746127828.22 total 20.01 last 116.90) - open/close 3559 (time 1746127838.24 total 30.03 last 120.12) - open/close 4784 (time 1746127848.24 total 40.03 last 122.46) - open/close 6097 (time 1746127858.25 total 50.04 last 131.21) - open/close 7340 (time 1746127868.25 total 60.04 last 124.30) - open/close 8615 (time 1746127878.25 total 70.04 last 127.49) - open/close 9848 (time 1746127888.25 total 80.04 last 123.26) - open/close 11000 (time 1746127898.25 total 90.05 last 115.18) - open/close 12212 (time 1746127908.25 total 100.05 last 121.19) - open/close 13352 (time 1746127918.26 total 110.05 last 113.94) - open/close 14482 (time 1746127928.27 total 120.06 last 112.91) - open/close 15595 (time 1746127938.27 total 130.07 last 111.23) - open/close 16595 (time 1746127948.28 total 140.07 last 99.95) - open/close 17643 (time 1746127958.28 total 150.07 last 104.76) - open/close 18632 (time 1746127968.28 total 160.08 last 98.89) - open/close 19686 (time 1746127978.29 total 170.08 last 105.36) - open/close 20000 (time 1746127981.19 total 172.99 last 107.96) - open/close 21075 (time 1746127991.20 total 182.99 last 107.44) - open/close 22210 (time 1746128001.21 total 193.00 last 113.38) - open/close 23363 (time 1746128011.22 total 203.01 last 115.23) - open/close 24490 (time 1746128021.22 total 213.01 last 112.65) - open/close 25630 (time 1746128031.22 total 223.02 last 113.96) - open/close 26747 (time 1746128041.22 total 233.02 last 111.70) - open/close 27916 (time 1746128051.23 total 243.02 last 116.86) - open/close 28991 (time 1746128061.23 total 253.02 last 107.48) - open/close 30000 (time 1746128070.76 total 262.55 last 105.87) - open/close 30998 (time 1746128080.76 total 272.56 last 99.78) - open/close 32232 (time 1746128090.76 total 282.56 last 123.39) - open/close 33543 (time 1746128100.77 total 292.56 last 131.04) - open/close 34771 (time 1746128110.78 total 302.58 last 122.62) - open/close 36073 (time 1746128120.79 total 312.58 last 130.12) - open/close 37379 (time 1746128130.80 total 322.59 last 130.45) - open/close 38516 (time 1746128140.82 total 332.62 last 113.46) - open/close 39531 (time 1746128150.84 total 342.63 last 101.36) - open/close 40000 (time 1746128155.07 total 346.86 last 110.78) - open/close 40886 (time 1746128165.07 total 356.86 last 88.59) - open/close 41727 (time 1746128175.07 total 366.87 last 84.08) - open/close 42604 (time 1746128185.07 total 376.87 last 87.70) - open/close 43609 (time 1746128195.08 total 386.87 last 100.46) - open/close 44485 (time 1746128205.08 total 396.87 last 87.58) - open/close 45452 (time 1746128215.08 total 406.87 last 96.67) - open/close 46449 (time 1746128225.09 total 416.88 last 99.65) - open/close 47393 (time 1746128235.09 total 426.89 last 94.35) - open/close 48295 (time 1746128245.10 total 436.89 last 90.16) - open/close 49225 (time 1746128255.10 total 446.90 last 92.94) - open/close 50000 (time 1746128263.05 total 454.84 last 97.57) - open/close 51095 (time 1746128273.05 total 464.84 last 109.49) - open/close 52192 (time 1746128283.05 total 474.85 last 109.65) - open/close 53301 (time 1746128293.06 total 484.85 last 110.85) - open/close 54508 (time 1746128303.06 total 494.85 last 120.67) - open/close 55722 (time 1746128313.06 total 504.86 last 121.35) - open/close 57056 (time 1746128323.06 total 514.86 last 133.39) - open/close 58219 (time 1746128333.07 total 524.87 last 116.17) - open/close 59475 (time 1746128343.08 total 534.88 last 125.47) - open/close 60000 (time 1746128347.40 total 539.19 last 121.77) - open/close 61315 (time 1746128357.40 total 549.19 last 131.43) - open/close 62476 (time 1746128367.41 total 559.20 last 116.04) - open/close 63724 (time 1746128377.41 total 569.20 last 124.75) total: 64768 open/close in 577.34 seconds: 112.18 ops/second - /unlink 1941 (time 1746128397.52 total 10.00 last 194.03) - /unlink 3990 (time 1746128407.52 total 20.00 last 204.88) - /unlink 5892 (time 1746128417.52 total 30.01 last 190.18) - /unlink 7968 (time 1746128427.53 total 40.01 last 207.50) - /unlink 9928 (time 1746128437.53 total 50.01 last 195.94) - /unlink 12083 (time 1746128447.53 total 60.01 last 215.49) - /unlink 14271 (time 1746128457.54 total 70.02 last 218.70) - /unlink 16294 (time 1746128467.54 total 80.02 last 202.30) - /unlink 18015 (time 1746128477.56 total 90.04 last 171.72) - /unlink 19988 (time 1746128487.56 total 100.04 last 197.29) - /unlink 21835 (time 1746128497.56 total 110.04 last 184.68) - /unlink 23835 (time 1746128507.56 total 120.04 last 199.99) - /unlink 25945 (time 1746128517.57 total 130.05 last 210.88) - /unlink 28213 (time 1746128527.57 total 140.05 last 226.65) - /unlink 30000 (time 1746128535.74 total 148.22 last 218.77) - /unlink 32012 (time 1746128545.80 total 158.28 last 200.09) - /unlink 34018 (time 1746128555.80 total 168.28 last 200.52) - /unlink 36065 (time 1746128565.80 total 178.28 last 204.69) - /unlink 38221 (time 1746128575.80 total 188.29 last 215.56) - /unlink 40000 (time 1746128583.80 total 196.28 last 222.51) - /unlink 42255 (time 1746128593.80 total 206.28 last 225.50) - /unlink 44222 (time 1746128603.80 total 216.28 last 196.65) - /unlink 46072 (time 1746128613.81 total 226.29 last 184.87) - /unlink 48139 (time 1746128623.81 total 236.29 last 206.67) - /unlink 50000 (time 1746128632.49 total 244.97 last 214.35) - /unlink 52139 (time 1746128642.49 total 254.97 last 213.87) - /unlink 54214 (time 1746128652.49 total 264.98 last 207.46) - /unlink 56159 (time 1746128662.50 total 274.98 last 194.40) - /unlink 58071 (time 1746128672.51 total 284.99 last 191.08) - /unlink 59953 (time 1746128682.51 total 294.99 last 188.10) - /unlink 62053 (time 1746128692.51 total 304.99 last 209.99) - /unlink 63802 (time 1746128702.51 total 314.99 last 174.87) total: 64768 /unlink in 320.42 seconds: 202.14 ops/second Waiting for MDT destroys to complete - open/close 999 (time 1746128729.40 total 10.01 last 99.83) - open/close 1917 (time 1746128739.41 total 20.01 last 91.76) - open/close 3044 (time 1746128749.42 total 30.02 last 112.58) - open/close 3985 (time 1746128759.43 total 40.03 last 94.03) - open/close 4990 (time 1746128769.43 total 50.03 last 100.46) - open/close 6074 (time 1746128779.43 total 60.04 last 108.38) - open/close 7112 (time 1746128789.43 total 70.04 last 103.79) - open/close 8114 (time 1746128799.44 total 80.05 last 100.09) - open/close 9133 (time 1746128809.45 total 90.06 last 101.80) - open/close 10000 (time 1746128818.72 total 99.33 last 93.55) - open/close 10988 (time 1746128828.72 total 109.33 last 98.78) - open/close 11922 (time 1746128838.73 total 119.33 last 93.35) - open/close 12968 (time 1746128848.73 total 129.33 last 104.59) - open/close 13948 (time 1746128858.73 total 139.34 last 97.98) - open/close 14924 (time 1746128868.74 total 149.34 last 97.55) - open/close 15916 (time 1746128878.76 total 159.36 last 99.01) - open/close 16887 (time 1746128888.76 total 169.37 last 97.03) - open/close 17970 (time 1746128898.77 total 179.37 last 108.23) - open/close 19024 (time 1746128908.78 total 189.38 last 105.34) - open/close 20000 (time 1746128918.09 total 198.69 last 104.79) - open/close 21136 (time 1746128928.09 total 208.69 last 113.60) - open/close 22190 (time 1746128938.09 total 218.69 last 105.40) - open/close 23245 (time 1746128948.10 total 228.70 last 105.43) - open/close 24379 (time 1746128958.10 total 238.70 last 113.39) - open/close 25567 (time 1746128968.10 total 248.70 last 118.80) - open/close 26602 (time 1746128978.11 total 258.72 last 103.37) - open/close 27673 (time 1746128988.11 total 268.72 last 107.08) - open/close 28784 (time 1746128998.12 total 278.73 last 110.99) - open/close 29835 (time 1746129008.13 total 288.73 last 105.02) - open/close 30814 (time 1746129018.13 total 298.74 last 97.86) - open/close 31745 (time 1746129028.16 total 308.76 last 92.88) - open/close 32733 (time 1746129038.16 total 318.77 last 98.75) - open/close 33733 (time 1746129048.17 total 328.77 last 99.94) - open/close 34727 (time 1746129058.17 total 338.77 last 99.40) - open/close 35922 (time 1746129068.17 total 348.77 last 119.50) - open/close 37189 (time 1746129078.17 total 358.78 last 126.66) - open/close 38375 (time 1746129088.18 total 368.78 last 118.54) - open/close 39623 (time 1746129098.18 total 378.78 last 124.79) - open/close 40000 (time 1746129100.95 total 381.56 last 135.94) - open/close 41351 (time 1746129110.96 total 391.56 last 135.01) - open/close 42675 (time 1746129120.96 total 401.57 last 132.36) - open/close 44012 (time 1746129130.97 total 411.57 last 133.64) - open/close 45297 (time 1746129140.98 total 421.58 last 128.38) - open/close 46707 (time 1746129150.98 total 431.58 last 140.99) - open/close 48081 (time 1746129160.98 total 441.58 last 137.39) - open/close 49471 (time 1746129170.98 total 451.59 last 138.95) - open/close 50000 (time 1746129175.59 total 456.19 last 114.86) - open/close 51280 (time 1746129185.59 total 466.19 last 127.99) - open/close 52630 (time 1746129195.59 total 476.19 last 134.98) - open/close 53919 (time 1746129205.59 total 486.19 last 128.89) - open/close 55178 (time 1746129215.60 total 496.20 last 125.81) - open/close 56531 (time 1746129225.60 total 506.20 last 135.28) - open/close 57790 (time 1746129235.62 total 516.23 last 125.62) - open/close 59012 (time 1746129245.63 total 526.23 last 122.14) - open/close 60000 (time 1746129253.49 total 534.09 last 125.64) - open/close 61200 (time 1746129263.49 total 544.10 last 119.97) - open/close 62443 (time 1746129273.50 total 554.11 last 124.18) - open/close 63805 (time 1746129283.50 total 564.11 last 136.20) total: 64768 open/close in 571.74 seconds: 113.28 ops/second - /unlink 1782 (time 1746129303.03 total 10.00 last 178.18) - /unlink 3806 (time 1746129313.03 total 20.00 last 202.38) - /unlink 5709 (time 1746129323.04 total 30.01 last 190.24) - /unlink 7729 (time 1746129333.04 total 40.01 last 201.86) - /unlink 10000 (time 1746129343.03 total 50.00 last 227.40) - /unlink 12413 (time 1746129353.03 total 60.00 last 241.25) - /unlink 14653 (time 1746129363.03 total 70.00 last 223.97) - /unlink 16826 (time 1746129373.03 total 80.00 last 217.25) - /unlink 19090 (time 1746129383.04 total 90.01 last 226.36) - /unlink 20000 (time 1746129387.12 total 94.09 last 223.05) - /unlink 22766 (time 1746129397.12 total 104.09 last 276.46) - /unlink 25081 (time 1746129407.12 total 114.09 last 231.43) - /unlink 27327 (time 1746129417.13 total 124.10 last 224.56) - /unlink 29694 (time 1746129427.13 total 134.10 last 236.70) - /unlink 32306 (time 1746129437.13 total 144.10 last 261.17) - /unlink 35025 (time 1746129447.13 total 154.10 last 271.79) - /unlink 37438 (time 1746129457.13 total 164.10 last 241.28) - /unlink 39729 (time 1746129467.13 total 174.10 last 229.04) - /unlink 42114 (time 1746129477.14 total 184.11 last 238.38) - /unlink 44409 (time 1746129487.14 total 194.11 last 229.39) - /unlink 46679 (time 1746129497.15 total 204.12 last 226.96) - /unlink 49523 (time 1746129507.15 total 214.12 last 284.38) - /unlink 52245 (time 1746129517.15 total 224.12 last 272.08) - /unlink 55142 (time 1746129527.15 total 234.12 last 289.65) - /unlink 58041 (time 1746129537.15 total 244.12 last 289.87) - /unlink 60000 (time 1746129543.47 total 250.44 last 310.06) - /unlink 62600 (time 1746129553.48 total 260.45 last 259.92) total: 64768 /unlink in 269.17 seconds: 240.62 ops/second Waiting for MDT destroys to complete - open/close 1418 (time 1746129583.99 total 10.01 last 141.70) - open/close 2817 (time 1746129593.99 total 20.01 last 139.90) - open/close 4299 (time 1746129603.99 total 30.01 last 148.19) - open/close 5844 (time 1746129614.00 total 40.01 last 154.41) - open/close 7298 (time 1746129624.00 total 50.01 last 145.39) - open/close 8728 (time 1746129634.00 total 60.02 last 142.91) - open/close 10000 (time 1746129643.34 total 69.36 last 136.23) - open/close 11385 (time 1746129653.34 total 79.36 last 138.48) - open/close 12768 (time 1746129663.35 total 89.36 last 138.26) - open/close 14162 (time 1746129673.36 total 99.37 last 139.26) - open/close 15583 (time 1746129683.36 total 109.38 last 142.02) - open/close 16976 (time 1746129693.37 total 119.39 last 139.13) - open/close 18387 (time 1746129703.38 total 129.39 last 141.07) - open/close 19488 (time 1746129713.38 total 139.40 last 110.05) - open/close 20000 (time 1746129717.90 total 143.92 last 113.22) - open/close 21292 (time 1746129727.91 total 153.93 last 129.12) - open/close 22587 (time 1746129737.91 total 163.93 last 129.48) - open/close 23810 (time 1746129747.91 total 173.93 last 122.26) - open/close 25015 (time 1746129757.93 total 183.94 last 120.35) - open/close 26303 (time 1746129767.93 total 193.95 last 128.77) - open/close 27615 (time 1746129777.93 total 203.95 last 131.19) - open/close 28991 (time 1746129787.94 total 213.95 last 137.48) - open/close 30000 (time 1746129795.31 total 221.32 last 136.91) - open/close 31404 (time 1746129805.31 total 231.33 last 140.32) - open/close 32677 (time 1746129815.32 total 241.33 last 127.26) - open/close 34036 (time 1746129825.32 total 251.34 last 135.84) - open/close 35410 (time 1746129835.32 total 261.34 last 137.36) - open/close 36863 (time 1746129845.33 total 271.35 last 145.22) - open/close 38434 (time 1746129855.33 total 281.35 last 157.03) - open/close 40000 (time 1746129864.21 total 290.22 last 176.49) - open/close 42069 (time 1746129874.21 total 300.23 last 206.84) - open/close 44523 (time 1746129884.22 total 310.23 last 245.23) - open/close 46583 (time 1746129894.22 total 320.24 last 205.94) - open/close 48633 (time 1746129904.22 total 330.24 last 204.97) - open/close 50000 (time 1746129911.08 total 337.09 last 199.40) - open/close 52238 (time 1746129921.08 total 347.10 last 223.72) - open/close 54097 (time 1746129931.08 total 357.10 last 185.80) - open/close 55782 (time 1746129941.10 total 367.11 last 168.32) - open/close 57186 (time 1746129951.10 total 377.12 last 140.31) - open/close 58947 (time 1746129961.10 total 387.12 last 176.04) - open/close 60000 (time 1746129967.77 total 393.79 last 157.88) - open/close 62312 (time 1746129977.78 total 403.79 last 231.19) total: 64768 open/close in 413.47 seconds: 156.64 ops/second - /unlink 3843 (time 1746129998.58 total 10.00 last 384.26) - /unlink 8938 (time 1746130008.58 total 20.00 last 509.48) - /unlink 10000 (time 1746130011.76 total 23.18 last 334.53) - /unlink 13583 (time 1746130021.76 total 33.18 last 358.25) - /unlink 17500 (time 1746130031.76 total 43.18 last 391.54) - /unlink 20000 (time 1746130039.81 total 51.22 last 310.83) - /unlink 22781 (time 1746130049.81 total 61.23 last 278.00) - /unlink 26096 (time 1746130059.82 total 71.24 last 331.19) - /unlink 28662 (time 1746130069.82 total 81.24 last 256.46) - /unlink 30000 (time 1746130074.81 total 86.23 last 268.40) - /unlink 32458 (time 1746130084.82 total 96.24 last 245.54) - /unlink 35197 (time 1746130094.82 total 106.24 last 273.89) - /unlink 38344 (time 1746130104.82 total 116.24 last 314.70) - /unlink 40000 (time 1746130110.44 total 121.86 last 294.56) - /unlink 43120 (time 1746130120.44 total 131.86 last 311.94) - /unlink 46577 (time 1746130130.44 total 141.86 last 345.67) - /unlink 49742 (time 1746130140.45 total 151.87 last 316.40) - /unlink 53130 (time 1746130150.45 total 161.87 last 338.70) - /unlink 55762 (time 1746130160.45 total 171.87 last 263.19) - /unlink 58215 (time 1746130170.45 total 181.87 last 245.21) - /unlink 60000 (time 1746130176.36 total 187.78 last 302.24) - /unlink 63208 (time 1746130186.36 total 197.78 last 320.79) total: 64768 /unlink in 203.02 seconds: 319.02 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2