== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:03:25 (1746126205) Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) oleg356-server: oleg356-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: MODOPTS_LIBCFS= oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg356-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 336 65472 1% /mnt/lustre[OST:0] filesystem_summary: 66001 529 65472 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1035 (time 1746126282.06 total 10.01 last 103.44) - open/close 2054 (time 1746126292.07 total 20.02 last 101.78) - open/close 3103 (time 1746126302.07 total 30.02 last 104.89) - open/close 4145 (time 1746126312.08 total 40.03 last 104.06) - open/close 5148 (time 1746126322.08 total 50.03 last 100.27) - open/close 6246 (time 1746126332.09 total 60.04 last 109.75) - open/close 7306 (time 1746126342.09 total 70.04 last 105.98) - open/close 8400 (time 1746126352.10 total 80.05 last 109.30) - open/close 9396 (time 1746126362.12 total 90.07 last 99.42) - open/close 10000 (time 1746126367.37 total 95.32 last 115.12) - open/close 11154 (time 1746126377.37 total 105.32 last 115.33) - open/close 12327 (time 1746126387.38 total 115.33 last 117.19) - open/close 13375 (time 1746126397.39 total 125.34 last 104.74) - open/close 14532 (time 1746126407.39 total 135.34 last 115.68) - open/close 15696 (time 1746126417.40 total 145.35 last 116.31) - open/close 16867 (time 1746126427.40 total 155.35 last 117.00) - open/close 18031 (time 1746126437.41 total 165.36 last 116.33) - open/close 19262 (time 1746126447.42 total 175.37 last 123.03) - open/close 20000 (time 1746126454.37 total 182.32 last 106.18) - open/close 21163 (time 1746126464.37 total 192.32 last 116.25) - open/close 22258 (time 1746126474.37 total 202.32 last 109.49) - open/close 23368 (time 1746126484.38 total 212.33 last 110.90) - open/close 24384 (time 1746126494.39 total 222.34 last 101.47) - open/close 25441 (time 1746126504.40 total 232.35 last 105.61) - open/close 26602 (time 1746126514.40 total 242.35 last 116.06) - open/close 27910 (time 1746126524.41 total 252.36 last 130.71) - open/close 29135 (time 1746126534.42 total 262.37 last 122.42) - open/close 30000 (time 1746126541.49 total 269.44 last 122.39) - open/close 31205 (time 1746126551.49 total 279.44 last 120.44) - open/close 32325 (time 1746126561.50 total 289.45 last 111.94) - open/close 33565 (time 1746126571.51 total 299.46 last 123.86) - open/close 34756 (time 1746126581.51 total 309.46 last 119.08) - open/close 36068 (time 1746126591.52 total 319.47 last 131.09) - open/close 37233 (time 1746126601.54 total 329.49 last 116.28) - open/close 38310 (time 1746126611.54 total 339.49 last 107.61) - open/close 39326 (time 1746126621.55 total 349.50 last 101.57) - open/close 40000 (time 1746126627.56 total 355.51 last 112.17) - open/close 41167 (time 1746126637.56 total 365.51 last 116.69) - open/close 42248 (time 1746126647.56 total 375.51 last 108.09) - open/close 43417 (time 1746126657.56 total 385.51 last 116.86) - open/close 44764 (time 1746126667.56 total 395.51 last 134.68) - open/close 45993 (time 1746126677.58 total 405.53 last 122.72) - open/close 47216 (time 1746126687.58 total 415.53 last 122.26) - open/close 48422 (time 1746126697.59 total 425.54 last 120.45) - open/close 49594 (time 1746126707.60 total 435.55 last 117.16) - open/close 50000 (time 1746126710.83 total 438.78 last 125.51) - open/close 51359 (time 1746126720.84 total 448.79 last 135.83) - open/close 52653 (time 1746126730.84 total 458.79 last 129.39) - open/close 53948 (time 1746126740.84 total 468.79 last 129.49) - open/close 55379 (time 1746126750.84 total 478.79 last 143.07) - open/close 56782 (time 1746126760.84 total 488.79 last 140.25) - open/close 58063 (time 1746126770.85 total 498.80 last 128.06) - open/close 59435 (time 1746126780.85 total 508.80 last 137.14) - open/close 60000 (time 1746126784.94 total 512.89 last 138.24) - open/close 61244 (time 1746126794.95 total 522.90 last 124.26) - open/close 62437 (time 1746126804.95 total 532.90 last 119.27) - open/close 63686 (time 1746126814.96 total 542.91 last 124.77) total: 64768 open/close in 551.70 seconds: 117.40 ops/second - /unlink 2393 (time 1746126835.26 total 10.00 last 239.26) - /unlink 4813 (time 1746126845.26 total 20.00 last 241.92) - /unlink 7616 (time 1746126855.26 total 30.01 last 280.22) - /unlink 9883 (time 1746126865.27 total 40.01 last 226.56) - /unlink 12070 (time 1746126875.27 total 50.01 last 218.69) - /unlink 14251 (time 1746126885.27 total 60.02 last 218.02) - /unlink 16603 (time 1746126895.28 total 70.03 last 234.93) - /unlink 18898 (time 1746126905.29 total 80.03 last 229.40) - /unlink 20000 (time 1746126909.66 total 84.40 last 252.18) - /unlink 22827 (time 1746126919.66 total 94.41 last 282.62) - /unlink 27179 (time 1746126929.66 total 104.41 last 435.14) - /unlink 30000 (time 1746126938.94 total 113.68 last 304.11) - /unlink 32722 (time 1746126948.94 total 123.69 last 272.08) - /unlink 36334 (time 1746126958.94 total 133.69 last 361.17) - /unlink 40000 (time 1746126967.91 total 142.65 last 409.09) - /unlink 42992 (time 1746126977.91 total 152.65 last 299.13) - /unlink 45893 (time 1746126987.91 total 162.65 last 290.09) - /unlink 49403 (time 1746126997.91 total 172.66 last 350.93) - /unlink 50000 (time 1746127000.31 total 175.06 last 248.46) - /unlink 52616 (time 1746127010.31 total 185.06 last 261.59) - /unlink 55224 (time 1746127020.32 total 195.06 last 260.63) - /unlink 57867 (time 1746127030.32 total 205.07 last 264.21) - /unlink 60000 (time 1746127037.69 total 212.43 last 289.63) - /unlink 62668 (time 1746127047.69 total 222.43 last 266.75) total: 64768 /unlink in 229.43 seconds: 282.30 ops/second Waiting for MDT destroys to complete - open/close 2070 (time 1746127068.92 total 10.00 last 206.99) - open/close 3930 (time 1746127078.93 total 20.00 last 185.92) - open/close 5676 (time 1746127088.93 total 30.00 last 174.60) - open/close 7545 (time 1746127098.94 total 40.01 last 186.75) - open/close 9418 (time 1746127108.94 total 50.02 last 187.22) - open/close 10000 (time 1746127112.16 total 53.23 last 180.91) - open/close 11772 (time 1746127122.16 total 63.24 last 177.16) - open/close 13383 (time 1746127132.16 total 73.24 last 161.07) - open/close 15068 (time 1746127142.17 total 83.24 last 168.41) - open/close 16895 (time 1746127152.17 total 93.25 last 182.63) - open/close 18896 (time 1746127162.17 total 103.25 last 200.02) - open/close 20000 (time 1746127168.35 total 109.42 last 178.83) - open/close 22115 (time 1746127178.35 total 119.43 last 211.46) - open/close 24471 (time 1746127188.36 total 129.43 last 235.49) - open/close 26137 (time 1746127198.37 total 139.44 last 166.42) - open/close 27716 (time 1746127208.37 total 149.45 last 157.85) - open/close 29323 (time 1746127218.37 total 159.45 last 160.61) - open/close 30000 (time 1746127223.44 total 164.52 last 133.58) - open/close 31259 (time 1746127233.45 total 174.52 last 125.87) - open/close 32739 (time 1746127243.45 total 184.52 last 147.99) - open/close 34248 (time 1746127253.45 total 194.53 last 150.76) - open/close 35830 (time 1746127263.46 total 204.54 last 158.12) - open/close 37327 (time 1746127273.46 total 214.54 last 149.63) - open/close 38737 (time 1746127283.47 total 224.55 last 140.88) - open/close 40000 (time 1746127290.55 total 231.62 last 178.51) - open/close 41396 (time 1746127300.55 total 241.63 last 139.57) - open/close 42748 (time 1746127310.56 total 251.63 last 135.12) - open/close 44446 (time 1746127320.56 total 261.64 last 169.76) - open/close 46648 (time 1746127330.56 total 271.64 last 220.18) - open/close 48779 (time 1746127340.56 total 281.64 last 213.08) - open/close 50000 (time 1746127346.40 total 287.48 last 209.06) - open/close 51680 (time 1746127356.41 total 297.48 last 167.90) - open/close 53134 (time 1746127366.41 total 307.49 last 145.36) - open/close 54735 (time 1746127376.41 total 317.49 last 160.07) - open/close 56304 (time 1746127386.42 total 327.49 last 156.81) - open/close 57999 (time 1746127396.42 total 337.50 last 169.48) - open/close 59696 (time 1746127406.42 total 347.50 last 169.60) - open/close 61703 (time 1746127416.43 total 357.51 last 200.61) - open/close 63799 (time 1746127426.43 total 367.51 last 209.54) total: 64768 open/close in 373.88 seconds: 173.23 ops/second - /unlink 3901 (time 1746127444.07 total 10.00 last 390.01) - /unlink 6927 (time 1746127454.07 total 20.01 last 302.51) - /unlink 9780 (time 1746127464.08 total 30.01 last 285.11) - /unlink 12587 (time 1746127474.08 total 40.01 last 280.61) - /unlink 15406 (time 1746127484.09 total 50.02 last 281.81) - /unlink 18210 (time 1746127494.09 total 60.02 last 280.33) - /unlink 20000 (time 1746127499.43 total 65.36 last 335.25) - /unlink 23383 (time 1746127509.43 total 75.36 last 338.30) - /unlink 26901 (time 1746127519.43 total 85.37 last 351.59) - /unlink 30000 (time 1746127527.40 total 93.33 last 388.96) - /unlink 33424 (time 1746127537.40 total 103.33 last 342.36) - /unlink 36432 (time 1746127547.41 total 113.34 last 300.68) - /unlink 39367 (time 1746127557.41 total 123.34 last 293.49) - /unlink 40000 (time 1746127559.98 total 125.91 last 246.22) - /unlink 42462 (time 1746127569.98 total 135.91 last 246.12) - /unlink 44786 (time 1746127579.99 total 145.92 last 232.30) - /unlink 47163 (time 1746127589.99 total 155.92 last 237.60) - /unlink 49506 (time 1746127599.99 total 165.92 last 234.25) - /unlink 52096 (time 1746127610.00 total 175.93 last 258.90) - /unlink 54805 (time 1746127620.00 total 185.93 last 270.90) - /unlink 57703 (time 1746127630.00 total 195.93 last 289.72) - /unlink 60000 (time 1746127638.57 total 204.50 last 267.92) - /unlink 62894 (time 1746127648.58 total 214.51 last 289.27) total: 64768 /unlink in 221.48 seconds: 292.43 ops/second Waiting for MDT destroys to complete - open/close 1287 (time 1746127676.13 total 10.00 last 128.68) - open/close 2774 (time 1746127686.13 total 20.00 last 148.67) - open/close 4210 (time 1746127696.13 total 30.00 last 143.59) - open/close 5831 (time 1746127706.13 total 40.00 last 162.08) - open/close 7258 (time 1746127716.14 total 50.01 last 142.64) - open/close 8537 (time 1746127726.15 total 60.02 last 127.80) - open/close 9944 (time 1746127736.15 total 70.02 last 140.62) - open/close 11289 (time 1746127746.16 total 80.03 last 134.41) - open/close 12550 (time 1746127756.16 total 90.03 last 126.05) - open/close 13886 (time 1746127766.17 total 100.04 last 133.54) - open/close 15140 (time 1746127776.17 total 110.04 last 125.37) - open/close 16498 (time 1746127786.17 total 120.04 last 135.75) - open/close 17687 (time 1746127796.18 total 130.05 last 118.85) - open/close 19021 (time 1746127806.18 total 140.05 last 133.38) - open/close 20000 (time 1746127814.32 total 148.19 last 120.21) - open/close 21219 (time 1746127824.32 total 158.19 last 121.89) - open/close 22493 (time 1746127834.32 total 168.20 last 127.38) - open/close 23809 (time 1746127844.33 total 178.20 last 131.57) - open/close 25072 (time 1746127854.33 total 188.20 last 126.24) - open/close 26274 (time 1746127864.34 total 198.21 last 120.07) - open/close 27849 (time 1746127874.35 total 208.22 last 157.38) - open/close 29408 (time 1746127884.35 total 218.22 last 155.89) - open/close 30000 (time 1746127888.56 total 222.43 last 140.76) - open/close 31482 (time 1746127898.56 total 232.43 last 148.19) - open/close 32837 (time 1746127908.56 total 242.43 last 135.43) - open/close 34224 (time 1746127918.56 total 252.43 last 138.70) - open/close 35628 (time 1746127928.57 total 262.44 last 140.36) - open/close 36951 (time 1746127938.57 total 272.44 last 132.25) - open/close 38320 (time 1746127948.57 total 282.44 last 136.88) - open/close 39598 (time 1746127958.58 total 292.45 last 127.74) - open/close 40000 (time 1746127962.21 total 296.08 last 110.61) - open/close 41278 (time 1746127972.21 total 306.08 last 127.76) - open/close 42533 (time 1746127982.22 total 316.09 last 125.47) - open/close 43876 (time 1746127992.22 total 326.09 last 134.28) - open/close 45203 (time 1746128002.22 total 336.09 last 132.65) - open/close 46354 (time 1746128012.23 total 346.10 last 115.02) - open/close 47715 (time 1746128022.23 total 356.10 last 136.09) - open/close 49108 (time 1746128032.24 total 366.11 last 139.21) - open/close 50000 (time 1746128038.73 total 372.60 last 137.36) - open/close 51263 (time 1746128048.73 total 382.60 last 126.28) - open/close 52563 (time 1746128058.73 total 392.60 last 129.98) - open/close 53881 (time 1746128068.73 total 402.60 last 131.80) - open/close 55107 (time 1746128078.74 total 412.61 last 122.48) - open/close 56351 (time 1746128088.75 total 422.62 last 124.35) - open/close 57543 (time 1746128098.76 total 432.63 last 119.08) - open/close 58688 (time 1746128108.76 total 442.63 last 114.49) - open/close 59755 (time 1746128118.76 total 452.63 last 106.66) - open/close 60000 (time 1746128121.09 total 454.96 last 105.35) - open/close 61340 (time 1746128131.09 total 464.96 last 133.98) - open/close 62503 (time 1746128141.09 total 474.96 last 116.25) - open/close 63622 (time 1746128151.10 total 484.97 last 111.89) - open/close 64733 (time 1746128161.10 total 494.97 last 111.07) total: 64768 open/close in 495.23 seconds: 130.78 ops/second - /unlink 1758 (time 1746128173.95 total 10.00 last 175.76) - /unlink 3569 (time 1746128183.96 total 20.01 last 181.00) - /unlink 5451 (time 1746128193.96 total 30.01 last 188.18) - /unlink 7292 (time 1746128203.96 total 40.01 last 184.07) - /unlink 9214 (time 1746128213.96 total 50.01 last 192.18) - /unlink 10000 (time 1746128218.67 total 54.72 last 166.88) - /unlink 11746 (time 1746128228.68 total 64.73 last 174.49) - /unlink 13774 (time 1746128238.68 total 74.73 last 202.68) - /unlink 15743 (time 1746128248.69 total 84.74 last 196.85) - /unlink 17617 (time 1746128258.70 total 94.75 last 187.20) - /unlink 19639 (time 1746128268.70 total 104.75 last 202.14) - /unlink 21925 (time 1746128278.70 total 114.75 last 228.60) - /unlink 23791 (time 1746128288.70 total 124.75 last 186.59) - /unlink 25676 (time 1746128298.71 total 134.76 last 188.40) - /unlink 27638 (time 1746128308.71 total 144.76 last 196.07) - /unlink 29693 (time 1746128318.71 total 154.76 last 205.49) - /unlink 31755 (time 1746128328.72 total 164.77 last 206.14) - /unlink 33701 (time 1746128338.72 total 174.77 last 194.54) - /unlink 35907 (time 1746128348.72 total 184.77 last 220.60) - /unlink 38131 (time 1746128358.72 total 194.77 last 222.37) - /unlink 40000 (time 1746128367.71 total 203.76 last 207.90) - /unlink 42326 (time 1746128377.78 total 213.83 last 230.97) - /unlink 44424 (time 1746128387.78 total 223.83 last 209.76) - /unlink 46719 (time 1746128397.79 total 233.83 last 229.46) - /unlink 48997 (time 1746128407.79 total 243.84 last 227.75) - /unlink 50000 (time 1746128411.72 total 247.76 last 255.37) - /unlink 52509 (time 1746128421.72 total 257.77 last 250.77) - /unlink 55166 (time 1746128431.72 total 267.77 last 265.68) - /unlink 57632 (time 1746128441.72 total 277.77 last 246.59) - /unlink 60000 (time 1746128451.18 total 287.23 last 250.29) - /unlink 62609 (time 1746128461.18 total 297.23 last 260.87) total: 64768 /unlink in 306.35 seconds: 211.42 ops/second Waiting for MDT destroys to complete - open/close 1370 (time 1746128488.45 total 10.00 last 136.98) - open/close 2865 (time 1746128498.45 total 20.00 last 149.46) - open/close 4319 (time 1746128508.45 total 30.01 last 145.38) - open/close 5670 (time 1746128518.47 total 40.02 last 134.87) - open/close 7111 (time 1746128528.48 total 50.03 last 144.01) - open/close 8483 (time 1746128538.48 total 60.04 last 137.11) - open/close 9917 (time 1746128548.49 total 70.04 last 143.36) - open/close 11274 (time 1746128558.49 total 80.04 last 135.64) - open/close 12682 (time 1746128568.49 total 90.04 last 140.80) - open/close 13961 (time 1746128578.50 total 100.05 last 127.84) - open/close 15425 (time 1746128588.50 total 110.05 last 146.39) - open/close 16653 (time 1746128598.50 total 120.05 last 122.73) - open/close 17693 (time 1746128608.51 total 130.06 last 103.91) - open/close 19003 (time 1746128618.52 total 140.07 last 130.93) - open/close 20000 (time 1746128625.79 total 147.34 last 137.16) - open/close 21214 (time 1746128635.79 total 157.34 last 121.38) - open/close 22478 (time 1746128645.80 total 167.35 last 126.31) - open/close 23755 (time 1746128655.80 total 177.35 last 127.69) - open/close 25101 (time 1746128665.80 total 187.35 last 134.56) - open/close 26561 (time 1746128675.80 total 197.35 last 145.95) - open/close 28106 (time 1746128685.81 total 207.36 last 154.45) - open/close 29559 (time 1746128695.81 total 217.36 last 145.23) - open/close 30000 (time 1746128698.86 total 220.41 last 144.79) - open/close 31367 (time 1746128708.86 total 230.41 last 136.67) - open/close 32686 (time 1746128718.86 total 240.41 last 131.82) - open/close 33969 (time 1746128728.87 total 250.42 last 128.19) - open/close 35335 (time 1746128738.87 total 260.42 last 136.58) - open/close 36634 (time 1746128748.88 total 270.43 last 129.80) - open/close 38059 (time 1746128758.88 total 280.43 last 142.49) - open/close 39347 (time 1746128768.89 total 290.45 last 128.64) - open/close 40000 (time 1746128774.08 total 295.63 last 125.83) - open/close 41150 (time 1746128784.09 total 305.64 last 114.96) - open/close 42331 (time 1746128794.09 total 315.64 last 118.09) - open/close 43707 (time 1746128804.09 total 325.64 last 137.56) - open/close 45071 (time 1746128814.10 total 335.65 last 136.23) - open/close 46390 (time 1746128824.11 total 345.66 last 131.78) - open/close 47627 (time 1746128834.12 total 355.67 last 123.64) - open/close 48762 (time 1746128844.12 total 365.67 last 113.49) - open/close 50000 (time 1746128853.40 total 374.95 last 133.38) - open/close 51380 (time 1746128863.40 total 384.96 last 137.94) - open/close 52692 (time 1746128873.41 total 394.96 last 131.18) - open/close 53761 (time 1746128883.41 total 404.96 last 106.82) - open/close 55020 (time 1746128893.42 total 414.97 last 125.85) - open/close 56191 (time 1746128903.42 total 424.97 last 117.06) - open/close 57374 (time 1746128913.42 total 434.98 last 118.25) - open/close 58477 (time 1746128923.43 total 444.98 last 110.26) - open/close 59548 (time 1746128933.43 total 454.98 last 107.09) - open/close 60000 (time 1746128937.25 total 458.80 last 118.40) - open/close 61209 (time 1746128947.25 total 468.80 last 120.90) - open/close 62295 (time 1746128957.26 total 478.81 last 108.44) - open/close 63530 (time 1746128967.26 total 488.81 last 123.49) - open/close 64740 (time 1746128977.27 total 498.82 last 120.89) total: 64768 open/close in 499.10 seconds: 129.77 ops/second - /unlink 1918 (time 1746128990.03 total 10.00 last 191.79) - /unlink 3805 (time 1746129000.03 total 20.00 last 188.69) - /unlink 5837 (time 1746129010.04 total 30.00 last 203.17) - /unlink 7854 (time 1746129020.04 total 40.01 last 201.62) - /unlink 9809 (time 1746129030.04 total 50.01 last 195.48) - /unlink 11692 (time 1746129040.05 total 60.02 last 188.15) - /unlink 13642 (time 1746129050.05 total 70.02 last 194.94) - /unlink 15467 (time 1746129060.05 total 80.02 last 182.46) - /unlink 17413 (time 1746129070.05 total 90.02 last 194.58) - /unlink 19363 (time 1746129080.06 total 100.03 last 194.80) - /unlink 20000 (time 1746129083.14 total 103.11 last 206.94) - /unlink 22300 (time 1746129093.14 total 113.11 last 229.97) - /unlink 24605 (time 1746129103.15 total 123.11 last 230.45) - /unlink 26776 (time 1746129113.15 total 133.12 last 217.02) - /unlink 28961 (time 1746129123.16 total 143.12 last 218.38) - /unlink 30000 (time 1746129128.54 total 148.51 last 192.89) - /unlink 32358 (time 1746129138.54 total 158.51 last 235.77) - /unlink 34773 (time 1746129148.55 total 168.51 last 241.45) - /unlink 37536 (time 1746129158.55 total 178.51 last 276.27) - /unlink 39763 (time 1746129168.55 total 188.52 last 222.64) - /unlink 42074 (time 1746129178.55 total 198.52 last 231.05) - /unlink 43958 (time 1746129188.55 total 208.52 last 188.38) - /unlink 46001 (time 1746129198.55 total 218.52 last 204.30) - /unlink 48156 (time 1746129208.55 total 228.52 last 215.45) - /unlink 50000 (time 1746129216.53 total 236.49 last 231.30) - /unlink 52378 (time 1746129226.53 total 246.50 last 237.74) - /unlink 55139 (time 1746129236.53 total 256.50 last 275.99) - /unlink 57649 (time 1746129246.53 total 266.50 last 250.99) - /unlink 59985 (time 1746129256.54 total 276.50 last 233.52) - /unlink 62504 (time 1746129266.54 total 286.51 last 251.84) total: 64768 /unlink in 296.06 seconds: 218.77 ops/second Waiting for MDT destroys to complete - open/close 1299 (time 1746129296.02 total 10.01 last 129.82) - open/close 2557 (time 1746129306.03 total 20.01 last 125.72) - open/close 4062 (time 1746129316.03 total 30.02 last 150.41) - open/close 5623 (time 1746129326.04 total 40.02 last 156.00) - open/close 7254 (time 1746129336.04 total 50.03 last 163.03) - open/close 8700 (time 1746129346.04 total 60.03 last 144.59) - open/close 10000 (time 1746129355.33 total 69.31 last 140.08) - open/close 11430 (time 1746129365.33 total 79.31 last 142.97) - open/close 12916 (time 1746129375.33 total 89.31 last 148.58) - open/close 14577 (time 1746129385.33 total 99.31 last 166.08) - open/close 16223 (time 1746129395.33 total 109.32 last 164.56) - open/close 17930 (time 1746129405.34 total 119.32 last 170.62) - open/close 19857 (time 1746129415.34 total 129.33 last 192.62) - open/close 21468 (time 1746129425.35 total 139.33 last 161.01) - open/close 23237 (time 1746129435.35 total 149.33 last 176.83) - open/close 24872 (time 1746129445.35 total 159.34 last 163.47) - open/close 26668 (time 1746129455.35 total 169.34 last 179.58) - open/close 28484 (time 1746129465.36 total 179.34 last 181.52) - open/close 30000 (time 1746129473.56 total 187.55 last 184.75) - open/close 31872 (time 1746129483.57 total 197.55 last 187.14) - open/close 33326 (time 1746129493.57 total 207.56 last 145.31) - open/close 34959 (time 1746129503.57 total 217.56 last 163.26) - open/close 36652 (time 1746129513.58 total 227.56 last 169.21) - open/close 38480 (time 1746129523.59 total 237.57 last 182.67) - open/close 40000 (time 1746129533.28 total 247.27 last 156.79) - open/close 41618 (time 1746129543.28 total 257.27 last 161.80) - open/close 43133 (time 1746129553.29 total 267.27 last 151.41) - open/close 44496 (time 1746129563.29 total 277.28 last 136.23) - open/close 45870 (time 1746129573.30 total 287.28 last 137.35) - open/close 47281 (time 1746129583.30 total 297.29 last 141.01) - open/close 48732 (time 1746129593.30 total 307.29 last 145.08) - open/close 50000 (time 1746129602.46 total 316.45 last 138.42) - open/close 51508 (time 1746129612.47 total 326.45 last 150.75) - open/close 52881 (time 1746129622.47 total 336.45 last 137.26) - open/close 54066 (time 1746129632.49 total 346.48 last 118.22) - open/close 55125 (time 1746129642.50 total 356.49 last 105.80) - open/close 56260 (time 1746129652.50 total 366.49 last 113.49) - open/close 57560 (time 1746129662.51 total 376.49 last 129.92) - open/close 59109 (time 1746129672.51 total 386.50 last 154.85) - open/close 60000 (time 1746129678.41 total 392.39 last 151.14) - open/close 61482 (time 1746129688.42 total 402.41 last 148.01) - open/close 62744 (time 1746129698.43 total 412.42 last 126.05) - open/close 63932 (time 1746129708.44 total 422.42 last 118.77) total: 64768 open/close in 428.92 seconds: 151.00 ops/second - /unlink 2306 (time 1746129726.92 total 10.00 last 230.56) - /unlink 4579 (time 1746129736.92 total 20.00 last 227.28) - /unlink 6961 (time 1746129746.92 total 30.01 last 238.14) - /unlink 9494 (time 1746129756.92 total 40.01 last 253.30) - /unlink 11840 (time 1746129766.92 total 50.01 last 234.57) - /unlink 14006 (time 1746129776.93 total 60.01 last 216.52) - /unlink 16192 (time 1746129786.93 total 70.02 last 218.42) - /unlink 18371 (time 1746129796.93 total 80.02 last 217.89) - /unlink 20000 (time 1746129805.34 total 88.42 last 193.82) - /unlink 22489 (time 1746129815.34 total 98.42 last 248.87) - /unlink 25276 (time 1746129825.34 total 108.43 last 278.66) - /unlink 28036 (time 1746129835.34 total 118.43 last 275.99) - /unlink 30000 (time 1746129843.55 total 126.63 last 239.41) - /unlink 32776 (time 1746129853.55 total 136.63 last 277.59) - /unlink 35014 (time 1746129863.56 total 146.64 last 223.55) - /unlink 37326 (time 1746129873.56 total 156.64 last 231.15) - /unlink 39671 (time 1746129883.56 total 166.64 last 234.47) - /unlink 41647 (time 1746129893.56 total 176.65 last 197.57) - /unlink 43515 (time 1746129903.57 total 186.65 last 186.72) - /unlink 45712 (time 1746129913.57 total 196.65 last 219.64) - /unlink 47763 (time 1746129923.57 total 206.65 last 205.06) - /unlink 50000 (time 1746129933.39 total 216.47 last 227.86) - /unlink 52314 (time 1746129943.39 total 226.48 last 231.28) - /unlink 54585 (time 1746129953.40 total 236.48 last 226.94) - /unlink 57009 (time 1746129963.40 total 246.49 last 242.37) - /unlink 59549 (time 1746129973.40 total 256.49 last 253.93) - /unlink 62071 (time 1746129983.41 total 266.49 last 252.19) - /unlink 64280 (time 1746129993.41 total 276.49 last 220.87) total: 64768 /unlink in 278.43 seconds: 232.62 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules via unload_modules_local on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2