== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:23:29 (1761283409) Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) oleg141-server: oleg141-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg141-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg141-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 279 79713 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 257 79735 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65976 536 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3815 (time 1761283461.18 total 10.00 last 381.48) - open/close 7081 (time 1761283471.18 total 20.00 last 326.53) - open/close 9428 (time 1761283481.18 total 30.01 last 234.62) - open/close 10000 (time 1761283483.41 total 32.23 last 256.73) - open/close 13037 (time 1761283493.41 total 42.24 last 303.63) - open/close 16777 (time 1761283503.41 total 52.24 last 373.99) - open/close 20000 (time 1761283511.28 total 60.10 last 409.97) - open/close 23523 (time 1761283521.28 total 70.10 last 352.26) - open/close 26688 (time 1761283531.28 total 80.10 last 316.47) - open/close 30000 (time 1761283539.56 total 88.39 last 399.72) - open/close 34447 (time 1761283549.57 total 98.39 last 444.66) - open/close 38618 (time 1761283559.57 total 108.39 last 417.00) - open/close 40000 (time 1761283563.01 total 111.83 last 401.71) - open/close 43619 (time 1761283573.01 total 121.83 last 361.90) - open/close 47439 (time 1761283583.01 total 131.83 last 381.93) - open/close 50000 (time 1761283591.22 total 140.04 last 311.95) - open/close 53159 (time 1761283601.22 total 150.04 last 315.87) - open/close 56410 (time 1761283611.22 total 160.05 last 324.98) - open/close 59877 (time 1761283621.23 total 170.05 last 346.66) - open/close 63385 (time 1761283631.23 total 180.05 last 350.80) total: 64768 open/close in 183.31 seconds: 353.33 ops/second - /unlink 6841 (time 1761283645.88 total 10.00 last 684.06) - /unlink 10000 (time 1761283650.54 total 14.66 last 678.07) - /unlink 17075 (time 1761283660.54 total 24.66 last 707.49) - /unlink 20000 (time 1761283664.05 total 28.17 last 833.30) - /unlink 28366 (time 1761283674.05 total 38.17 last 836.46) - /unlink 30000 (time 1761283676.07 total 40.20 last 807.01) - /unlink 37809 (time 1761283686.07 total 50.20 last 780.83) - /unlink 40000 (time 1761283689.30 total 53.42 last 680.23) - /unlink 47131 (time 1761283699.30 total 63.42 last 713.00) - /unlink 50000 (time 1761283703.08 total 67.20 last 759.18) - /unlink 56585 (time 1761283713.08 total 77.20 last 658.49) - /unlink 60000 (time 1761283717.59 total 81.71 last 757.09) total: 64768 /unlink in 88.93 seconds: 728.33 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 3613 (time 1761283763.50 total 10.00 last 361.29) - open/close 7092 (time 1761283773.50 total 20.00 last 347.83) - open/close 10000 (time 1761283781.57 total 28.07 last 360.38) - open/close 13451 (time 1761283791.57 total 38.07 last 345.05) - open/close 16872 (time 1761283801.57 total 48.07 last 342.08) - open/close 20000 (time 1761283809.08 total 55.59 last 416.31) - open/close 23876 (time 1761283819.09 total 65.59 last 387.51) - open/close 27799 (time 1761283829.09 total 75.59 last 392.26) - open/close 30000 (time 1761283834.56 total 81.06 last 402.47) - open/close 34347 (time 1761283844.56 total 91.06 last 434.61) - open/close 38301 (time 1761283854.56 total 101.06 last 395.35) - open/close 40000 (time 1761283858.98 total 105.48 last 384.33) - open/close 43584 (time 1761283868.98 total 115.48 last 358.38) - open/close 47609 (time 1761283878.98 total 125.48 last 402.49) - open/close 50000 (time 1761283885.12 total 131.62 last 389.51) - open/close 53811 (time 1761283895.12 total 141.62 last 381.04) - open/close 58269 (time 1761283905.12 total 151.63 last 445.74) - open/close 60000 (time 1761283909.71 total 156.21 last 377.72) - open/close 64206 (time 1761283919.71 total 166.21 last 420.57) total: 64768 open/close in 167.29 seconds: 387.16 ops/second - /unlink 7768 (time 1761283932.12 total 10.00 last 776.76) - /unlink 10000 (time 1761283934.85 total 12.73 last 817.38) - /unlink 17726 (time 1761283944.85 total 22.73 last 772.55) - /unlink 20000 (time 1761283947.90 total 25.79 last 744.44) - /unlink 27727 (time 1761283957.90 total 35.79 last 772.70) - /unlink 30000 (time 1761283960.52 total 38.40 last 869.59) - /unlink 38517 (time 1761283970.52 total 48.40 last 851.67) - /unlink 47271 (time 1761283980.53 total 58.41 last 874.54) - /unlink 50000 (time 1761283983.61 total 61.50 last 884.07) - /unlink 58216 (time 1761283993.61 total 71.50 last 821.56) - /unlink 60000 (time 1761283995.61 total 73.50 last 891.92) total: 64768 /unlink in 78.75 seconds: 822.47 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 3389 (time 1761284061.22 total 10.00 last 338.86) - open/close 6746 (time 1761284071.22 total 20.00 last 335.68) - open/close 10000 (time 1761284079.97 total 28.76 last 371.58) - open/close 13821 (time 1761284089.98 total 38.76 last 382.07) - open/close 17466 (time 1761284099.98 total 48.76 last 364.40) - open/close 20000 (time 1761284107.59 total 56.38 last 332.74) - open/close 23664 (time 1761284117.59 total 66.38 last 366.36) - open/close 27873 (time 1761284127.60 total 76.38 last 420.86) - open/close 30000 (time 1761284132.47 total 81.25 last 436.48) - open/close 34739 (time 1761284142.47 total 91.25 last 473.85) - open/close 39664 (time 1761284152.47 total 101.26 last 492.37) - open/close 44585 (time 1761284162.47 total 111.26 last 492.04) - open/close 49323 (time 1761284172.47 total 121.26 last 473.74) - open/close 54414 (time 1761284182.48 total 131.26 last 509.02) - open/close 59426 (time 1761284192.48 total 141.26 last 501.18) - open/close 64601 (time 1761284202.48 total 151.26 last 517.43) total: 64768 open/close in 151.60 seconds: 427.24 ops/second - /unlink 8887 (time 1761284214.13 total 10.00 last 888.67) - /unlink 17078 (time 1761284224.13 total 20.01 last 818.66) - /unlink 20000 (time 1761284227.72 total 23.59 last 815.85) - /unlink 27845 (time 1761284237.72 total 33.59 last 784.48) - /unlink 30000 (time 1761284240.48 total 36.35 last 779.40) - /unlink 38857 (time 1761284250.48 total 46.35 last 885.63) - /unlink 47752 (time 1761284260.48 total 56.35 last 889.48) - /unlink 50000 (time 1761284262.89 total 58.76 last 933.22) - /unlink 59272 (time 1761284272.89 total 68.76 last 927.18) total: 64768 /unlink in 75.02 seconds: 863.36 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 861 (time 1761284379.69 total 10.01 last 86.05) - open/close 1763 (time 1761284389.71 total 20.02 last 90.09) - open/close 2751 (time 1761284399.71 total 30.02 last 98.78) - open/close 3820 (time 1761284409.71 total 40.02 last 106.88) - open/close 5055 (time 1761284419.72 total 50.03 last 123.37) - open/close 6030 (time 1761284429.72 total 60.04 last 97.48) - open/close 6990 (time 1761284439.73 total 70.04 last 95.92) - open/close 7951 (time 1761284449.74 total 80.05 last 96.04) - open/close 9024 (time 1761284459.74 total 90.05 last 107.30) - open/close 10000 (time 1761284467.28 total 97.60 last 129.36) - open/close 11370 (time 1761284477.29 total 107.60 last 136.96) - open/close 12939 (time 1761284487.29 total 117.60 last 156.84) - open/close 14149 (time 1761284497.30 total 127.61 last 120.94) - open/close 15251 (time 1761284507.30 total 137.62 last 110.10) - open/close 16257 (time 1761284517.31 total 147.62 last 100.57) - open/close 17042 (time 1761284527.31 total 157.62 last 78.46) - open/close 17773 (time 1761284537.32 total 167.63 last 73.05) - open/close 18617 (time 1761284547.32 total 177.63 last 84.40) - open/close 19485 (time 1761284557.33 total 187.64 last 86.74) - open/close 20000 (time 1761284563.00 total 193.31 last 90.77) - open/close 20891 (time 1761284573.01 total 203.32 last 89.03) - open/close 21770 (time 1761284583.01 total 213.33 last 87.85) - open/close 22674 (time 1761284593.02 total 223.33 last 90.35) - open/close 23549 (time 1761284603.02 total 233.34 last 87.47) - open/close 24419 (time 1761284613.03 total 243.34 last 86.96) - open/close 25294 (time 1761284623.05 total 253.36 last 87.35) - open/close 26194 (time 1761284633.06 total 263.37 last 89.87) - open/close 27051 (time 1761284643.07 total 273.38 last 85.62) - open/close 27957 (time 1761284653.08 total 283.39 last 90.52) - open/close 28950 (time 1761284663.08 total 293.39 last 99.27) - open/close 29864 (time 1761284673.09 total 303.40 last 91.36) - open/close 30940 (time 1761284683.10 total 313.41 last 107.48) - open/close 31941 (time 1761284693.10 total 323.41 last 100.07) - open/close 32949 (time 1761284703.11 total 333.42 last 100.72) - open/close 33888 (time 1761284713.11 total 343.42 last 93.89) - open/close 35170 (time 1761284723.11 total 353.42 last 128.19) - open/close 36466 (time 1761284733.11 total 363.43 last 129.52) - open/close 37278 (time 1761284743.12 total 373.43 last 81.19) - open/close 37913 (time 1761284753.12 total 383.43 last 63.49) - open/close 38640 (time 1761284763.14 total 393.45 last 72.55) - open/close 39485 (time 1761284773.14 total 403.45 last 84.49) - open/close 40000 (time 1761284780.04 total 410.35 last 74.64) - open/close 40913 (time 1761284790.05 total 420.36 last 91.22) - open/close 41765 (time 1761284800.05 total 430.36 last 85.16) - open/close 42677 (time 1761284810.06 total 440.37 last 91.17) - open/close 43577 (time 1761284820.07 total 450.38 last 89.89) - open/close 44549 (time 1761284830.07 total 460.39 last 97.15) - open/close 45473 (time 1761284840.08 total 470.39 last 92.35) - open/close 46503 (time 1761284850.09 total 480.40 last 102.91) - open/close 47422 (time 1761284860.09 total 490.41 last 91.84) - open/close 48320 (time 1761284870.10 total 500.41 last 89.76) - open/close 49204 (time 1761284880.10 total 510.41 last 88.39) - open/close 50000 (time 1761284888.64 total 518.95 last 93.19) - open/close 50922 (time 1761284898.65 total 528.96 last 92.13) - open/close 51775 (time 1761284908.65 total 538.96 last 85.28) - open/close 52659 (time 1761284918.66 total 548.97 last 88.35) - open/close 53616 (time 1761284928.67 total 558.98 last 95.61) - open/close 54657 (time 1761284938.68 total 568.99 last 104.01) - open/close 55562 (time 1761284948.68 total 578.99 last 90.50) - open/close 56439 (time 1761284958.68 total 588.99 last 87.65) - open/close 57336 (time 1761284968.69 total 599.00 last 89.65) - open/close 58204 (time 1761284978.70 total 609.01 last 86.72) - open/close 59085 (time 1761284988.72 total 619.03 last 87.92) - open/close 59983 (time 1761284998.72 total 629.03 last 89.75) - open/close 60901 (time 1761285008.74 total 639.05 last 91.67) - open/close 61787 (time 1761285018.74 total 649.05 last 88.59) - open/close 62802 (time 1761285028.74 total 659.05 last 101.46) - open/close 63750 (time 1761285038.75 total 669.06 last 94.71) - open/close 64684 (time 1761285048.76 total 679.07 last 93.32) total: 64768 open/close in 680.09 seconds: 95.23 ops/second - /unlink 828 (time 1761285062.81 total 10.00 last 82.77) - /unlink 1638 (time 1761285072.81 total 20.01 last 80.95) - /unlink 2446 (time 1761285082.82 total 30.02 last 80.72) - /unlink 3330 (time 1761285092.83 total 40.02 last 88.36) - /unlink 4200 (time 1761285102.83 total 50.03 last 86.95) - /unlink 5051 (time 1761285112.84 total 60.03 last 85.07) - /unlink 5908 (time 1761285122.84 total 70.04 last 85.65) - /unlink 6702 (time 1761285132.84 total 80.04 last 79.40) - /unlink 7538 (time 1761285142.86 total 90.06 last 83.45) - /unlink 8407 (time 1761285152.87 total 100.06 last 86.84) - /unlink 9291 (time 1761285162.87 total 110.07 last 88.34) - /unlink 10000 (time 1761285171.27 total 118.47 last 84.43) - /unlink 11002 (time 1761285181.27 total 128.47 last 100.17) - /unlink 11962 (time 1761285191.28 total 138.48 last 95.94) - /unlink 12904 (time 1761285201.29 total 148.48 last 94.14) - /unlink 13856 (time 1761285211.29 total 158.49 last 95.17) - /unlink 14778 (time 1761285221.30 total 168.49 last 92.15) - /unlink 15605 (time 1761285231.30 total 178.50 last 82.67) - /unlink 16470 (time 1761285241.30 total 188.50 last 86.48) - /unlink 17356 (time 1761285251.31 total 198.51 last 88.49) - /unlink 18343 (time 1761285261.32 total 208.52 last 98.65) - /unlink 19320 (time 1761285271.32 total 218.52 last 97.66) - /unlink 20000 (time 1761285278.74 total 225.94 last 91.70) - /unlink 20934 (time 1761285288.75 total 235.94 last 93.33) - /unlink 21776 (time 1761285298.76 total 245.96 last 84.11) - /unlink 22711 (time 1761285308.76 total 255.96 last 93.48) - /unlink 23590 (time 1761285318.76 total 265.96 last 87.87) - /unlink 24415 (time 1761285328.77 total 275.97 last 82.43) - /unlink 25209 (time 1761285338.77 total 285.97 last 79.38) - /unlink 26033 (time 1761285348.78 total 295.98 last 82.35) - /unlink 26874 (time 1761285358.78 total 305.98 last 84.07) - /unlink 27740 (time 1761285368.79 total 315.99 last 86.51) - /unlink 28607 (time 1761285378.80 total 325.99 last 86.68) - /unlink 29450 (time 1761285388.82 total 336.01 last 84.13) - /unlink 30000 (time 1761285395.41 total 342.61 last 83.42) - /unlink 30826 (time 1761285405.41 total 352.61 last 82.59) - /unlink 31759 (time 1761285415.42 total 362.61 last 93.25) - /unlink 32614 (time 1761285425.43 total 372.63 last 85.37) - /unlink 33460 (time 1761285435.44 total 382.64 last 84.50) - /unlink 34314 (time 1761285445.45 total 392.64 last 85.37) - /unlink 35174 (time 1761285455.45 total 402.65 last 85.95) - /unlink 36037 (time 1761285465.46 total 412.65 last 86.27) - /unlink 36892 (time 1761285475.47 total 422.66 last 85.41) - /unlink 37735 (time 1761285485.47 total 432.66 last 84.30) - /unlink 38655 (time 1761285495.47 total 442.67 last 91.96) - /unlink 39653 (time 1761285505.48 total 452.67 last 99.76) - /unlink 40000 (time 1761285509.19 total 456.39 last 93.40) - /unlink 41168 (time 1761285519.20 total 466.40 last 116.73) - /unlink 42166 (time 1761285529.21 total 476.40 last 99.72) - /unlink 43087 (time 1761285539.21 total 486.40 last 92.09) - /unlink 43969 (time 1761285549.21 total 496.41 last 88.15) - /unlink 44940 (time 1761285559.23 total 506.43 last 96.95) - /unlink 45852 (time 1761285569.23 total 516.43 last 91.18) - /unlink 46907 (time 1761285579.24 total 526.43 last 105.43) - /unlink 47942 (time 1761285589.25 total 536.45 last 103.33) - /unlink 48884 (time 1761285599.26 total 546.46 last 94.16) - /unlink 49907 (time 1761285609.27 total 556.47 last 102.18) - /unlink 50958 (time 1761285619.27 total 566.47 last 105.08) - /unlink 51920 (time 1761285629.28 total 576.48 last 96.13) - /unlink 52877 (time 1761285639.28 total 586.48 last 95.67) - /unlink 53894 (time 1761285649.29 total 596.49 last 101.62) - /unlink 54865 (time 1761285659.30 total 606.49 last 97.04) - /unlink 55694 (time 1761285669.30 total 616.50 last 82.83) - /unlink 56567 (time 1761285679.31 total 626.51 last 87.27) - /unlink 57432 (time 1761285689.31 total 636.51 last 86.46) - /unlink 58344 (time 1761285699.32 total 646.52 last 91.12) - /unlink 59196 (time 1761285709.33 total 656.53 last 85.10) - /unlink 60000 (time 1761285718.64 total 665.83 last 86.43) - /unlink 60828 (time 1761285728.64 total 675.83 last 82.79) - /unlink 61680 (time 1761285738.65 total 685.85 last 85.05) - /unlink 62512 (time 1761285748.66 total 695.86 last 83.13) - /unlink 63400 (time 1761285758.67 total 705.86 last 88.76) - /unlink 64293 (time 1761285768.67 total 715.87 last 89.27) total: 64768 /unlink in 721.11 seconds: 89.82 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 916 (time 1761285798.19 total 10.00 last 91.60) - open/close 1797 (time 1761285808.20 total 20.01 last 88.02) - open/close 2679 (time 1761285818.20 total 30.01 last 88.20) - open/close 3611 (time 1761285828.20 total 40.01 last 93.18) - open/close 4562 (time 1761285838.21 total 50.02 last 95.04) - open/close 5522 (time 1761285848.21 total 60.02 last 95.95) - open/close 6431 (time 1761285858.22 total 70.03 last 90.86) - open/close 7315 (time 1761285868.23 total 80.04 last 88.27) - open/close 8150 (time 1761285878.23 total 90.04 last 83.48) - open/close 9264 (time 1761285888.24 total 100.05 last 111.31) - open/close 10000 (time 1761285895.50 total 107.31 last 101.42) - open/close 11009 (time 1761285905.51 total 117.32 last 100.82) - open/close 11909 (time 1761285915.51 total 127.32 last 90.00) - open/close 12810 (time 1761285925.51 total 137.32 last 90.09) - open/close 13659 (time 1761285935.51 total 147.32 last 84.90) - open/close 14516 (time 1761285945.51 total 157.33 last 85.65) - open/close 15379 (time 1761285955.52 total 167.33 last 86.25) - open/close 16226 (time 1761285965.52 total 177.34 last 84.65) - open/close 17126 (time 1761285975.53 total 187.35 last 89.92) - open/close 18114 (time 1761285985.54 total 197.35 last 98.71) - open/close 19019 (time 1761285995.54 total 207.35 last 90.49) - open/close 19970 (time 1761286005.55 total 217.36 last 95.07) - open/close 20845 (time 1761286015.55 total 227.37 last 87.42) - open/close 21625 (time 1761286025.56 total 237.38 last 77.93) - open/close 22574 (time 1761286035.57 total 247.38 last 94.89) - open/close 23476 (time 1761286045.58 total 257.39 last 90.05) - open/close 24319 (time 1761286055.60 total 267.41 last 84.13) - open/close 25154 (time 1761286065.62 total 277.43 last 83.33) - open/close 26041 (time 1761286075.63 total 287.44 last 88.66) - open/close 26968 (time 1761286085.63 total 297.44 last 92.66) - open/close 27874 (time 1761286095.63 total 307.44 last 90.59) - open/close 28811 (time 1761286105.64 total 317.45 last 93.61) - open/close 29730 (time 1761286115.65 total 327.46 last 91.83) - open/close 30000 (time 1761286118.61 total 330.42 last 91.25) - open/close 30852 (time 1761286128.61 total 340.42 last 85.18) - open/close 31765 (time 1761286138.62 total 350.43 last 91.20) - open/close 32655 (time 1761286148.63 total 360.44 last 88.92) - open/close 33581 (time 1761286158.64 total 370.45 last 92.53) - open/close 34502 (time 1761286168.64 total 380.45 last 92.08) - open/close 35383 (time 1761286178.65 total 390.46 last 88.02) - open/close 36286 (time 1761286188.65 total 400.46 last 90.28) - open/close 37168 (time 1761286198.66 total 410.47 last 88.11) - open/close 38113 (time 1761286208.66 total 420.48 last 94.47) - open/close 39013 (time 1761286218.67 total 430.49 last 89.92) - open/close 39959 (time 1761286228.68 total 440.50 last 94.51) - open/close 40897 (time 1761286238.69 total 450.50 last 93.72) - open/close 41817 (time 1761286248.70 total 460.51 last 91.96) - open/close 42686 (time 1761286258.70 total 470.51 last 86.89) - open/close 43529 (time 1761286268.71 total 480.52 last 84.21) - open/close 44391 (time 1761286278.72 total 490.53 last 86.11) - open/close 45236 (time 1761286288.73 total 500.54 last 84.44) - open/close 46081 (time 1761286298.74 total 510.55 last 84.42) - open/close 46886 (time 1761286308.74 total 520.56 last 80.42) - open/close 47733 (time 1761286318.75 total 530.56 last 84.68) - open/close 48554 (time 1761286328.75 total 540.56 last 82.10) - open/close 49353 (time 1761286338.75 total 550.56 last 79.88) - open/close 50000 (time 1761286346.26 total 558.07 last 86.13) - open/close 50782 (time 1761286356.27 total 568.09 last 78.10) - open/close 51646 (time 1761286366.28 total 578.09 last 86.38) - open/close 52476 (time 1761286376.28 total 588.10 last 82.92) - open/close 53311 (time 1761286386.28 total 598.10 last 83.50) - open/close 54109 (time 1761286396.29 total 608.10 last 79.76) - open/close 54940 (time 1761286406.29 total 618.10 last 83.08) - open/close 55673 (time 1761286416.30 total 628.12 last 73.21) - open/close 56449 (time 1761286426.31 total 638.12 last 77.53) - open/close 57284 (time 1761286436.32 total 648.13 last 83.44) - open/close 58111 (time 1761286446.32 total 658.13 last 82.69) - open/close 58966 (time 1761286456.32 total 668.14 last 85.47) - open/close 59760 (time 1761286466.33 total 678.14 last 79.34) - open/close 60000 (time 1761286469.24 total 681.05 last 82.53) - open/close 60844 (time 1761286479.24 total 691.06 last 84.35) - open/close 61713 (time 1761286489.26 total 701.07 last 86.75) - open/close 62494 (time 1761286499.27 total 711.09 last 78.01) - open/close 63279 (time 1761286509.28 total 721.09 last 78.46) - open/close 64048 (time 1761286519.29 total 731.10 last 76.80) total: 64768 open/close in 740.06 seconds: 87.52 ops/second - /unlink 778 (time 1761286541.47 total 10.00 last 77.79) - /unlink 1521 (time 1761286551.48 total 20.01 last 74.22) - /unlink 2329 (time 1761286561.49 total 30.02 last 80.76) - /unlink 3095 (time 1761286571.50 total 40.03 last 76.49) - /unlink 3954 (time 1761286581.51 total 50.04 last 85.82) - /unlink 4843 (time 1761286591.51 total 60.04 last 88.88) - /unlink 5639 (time 1761286601.52 total 70.05 last 79.56) - /unlink 6442 (time 1761286611.53 total 80.06 last 80.23) - /unlink 7301 (time 1761286621.53 total 90.06 last 85.88) - /unlink 8199 (time 1761286631.54 total 100.07 last 89.74) - /unlink 9084 (time 1761286641.54 total 110.07 last 88.47) - /unlink 9928 (time 1761286651.55 total 120.09 last 84.27) - /unlink 10816 (time 1761286661.56 total 130.09 last 88.73) - /unlink 11819 (time 1761286671.56 total 140.09 last 100.30) - /unlink 12861 (time 1761286681.57 total 150.10 last 104.16) - /unlink 13740 (time 1761286691.57 total 160.11 last 87.82) - /unlink 14537 (time 1761286701.58 total 170.11 last 79.70) - /unlink 15409 (time 1761286711.58 total 180.11 last 87.17) - /unlink 16252 (time 1761286721.58 total 190.12 last 84.25) - /unlink 17079 (time 1761286731.59 total 200.12 last 82.69) - /unlink 17957 (time 1761286741.59 total 210.12 last 87.79) - /unlink 18772 (time 1761286751.60 total 220.13 last 81.43) - /unlink 19687 (time 1761286761.60 total 230.13 last 91.49) - /unlink 20000 (time 1761286765.16 total 233.70 last 87.75) - /unlink 20886 (time 1761286775.17 total 243.70 last 88.57) - /unlink 21971 (time 1761286785.17 total 253.70 last 108.46) - /unlink 22912 (time 1761286795.18 total 263.71 last 94.03) - /unlink 23760 (time 1761286805.19 total 273.72 last 84.72) - /unlink 24599 (time 1761286815.19 total 283.72 last 83.85) - /unlink 25519 (time 1761286825.21 total 293.74 last 91.89) - /unlink 26376 (time 1761286835.22 total 303.75 last 85.62) - /unlink 27204 (time 1761286845.22 total 313.75 last 82.79) - /unlink 27964 (time 1761286855.23 total 323.76 last 75.94) - /unlink 28728 (time 1761286865.24 total 333.77 last 76.30) - /unlink 29496 (time 1761286875.24 total 343.77 last 76.77) - /unlink 30000 (time 1761286881.30 total 349.83 last 83.25) - /unlink 30827 (time 1761286891.30 total 359.83 last 82.66) - /unlink 31636 (time 1761286901.30 total 369.83 last 80.89) - /unlink 32484 (time 1761286911.31 total 379.85 last 84.71) - /unlink 33248 (time 1761286921.32 total 389.85 last 76.33) - /unlink 34072 (time 1761286931.33 total 399.86 last 82.31) - /unlink 34872 (time 1761286941.35 total 409.88 last 79.86) - /unlink 35641 (time 1761286951.36 total 419.89 last 76.81) - /unlink 36437 (time 1761286961.37 total 429.91 last 79.52) - /unlink 37232 (time 1761286971.38 total 439.91 last 79.45) - /unlink 38061 (time 1761286981.39 total 449.92 last 82.85) - /unlink 38846 (time 1761286991.39 total 459.92 last 78.46) - /unlink 39671 (time 1761287001.40 total 469.93 last 82.46) - /unlink 40000 (time 1761287005.08 total 473.61 last 89.31) - /unlink 40849 (time 1761287015.09 total 483.62 last 84.83) - /unlink 41692 (time 1761287025.09 total 493.62 last 84.28) - /unlink 42570 (time 1761287035.10 total 503.63 last 87.75) - /unlink 43410 (time 1761287045.11 total 513.64 last 83.88) - /unlink 44236 (time 1761287055.12 total 523.65 last 82.50) - /unlink 45062 (time 1761287065.13 total 533.66 last 82.53) - /unlink 45897 (time 1761287075.13 total 543.67 last 83.48) - /unlink 46755 (time 1761287085.14 total 553.67 last 85.76) - /unlink 47654 (time 1761287095.14 total 563.67 last 89.89) - /unlink 48566 (time 1761287105.15 total 573.68 last 91.15) - /unlink 49399 (time 1761287115.15 total 583.68 last 83.26) - /unlink 50000 (time 1761287122.28 total 590.82 last 84.25) - /unlink 50828 (time 1761287132.30 total 600.83 last 82.67) - /unlink 51612 (time 1761287142.32 total 610.85 last 78.25) - /unlink 52408 (time 1761287152.33 total 620.86 last 79.54) - /unlink 53235 (time 1761287162.33 total 630.86 last 82.67) - /unlink 54080 (time 1761287172.33 total 640.86 last 84.48) - /unlink 54886 (time 1761287182.33 total 650.86 last 80.59) - /unlink 55728 (time 1761287192.34 total 660.87 last 84.13) - /unlink 56658 (time 1761287202.34 total 670.87 last 92.99) - /unlink 57530 (time 1761287212.35 total 680.88 last 87.14) - /unlink 58374 (time 1761287222.35 total 690.89 last 84.35) - /unlink 59203 (time 1761287232.36 total 700.89 last 82.83) - /unlink 60000 (time 1761287241.85 total 710.39 last 83.97) - /unlink 60892 (time 1761287251.86 total 720.40 last 89.10) - /unlink 61783 (time 1761287261.87 total 730.40 last 89.09) - /unlink 62784 (time 1761287271.87 total 740.40 last 100.04) - /unlink 63612 (time 1761287281.88 total 750.41 last 82.74) - /unlink 64312 (time 1761287291.89 total 760.42 last 69.94) total: 64768 /unlink in 766.82 seconds: 84.46 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2