== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:14:00 (1763349240) Stopping clients: oleg444-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg444-client.virtnet /mnt/lustre2 (opts:-f) oleg444-server: oleg444-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg444-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg444-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 871 (time 1763349348.53 total 10.00 last 87.10) - open/close 1694 (time 1763349358.54 total 20.00 last 82.29) - open/close 2535 (time 1763349368.55 total 30.02 last 83.98) - open/close 3355 (time 1763349378.56 total 40.02 last 81.94) - open/close 4216 (time 1763349388.57 total 50.03 last 86.01) - open/close 5073 (time 1763349398.57 total 60.03 last 85.68) - open/close 5893 (time 1763349408.57 total 70.04 last 81.96) - open/close 6676 (time 1763349418.58 total 80.04 last 78.26) - open/close 7535 (time 1763349428.58 total 90.05 last 85.84) - open/close 8359 (time 1763349438.60 total 100.06 last 82.29) - open/close 9115 (time 1763349448.60 total 110.06 last 75.60) - open/close 9831 (time 1763349458.61 total 120.07 last 71.54) - open/close 10000 (time 1763349460.95 total 122.42 last 72.00) - open/close 10841 (time 1763349470.96 total 132.43 last 84.04) - open/close 11621 (time 1763349480.97 total 142.43 last 77.97) - open/close 12375 (time 1763349490.97 total 152.44 last 75.34) - open/close 13113 (time 1763349500.98 total 162.44 last 73.78) - open/close 13818 (time 1763349510.98 total 172.45 last 70.49) - open/close 14629 (time 1763349520.99 total 182.45 last 81.03) - open/close 15493 (time 1763349531.00 total 192.47 last 86.27) - open/close 16330 (time 1763349541.02 total 202.49 last 83.56) - open/close 17147 (time 1763349551.03 total 212.49 last 81.63) - open/close 18059 (time 1763349561.03 total 222.50 last 91.17) - open/close 18896 (time 1763349571.04 total 232.50 last 83.64) - open/close 19755 (time 1763349581.05 total 242.52 last 85.79) - open/close 20000 (time 1763349584.12 total 245.59 last 79.84) - open/close 20890 (time 1763349594.12 total 255.59 last 88.98) - open/close 21708 (time 1763349604.12 total 265.59 last 81.77) - open/close 22515 (time 1763349614.13 total 275.60 last 80.65) - open/close 23359 (time 1763349624.14 total 285.61 last 84.29) - open/close 24195 (time 1763349634.14 total 295.61 last 83.60) - open/close 24917 (time 1763349644.15 total 305.61 last 72.19) - open/close 25692 (time 1763349654.15 total 315.61 last 77.49) - open/close 26443 (time 1763349664.16 total 325.62 last 75.02) - open/close 27197 (time 1763349674.17 total 335.64 last 75.31) - open/close 27943 (time 1763349684.18 total 345.65 last 74.52) - open/close 28751 (time 1763349694.20 total 355.66 last 80.68) - open/close 29609 (time 1763349704.20 total 365.67 last 85.76) - open/close 30000 (time 1763349709.01 total 370.47 last 81.37) - open/close 30780 (time 1763349719.02 total 380.48 last 77.91) - open/close 31520 (time 1763349729.02 total 390.49 last 73.96) - open/close 32267 (time 1763349739.03 total 400.49 last 74.66) - open/close 33058 (time 1763349749.03 total 410.50 last 79.07) - open/close 33769 (time 1763349759.04 total 420.50 last 71.05) - open/close 34546 (time 1763349769.05 total 430.51 last 77.63) - open/close 35215 (time 1763349779.05 total 440.51 last 66.90) - open/close 35895 (time 1763349789.05 total 450.52 last 67.97) - open/close 36622 (time 1763349799.07 total 460.53 last 72.60) - open/close 37410 (time 1763349809.07 total 470.53 last 78.80) - open/close 38156 (time 1763349819.07 total 480.54 last 74.58) - open/close 38937 (time 1763349829.07 total 490.54 last 78.06) - open/close 39686 (time 1763349839.08 total 500.55 last 74.86) - open/close 40000 (time 1763349843.02 total 504.48 last 79.77) - open/close 40835 (time 1763349853.02 total 514.49 last 83.45) - open/close 41610 (time 1763349863.03 total 524.50 last 77.44) - open/close 42356 (time 1763349873.03 total 534.50 last 74.58) - open/close 43107 (time 1763349883.04 total 544.51 last 75.06) - open/close 43849 (time 1763349893.06 total 554.52 last 74.08) - open/close 44565 (time 1763349903.06 total 564.52 last 71.59) - open/close 45286 (time 1763349913.07 total 574.53 last 72.02) - open/close 45982 (time 1763349923.08 total 584.55 last 69.51) - open/close 46674 (time 1763349933.09 total 594.56 last 69.15) - open/close 47428 (time 1763349943.10 total 604.57 last 75.31) - open/close 48175 (time 1763349953.11 total 614.58 last 74.63) - open/close 48867 (time 1763349963.12 total 624.58 last 69.17) - open/close 49613 (time 1763349973.12 total 634.58 last 74.59) - open/close 50000 (time 1763349978.56 total 640.03 last 71.06) - open/close 50762 (time 1763349988.57 total 650.03 last 76.17) - open/close 51539 (time 1763349998.57 total 660.04 last 77.68) - open/close 52328 (time 1763350008.58 total 670.05 last 78.80) - open/close 53067 (time 1763350018.59 total 680.05 last 73.86) - open/close 53794 (time 1763350028.60 total 690.06 last 72.63) - open/close 54529 (time 1763350038.60 total 700.07 last 73.45) - open/close 55358 (time 1763350048.61 total 710.08 last 82.84) - open/close 56161 (time 1763350058.61 total 720.08 last 80.29) - open/close 56936 (time 1763350068.62 total 730.09 last 77.42) - open/close 57674 (time 1763350078.63 total 740.09 last 73.75) - open/close 58450 (time 1763350088.64 total 750.11 last 77.50) - open/close 59185 (time 1763350098.65 total 760.12 last 73.43) - open/close 59925 (time 1763350108.65 total 770.12 last 73.98) - open/close 60650 (time 1763350118.66 total 780.13 last 72.44) - open/close 61326 (time 1763350128.66 total 790.13 last 67.59) - open/close 62047 (time 1763350138.67 total 800.13 last 72.07) - open/close 62840 (time 1763350148.68 total 810.14 last 79.22) - open/close 63562 (time 1763350158.69 total 820.15 last 72.13) - open/close 64245 (time 1763350168.69 total 830.16 last 68.28) total: 64768 open/close in 837.51 seconds: 77.33 ops/second - /unlink 740 (time 1763350190.01 total 10.01 last 73.94) - /unlink 1453 (time 1763350200.02 total 20.02 last 71.23) - /unlink 2182 (time 1763350210.02 total 30.02 last 72.89) - /unlink 2891 (time 1763350220.03 total 40.03 last 70.84) - /unlink 3541 (time 1763350230.04 total 50.03 last 64.95) - /unlink 4283 (time 1763350240.05 total 60.04 last 74.15) - /unlink 5001 (time 1763350250.05 total 70.05 last 71.76) - /unlink 5716 (time 1763350260.07 total 80.06 last 71.38) - /unlink 6380 (time 1763350270.10 total 90.09 last 66.20) - /unlink 7104 (time 1763350280.10 total 100.09 last 72.39) - /unlink 7854 (time 1763350290.11 total 110.10 last 74.95) - /unlink 8514 (time 1763350300.12 total 120.11 last 65.93) - /unlink 9177 (time 1763350310.13 total 130.12 last 66.22) - /unlink 9899 (time 1763350320.13 total 140.12 last 72.20) - /unlink 10649 (time 1763350330.13 total 150.13 last 74.97) - /unlink 11409 (time 1763350340.14 total 160.13 last 75.95) - /unlink 12168 (time 1763350350.14 total 170.14 last 75.87) - /unlink 12904 (time 1763350360.15 total 180.14 last 73.56) - /unlink 13589 (time 1763350370.17 total 190.16 last 68.36) - /unlink 14296 (time 1763350380.17 total 200.17 last 70.68) - /unlink 15009 (time 1763350390.18 total 210.18 last 71.21) - /unlink 15715 (time 1763350400.19 total 220.18 last 70.56) - /unlink 16410 (time 1763350410.20 total 230.20 last 69.42) - /unlink 17121 (time 1763350420.22 total 240.21 last 70.99) - /unlink 17833 (time 1763350430.23 total 250.22 last 71.14) - /unlink 18562 (time 1763350440.23 total 260.23 last 72.85) - /unlink 19244 (time 1763350450.24 total 270.23 last 68.17) - /unlink 19950 (time 1763350460.24 total 280.23 last 70.60) - /unlink 20638 (time 1763350470.24 total 290.24 last 68.76) - /unlink 21358 (time 1763350480.25 total 300.25 last 71.93) - /unlink 22110 (time 1763350490.25 total 310.25 last 75.18) - /unlink 22825 (time 1763350500.26 total 320.25 last 71.47) - /unlink 23541 (time 1763350510.27 total 330.26 last 71.55) - /unlink 24186 (time 1763350520.27 total 340.26 last 64.48) - /unlink 24923 (time 1763350530.28 total 350.28 last 73.60) - /unlink 25654 (time 1763350540.28 total 360.28 last 73.09) - /unlink 26350 (time 1763350550.29 total 370.28 last 69.56) - /unlink 27078 (time 1763350560.29 total 380.29 last 72.78) - /unlink 27854 (time 1763350570.29 total 390.29 last 77.59) - /unlink 28579 (time 1763350580.29 total 400.29 last 72.50) - /unlink 29304 (time 1763350590.30 total 410.29 last 72.46) - /unlink 29998 (time 1763350600.30 total 420.30 last 69.35) - /unlink 30712 (time 1763350610.31 total 430.30 last 71.39) - /unlink 31403 (time 1763350620.31 total 440.30 last 69.08) - /unlink 32122 (time 1763350630.31 total 450.30 last 71.89) - /unlink 32926 (time 1763350640.32 total 460.32 last 80.30) - /unlink 33650 (time 1763350650.33 total 470.33 last 72.31) - /unlink 34405 (time 1763350660.36 total 480.35 last 75.33) - /unlink 35127 (time 1763350670.38 total 490.37 last 72.06) - /unlink 35857 (time 1763350680.38 total 500.37 last 72.99) - /unlink 36605 (time 1763350690.38 total 510.38 last 74.79) - /unlink 37385 (time 1763350700.38 total 520.38 last 77.97) - /unlink 38114 (time 1763350710.39 total 530.38 last 72.89) - /unlink 38837 (time 1763350720.39 total 540.39 last 72.25) - /unlink 39553 (time 1763350730.40 total 550.39 last 71.56) - /unlink 40000 (time 1763350736.30 total 556.29 last 75.79) - /unlink 40712 (time 1763350746.30 total 566.29 last 71.19) - /unlink 41454 (time 1763350756.31 total 576.30 last 74.12) - /unlink 42236 (time 1763350766.31 total 586.31 last 78.18) - /unlink 43002 (time 1763350776.32 total 596.32 last 76.50) - /unlink 43729 (time 1763350786.33 total 606.33 last 72.63) - /unlink 44388 (time 1763350796.34 total 616.33 last 65.87) - /unlink 45070 (time 1763350806.35 total 626.35 last 68.12) - /unlink 45715 (time 1763350816.36 total 636.36 last 64.43) - /unlink 46346 (time 1763350826.37 total 646.36 last 63.07) - /unlink 47027 (time 1763350836.38 total 656.37 last 68.04) - /unlink 47757 (time 1763350846.38 total 666.37 last 72.99) - /unlink 48479 (time 1763350856.38 total 676.38 last 72.14) - /unlink 49217 (time 1763350866.38 total 686.38 last 73.80) - /unlink 49958 (time 1763350876.39 total 696.39 last 74.04) - /unlink 50744 (time 1763350886.40 total 706.39 last 78.58) - /unlink 51468 (time 1763350896.41 total 716.40 last 72.32) - /unlink 52089 (time 1763350906.41 total 726.40 last 62.09) - /unlink 52763 (time 1763350916.41 total 736.41 last 67.38) - /unlink 53483 (time 1763350926.41 total 746.41 last 72.00) - /unlink 54259 (time 1763350936.42 total 756.42 last 77.51) - /unlink 55006 (time 1763350946.44 total 766.43 last 74.60) - /unlink 55736 (time 1763350956.44 total 776.43 last 72.99) - /unlink 56473 (time 1763350966.44 total 786.44 last 73.69) - /unlink 57192 (time 1763350976.44 total 796.44 last 71.88) - /unlink 57905 (time 1763350986.45 total 806.45 last 71.24) - /unlink 58680 (time 1763350996.45 total 816.45 last 77.48) - /unlink 59392 (time 1763351006.46 total 826.45 last 71.17) - /unlink 60000 (time 1763351014.85 total 834.85 last 72.42) - /unlink 60756 (time 1763351024.86 total 844.85 last 75.57) - /unlink 61599 (time 1763351034.87 total 854.86 last 84.20) - /unlink 62391 (time 1763351044.88 total 864.88 last 79.08) - /unlink 63150 (time 1763351054.89 total 874.89 last 75.83) - /unlink 63851 (time 1763351064.91 total 884.90 last 70.02) - /unlink 64557 (time 1763351074.92 total 894.91 last 70.51) total: 64768 /unlink in 898.02 seconds: 72.12 ops/second Waiting for MDT destroys to complete oleg444-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 - open/close 640 (time 1763351102.26 total 10.00 last 63.98) - open/close 1355 (time 1763351112.28 total 20.02 last 71.41) - open/close 2106 (time 1763351122.29 total 30.03 last 74.99) - open/close 2927 (time 1763351132.30 total 40.04 last 82.03) - open/close 3666 (time 1763351142.31 total 50.04 last 73.86) - open/close 4463 (time 1763351152.32 total 60.05 last 79.61) - open/close 5190 (time 1763351162.32 total 70.06 last 72.66) - open/close 6065 (time 1763351172.33 total 80.07 last 87.39) - open/close 6846 (time 1763351182.35 total 90.08 last 78.02) - open/close 7608 (time 1763351192.36 total 100.09 last 76.12) - open/close 8405 (time 1763351202.36 total 110.10 last 79.68) - open/close 9155 (time 1763351212.37 total 120.11 last 74.92) - open/close 9943 (time 1763351222.38 total 130.12 last 78.70) - open/close 10697 (time 1763351232.40 total 140.14 last 75.28) - open/close 11464 (time 1763351242.41 total 150.14 last 76.63) - open/close 12235 (time 1763351252.41 total 160.14 last 77.10) - open/close 13001 (time 1763351262.41 total 170.15 last 76.59) - open/close 13752 (time 1763351272.41 total 180.15 last 75.06) - open/close 14487 (time 1763351282.43 total 190.16 last 73.40) - open/close 15229 (time 1763351292.43 total 200.17 last 74.17) - open/close 15878 (time 1763351302.45 total 210.18 last 64.80) - open/close 16541 (time 1763351312.46 total 220.20 last 66.19) - open/close 17140 (time 1763351322.48 total 230.22 last 59.81) - open/close 17817 (time 1763351332.49 total 240.22 last 67.64) - open/close 18456 (time 1763351342.49 total 250.22 last 63.90) - open/close 19175 (time 1763351352.49 total 260.23 last 71.85) - open/close 19928 (time 1763351362.51 total 270.25 last 75.17) - open/close 20699 (time 1763351372.51 total 280.25 last 77.08) - open/close 21426 (time 1763351382.51 total 290.25 last 72.69) - open/close 22252 (time 1763351392.52 total 300.26 last 82.51) - open/close 23049 (time 1763351402.54 total 310.27 last 79.59) - open/close 23834 (time 1763351412.54 total 320.28 last 78.44) - open/close 24589 (time 1763351422.55 total 330.28 last 75.49) - open/close 25443 (time 1763351432.55 total 340.29 last 85.35) - open/close 26263 (time 1763351442.56 total 350.30 last 81.92) - open/close 27083 (time 1763351452.57 total 360.31 last 81.93) - open/close 27971 (time 1763351462.58 total 370.32 last 88.74) - open/close 28819 (time 1763351472.60 total 380.33 last 84.64) - open/close 29475 (time 1763351482.60 total 390.34 last 65.58) - open/close 30000 (time 1763351489.60 total 397.34 last 74.99) - open/close 30818 (time 1763351499.60 total 407.34 last 81.80) - open/close 31616 (time 1763351509.61 total 417.34 last 79.75) - open/close 32412 (time 1763351519.61 total 427.35 last 79.53) - open/close 33174 (time 1763351529.62 total 437.36 last 76.18) - open/close 33946 (time 1763351539.62 total 447.36 last 77.15) - open/close 34743 (time 1763351549.63 total 457.36 last 79.69) - open/close 35585 (time 1763351559.63 total 467.37 last 84.16) - open/close 36383 (time 1763351569.63 total 477.37 last 79.79) - open/close 37182 (time 1763351579.63 total 487.37 last 79.88) - open/close 37984 (time 1763351589.64 total 497.38 last 80.17) - open/close 38788 (time 1763351599.64 total 507.38 last 80.38) - open/close 39550 (time 1763351609.64 total 517.38 last 76.17) - open/close 40000 (time 1763351615.30 total 523.04 last 79.56) - open/close 40860 (time 1763351625.30 total 533.04 last 85.98) - open/close 41632 (time 1763351635.32 total 543.05 last 77.10) - open/close 42297 (time 1763351645.32 total 553.06 last 66.47) - open/close 42979 (time 1763351655.33 total 563.07 last 68.15) - open/close 43655 (time 1763351665.33 total 573.07 last 67.56) - open/close 44415 (time 1763351675.34 total 583.07 last 75.98) - open/close 45159 (time 1763351685.34 total 593.08 last 74.35) - open/close 45887 (time 1763351695.35 total 603.09 last 72.72) - open/close 46622 (time 1763351705.36 total 613.10 last 73.44) - open/close 47386 (time 1763351715.37 total 623.11 last 76.35) - open/close 48126 (time 1763351725.37 total 633.11 last 73.98) - open/close 48865 (time 1763351735.38 total 643.11 last 73.86) - open/close 49614 (time 1763351745.38 total 653.12 last 74.85) - open/close 50000 (time 1763351750.36 total 658.10 last 77.59) - open/close 50773 (time 1763351760.37 total 668.11 last 77.19) - open/close 51500 (time 1763351770.38 total 678.12 last 72.64) - open/close 52285 (time 1763351780.39 total 688.12 last 78.46) - open/close 53019 (time 1763351790.39 total 698.13 last 73.37) - open/close 53762 (time 1763351800.40 total 708.13 last 74.26) - open/close 54512 (time 1763351810.41 total 718.15 last 74.87) - open/close 55291 (time 1763351820.42 total 728.15 last 77.87) - open/close 55979 (time 1763351830.42 total 738.16 last 68.78) - open/close 56753 (time 1763351840.43 total 748.17 last 77.32) - open/close 57547 (time 1763351850.43 total 758.17 last 79.36) - open/close 58305 (time 1763351860.46 total 768.20 last 75.63) - open/close 59090 (time 1763351870.46 total 778.20 last 78.44) - open/close 59847 (time 1763351880.48 total 788.21 last 75.61) - open/close 60000 (time 1763351882.52 total 790.26 last 74.96) - open/close 60791 (time 1763351892.52 total 800.26 last 79.07) - open/close 61630 (time 1763351902.53 total 810.27 last 83.83) - open/close 62442 (time 1763351912.54 total 820.28 last 81.13) - open/close 63320 (time 1763351922.55 total 830.29 last 87.72) - open/close 64124 (time 1763351932.55 total 840.29 last 80.38) total: 64768 open/close in 848.15 seconds: 76.36 ops/second - /unlink 750 (time 1763351953.43 total 10.01 last 74.93) - /unlink 1491 (time 1763351963.43 total 20.01 last 74.10) - /unlink 2265 (time 1763351973.44 total 30.02 last 77.36) - /unlink 3032 (time 1763351983.44 total 40.02 last 76.66) - /unlink 3795 (time 1763351993.44 total 50.02 last 76.29) - /unlink 4547 (time 1763352003.46 total 60.04 last 75.09) - /unlink 5346 (time 1763352013.47 total 70.05 last 79.79) - /unlink 6189 (time 1763352023.48 total 80.06 last 84.24) - /unlink 7001 (time 1763352033.49 total 90.07 last 81.11) - /unlink 7909 (time 1763352043.50 total 100.08 last 90.72) - /unlink 8799 (time 1763352053.50 total 110.08 last 88.98) - /unlink 9612 (time 1763352063.51 total 120.08 last 81.26) - /unlink 10000 (time 1763352068.52 total 125.10 last 77.40) - /unlink 10855 (time 1763352078.53 total 135.11 last 85.42) - /unlink 11705 (time 1763352088.53 total 145.11 last 84.95) - /unlink 12535 (time 1763352098.54 total 155.12 last 82.97) - /unlink 13308 (time 1763352108.55 total 165.13 last 77.22) - /unlink 14086 (time 1763352118.57 total 175.14 last 77.66) - /unlink 14932 (time 1763352128.57 total 185.14 last 84.60) - /unlink 15761 (time 1763352138.58 total 195.15 last 82.81) - /unlink 16570 (time 1763352148.59 total 205.16 last 80.83) - /unlink 17406 (time 1763352158.60 total 215.17 last 83.51) - /unlink 18198 (time 1763352168.60 total 225.18 last 79.16) - /unlink 19017 (time 1763352178.61 total 235.19 last 81.85) - /unlink 19847 (time 1763352188.61 total 245.19 last 82.98) - /unlink 20636 (time 1763352198.62 total 255.20 last 78.81) - /unlink 21463 (time 1763352208.63 total 265.21 last 82.59) - /unlink 22406 (time 1763352218.64 total 275.21 last 94.29) - /unlink 23320 (time 1763352228.64 total 285.22 last 91.35) - /unlink 24119 (time 1763352238.64 total 295.22 last 79.89) - /unlink 24931 (time 1763352248.65 total 305.22 last 81.16) - /unlink 25861 (time 1763352258.66 total 315.24 last 92.87) - /unlink 26697 (time 1763352268.67 total 325.25 last 83.51) - /unlink 27556 (time 1763352278.67 total 335.25 last 85.89) - /unlink 28602 (time 1763352288.67 total 345.25 last 104.60) - /unlink 29620 (time 1763352298.68 total 355.25 last 101.77) - /unlink 30000 (time 1763352302.01 total 358.59 last 114.05) - /unlink 30973 (time 1763352312.01 total 368.59 last 97.29) - /unlink 31881 (time 1763352322.02 total 378.60 last 90.73) - /unlink 33156 (time 1763352332.02 total 388.60 last 127.47) - /unlink 34396 (time 1763352342.02 total 398.60 last 123.96) - /unlink 35749 (time 1763352352.03 total 408.61 last 135.21) - /unlink 37099 (time 1763352362.03 total 418.61 last 134.96) - /unlink 38268 (time 1763352372.03 total 428.61 last 116.88) - /unlink 39270 (time 1763352382.04 total 438.62 last 100.12) - /unlink 40000 (time 1763352389.46 total 446.04 last 98.43) - /unlink 41218 (time 1763352399.46 total 456.04 last 121.74) - /unlink 42378 (time 1763352409.47 total 466.04 last 115.98) - /unlink 43405 (time 1763352419.47 total 476.05 last 102.61) - /unlink 44366 (time 1763352429.48 total 486.06 last 96.05) - /unlink 45449 (time 1763352439.49 total 496.06 last 108.23) - /unlink 46440 (time 1763352449.49 total 506.07 last 99.05) - /unlink 47460 (time 1763352459.50 total 516.08 last 101.94) - /unlink 48660 (time 1763352469.50 total 526.08 last 119.99) - /unlink 49799 (time 1763352479.51 total 536.08 last 113.82) - /unlink 50910 (time 1763352489.51 total 546.09 last 111.03) - /unlink 51951 (time 1763352499.52 total 556.09 last 104.06) - /unlink 53155 (time 1763352509.53 total 566.11 last 120.26) - /unlink 54266 (time 1763352519.54 total 576.11 last 111.01) - /unlink 55137 (time 1763352529.54 total 586.12 last 87.03) - /unlink 56256 (time 1763352539.55 total 596.13 last 111.86) - /unlink 57314 (time 1763352549.55 total 606.13 last 105.76) - /unlink 58377 (time 1763352559.56 total 616.14 last 106.24) - /unlink 59331 (time 1763352569.56 total 626.14 last 95.35) - /unlink 60000 (time 1763352575.55 total 632.13 last 111.73) - /unlink 61129 (time 1763352585.55 total 642.13 last 112.85) - /unlink 62190 (time 1763352595.56 total 652.14 last 106.07) - /unlink 63309 (time 1763352605.56 total 662.14 last 111.86) - /unlink 64387 (time 1763352615.57 total 672.14 last 107.74) total: 64768 /unlink in 675.59 seconds: 95.87 ops/second Waiting for MDT destroys to complete oleg444-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 - open/close 1013 (time 1763352641.25 total 10.00 last 101.26) - open/close 2036 (time 1763352651.26 total 20.01 last 102.21) - open/close 3046 (time 1763352661.27 total 30.02 last 100.91) - open/close 4117 (time 1763352671.28 total 40.03 last 107.03) - open/close 5138 (time 1763352681.28 total 50.04 last 102.02) - open/close 6321 (time 1763352691.30 total 60.05 last 118.10) - open/close 7413 (time 1763352701.30 total 70.05 last 109.18) - open/close 8384 (time 1763352711.31 total 80.06 last 97.05) - open/close 9584 (time 1763352721.32 total 90.07 last 119.88) - open/close 10000 (time 1763352724.35 total 93.10 last 137.08) - open/close 11203 (time 1763352734.35 total 103.11 last 120.28) - open/close 12592 (time 1763352744.36 total 113.11 last 138.89) - open/close 13810 (time 1763352754.36 total 123.11 last 121.73) - open/close 14949 (time 1763352764.37 total 133.12 last 113.80) - open/close 15983 (time 1763352774.37 total 143.12 last 103.39) - open/close 16894 (time 1763352784.38 total 153.13 last 91.01) - open/close 17790 (time 1763352794.39 total 163.14 last 89.55) - open/close 18851 (time 1763352804.39 total 173.14 last 106.09) - open/close 20000 (time 1763352813.32 total 182.07 last 128.57) - open/close 21111 (time 1763352823.32 total 192.07 last 111.10) - open/close 22373 (time 1763352833.33 total 202.08 last 126.14) - open/close 23682 (time 1763352843.33 total 212.08 last 130.85) - open/close 24995 (time 1763352853.34 total 222.09 last 131.18) - open/close 26202 (time 1763352863.35 total 232.10 last 120.65) - open/close 27504 (time 1763352873.35 total 242.10 last 130.18) - open/close 28554 (time 1763352883.35 total 252.10 last 104.93) - open/close 29687 (time 1763352893.35 total 262.10 last 113.30) - open/close 30000 (time 1763352895.91 total 264.66 last 122.43) - open/close 31218 (time 1763352905.91 total 274.66 last 121.76) - open/close 32545 (time 1763352915.91 total 284.67 last 132.70) - open/close 33765 (time 1763352925.92 total 294.67 last 121.90) - open/close 34974 (time 1763352935.92 total 304.67 last 120.89) - open/close 36583 (time 1763352945.93 total 314.68 last 160.82) - open/close 37971 (time 1763352955.93 total 324.68 last 138.78) - open/close 39070 (time 1763352965.93 total 334.69 last 109.84) - open/close 40000 (time 1763352974.37 total 343.12 last 110.24) - open/close 41214 (time 1763352984.37 total 353.12 last 121.39) - open/close 42347 (time 1763352994.37 total 363.12 last 113.29) - open/close 43488 (time 1763353004.38 total 373.13 last 114.04) - open/close 44873 (time 1763353014.38 total 383.13 last 138.42) - open/close 46409 (time 1763353024.39 total 393.14 last 153.56) - open/close 47741 (time 1763353034.40 total 403.15 last 133.08) - open/close 49190 (time 1763353044.40 total 413.15 last 144.87) - open/close 50000 (time 1763353050.28 total 419.03 last 137.78) - open/close 51505 (time 1763353060.28 total 429.03 last 150.41) - open/close 53094 (time 1763353070.28 total 439.03 last 158.90) - open/close 54806 (time 1763353080.29 total 449.04 last 171.14) - open/close 56406 (time 1763353090.29 total 459.04 last 159.95) - open/close 57676 (time 1763353100.29 total 469.04 last 126.95) - open/close 58916 (time 1763353110.29 total 479.05 last 123.98) - open/close 60000 (time 1763353118.56 total 487.32 last 131.06) - open/close 61147 (time 1763353128.56 total 497.32 last 114.70) - open/close 62552 (time 1763353138.57 total 507.32 last 140.45) - open/close 63894 (time 1763353148.57 total 517.32 last 134.16) total: 64768 open/close in 524.19 seconds: 123.56 ops/second - /unlink 2525 (time 1763353167.44 total 10.00 last 252.38) - /unlink 5444 (time 1763353177.44 total 20.01 last 291.86) - /unlink 9396 (time 1763353187.44 total 30.01 last 395.12) - /unlink 12408 (time 1763353197.44 total 40.01 last 301.14) - /unlink 15271 (time 1763353207.44 total 50.01 last 286.27) - /unlink 17787 (time 1763353217.44 total 60.01 last 251.58) - /unlink 20000 (time 1763353225.51 total 68.08 last 274.37) - /unlink 23188 (time 1763353235.51 total 78.08 last 318.68) - /unlink 25978 (time 1763353245.51 total 88.08 last 279.00) - /unlink 28977 (time 1763353255.52 total 98.08 last 299.80) - /unlink 30000 (time 1763353258.47 total 101.04 last 345.99) - /unlink 32721 (time 1763353268.48 total 111.04 last 272.00) - /unlink 35238 (time 1763353278.48 total 121.05 last 251.65) - /unlink 37959 (time 1763353288.48 total 131.05 last 272.09) - /unlink 40000 (time 1763353296.43 total 139.00 last 256.54) - /unlink 43451 (time 1763353306.44 total 149.00 last 345.07) - /unlink 46642 (time 1763353316.44 total 159.01 last 319.05) - /unlink 50000 (time 1763353325.33 total 167.90 last 377.46) - /unlink 53317 (time 1763353335.34 total 177.91 last 331.59) - /unlink 56214 (time 1763353345.34 total 187.91 last 289.60) - /unlink 58809 (time 1763353355.34 total 197.91 last 259.47) - /unlink 60000 (time 1763353359.94 total 202.51 last 258.75) - /unlink 62694 (time 1763353369.95 total 212.52 last 269.31) total: 64768 /unlink in 220.44 seconds: 293.81 ops/second Waiting for MDT destroys to complete oleg444-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 - open/close 1472 (time 1763353397.65 total 10.00 last 147.14) - open/close 3198 (time 1763353407.66 total 20.01 last 172.58) - open/close 4850 (time 1763353417.66 total 30.01 last 165.17) - open/close 6474 (time 1763353427.66 total 40.01 last 162.36) - open/close 8150 (time 1763353437.66 total 50.01 last 167.59) - open/close 10000 (time 1763353447.35 total 59.70 last 190.84) - open/close 11830 (time 1763353457.36 total 69.71 last 182.96) - open/close 13579 (time 1763353467.36 total 79.71 last 174.81) - open/close 15240 (time 1763353477.37 total 89.72 last 166.03) - open/close 16755 (time 1763353487.37 total 99.72 last 151.50) - open/close 18611 (time 1763353497.38 total 109.73 last 185.28) - open/close 20000 (time 1763353504.81 total 117.16 last 187.16) - open/close 21625 (time 1763353514.81 total 127.16 last 162.47) - open/close 23371 (time 1763353524.81 total 137.16 last 174.53) - open/close 25060 (time 1763353534.82 total 147.17 last 168.81) - open/close 27346 (time 1763353544.82 total 157.17 last 228.54) - open/close 29495 (time 1763353554.82 total 167.17 last 214.88) - open/close 30000 (time 1763353556.91 total 169.26 last 241.48) - open/close 31888 (time 1763353566.92 total 179.27 last 188.70) - open/close 33900 (time 1763353576.92 total 189.27 last 201.17) - open/close 36258 (time 1763353586.92 total 199.27 last 235.69) - open/close 38236 (time 1763353596.93 total 209.28 last 197.74) - open/close 40000 (time 1763353603.46 total 215.81 last 270.03) - open/close 43143 (time 1763353613.46 total 225.81 last 314.23) - open/close 45973 (time 1763353623.46 total 235.81 last 282.99) - open/close 48523 (time 1763353633.46 total 245.81 last 254.98) - open/close 50000 (time 1763353640.37 total 252.72 last 213.87) - open/close 51955 (time 1763353650.37 total 262.72 last 195.48) - open/close 53906 (time 1763353660.37 total 272.72 last 195.04) - open/close 56061 (time 1763353670.37 total 282.72 last 215.49) - open/close 58088 (time 1763353680.37 total 292.72 last 202.70) - open/close 60000 (time 1763353687.89 total 300.24 last 254.22) - open/close 62386 (time 1763353697.90 total 310.25 last 238.51) - open/close 64616 (time 1763353707.90 total 320.25 last 222.94) total: 64768 open/close in 321.04 seconds: 201.75 ops/second - /unlink 4903 (time 1763353720.55 total 10.00 last 490.26) - /unlink 9808 (time 1763353730.55 total 20.00 last 490.39) - /unlink 15656 (time 1763353740.55 total 30.00 last 584.72) - /unlink 20000 (time 1763353748.80 total 38.25 last 526.89) - /unlink 26477 (time 1763353758.80 total 48.25 last 647.64) - /unlink 30000 (time 1763353764.78 total 54.23 last 589.01) - /unlink 36054 (time 1763353774.79 total 64.24 last 604.82) - /unlink 40000 (time 1763353780.55 total 70.00 last 685.40) - /unlink 45300 (time 1763353790.55 total 80.00 last 529.91) - /unlink 50000 (time 1763353798.61 total 88.06 last 583.29) - /unlink 57200 (time 1763353808.61 total 98.06 last 719.85) - /unlink 60000 (time 1763353812.24 total 101.69 last 770.92) total: 64768 /unlink in 109.28 seconds: 592.66 ops/second Waiting for MDT destroys to complete oleg444-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 - open/close 3100 (time 1763353840.74 total 10.00 last 309.97) - open/close 5987 (time 1763353850.74 total 20.00 last 288.61) - open/close 10000 (time 1763353860.63 total 29.89 last 405.84) - open/close 13875 (time 1763353870.63 total 39.89 last 387.47) - open/close 17355 (time 1763353880.63 total 49.90 last 347.92) - open/close 20000 (time 1763353887.79 total 57.05 last 369.78) - open/close 22359 (time 1763353897.79 total 67.05 last 235.90) - open/close 25705 (time 1763353907.79 total 77.05 last 334.53) - open/close 30000 (time 1763353917.77 total 87.03 last 430.37) - open/close 33557 (time 1763353927.77 total 97.03 last 355.65) - open/close 37472 (time 1763353937.77 total 107.03 last 391.42) - open/close 40000 (time 1763353944.58 total 113.84 last 371.27) - open/close 43593 (time 1763353954.58 total 123.84 last 359.24) - open/close 47652 (time 1763353964.58 total 133.85 last 405.82) - open/close 50000 (time 1763353969.53 total 138.79 last 474.67) - open/close 54252 (time 1763353979.53 total 148.79 last 425.16) - open/close 57648 (time 1763353989.53 total 158.79 last 339.59) - open/close 60000 (time 1763353994.78 total 164.05 last 447.87) - open/close 63568 (time 1763354004.78 total 174.05 last 356.76) total: 64768 open/close in 177.27 seconds: 365.37 ops/second - /unlink 6359 (time 1763354019.65 total 10.00 last 635.89) - /unlink 10000 (time 1763354024.09 total 14.44 last 819.35) - /unlink 18840 (time 1763354034.09 total 24.44 last 883.97) - /unlink 26699 (time 1763354044.09 total 34.45 last 785.83) - /unlink 30000 (time 1763354048.26 total 38.61 last 793.13) - /unlink 38317 (time 1763354058.26 total 48.61 last 831.62) - /unlink 46644 (time 1763354068.26 total 58.61 last 832.67) - /unlink 50000 (time 1763354071.98 total 62.33 last 901.97) - /unlink 58251 (time 1763354081.98 total 72.33 last 825.07) - /unlink 60000 (time 1763354084.51 total 74.87 last 689.79) total: 64768 /unlink in 82.09 seconds: 789.03 ops/second Waiting for MDT destroys to complete oleg444-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg444-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2