== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:12:44 (1774887164) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 753 (time 1774887318.61 total 10.00 last 75.26) - open/close 1483 (time 1774887328.62 total 20.01 last 72.95) - open/close 2239 (time 1774887338.63 total 30.02 last 75.51) - open/close 3039 (time 1774887348.64 total 40.03 last 79.94) - open/close 3750 (time 1774887358.67 total 50.06 last 70.90) - open/close 4433 (time 1774887368.68 total 60.07 last 68.26) - open/close 5333 (time 1774887378.68 total 70.07 last 89.95) - open/close 6002 (time 1774887388.68 total 80.07 last 66.87) - open/close 6645 (time 1774887398.68 total 90.07 last 64.30) - open/close 7168 (time 1774887408.70 total 100.09 last 52.24) - open/close 7765 (time 1774887418.70 total 110.09 last 59.66) - open/close 8331 (time 1774887428.71 total 120.10 last 56.56) - open/close 8941 (time 1774887438.72 total 130.11 last 60.95) - open/close 9543 (time 1774887448.72 total 140.11 last 60.17) - open/close 10000 (time 1774887455.34 total 146.73 last 69.02) - open/close 10649 (time 1774887465.37 total 156.76 last 64.76) - open/close 11236 (time 1774887475.38 total 166.77 last 58.64) - open/close 11845 (time 1774887485.38 total 176.77 last 60.88) - open/close 12549 (time 1774887495.39 total 186.78 last 70.34) - open/close 13257 (time 1774887505.39 total 196.78 last 70.76) - open/close 13962 (time 1774887515.39 total 206.78 last 70.49) - open/close 14736 (time 1774887525.42 total 216.81 last 77.21) - open/close 15465 (time 1774887535.44 total 226.83 last 72.77) - open/close 16119 (time 1774887545.44 total 236.83 last 65.39) - open/close 16753 (time 1774887555.44 total 246.83 last 63.39) - open/close 17481 (time 1774887565.46 total 256.85 last 72.63) - open/close 18214 (time 1774887575.46 total 266.85 last 73.29) - open/close 18959 (time 1774887585.47 total 276.86 last 74.44) - open/close 19552 (time 1774887595.48 total 286.87 last 59.26) - open/close 20000 (time 1774887602.05 total 293.44 last 68.17) - open/close 20678 (time 1774887612.06 total 303.45 last 67.75) - open/close 21364 (time 1774887622.06 total 313.45 last 68.58) - open/close 21929 (time 1774887632.07 total 323.46 last 56.47) - open/close 22601 (time 1774887642.08 total 333.47 last 67.11) - open/close 23394 (time 1774887652.09 total 343.48 last 79.20) - open/close 24029 (time 1774887662.10 total 353.49 last 63.43) - open/close 24717 (time 1774887672.12 total 363.51 last 68.65) - open/close 25431 (time 1774887682.13 total 373.52 last 71.39) - open/close 26074 (time 1774887692.13 total 383.52 last 64.26) - open/close 26625 (time 1774887702.14 total 393.53 last 55.07) - open/close 27220 (time 1774887712.14 total 403.53 last 59.48) - open/close 27797 (time 1774887722.14 total 413.53 last 57.69) - open/close 28447 (time 1774887732.14 total 423.53 last 64.99) - open/close 29108 (time 1774887742.15 total 433.54 last 66.03) - open/close 29737 (time 1774887752.18 total 443.57 last 62.76) - open/close 30000 (time 1774887756.02 total 447.41 last 68.48) - open/close 30547 (time 1774887766.03 total 457.42 last 54.61) - open/close 31139 (time 1774887776.04 total 467.43 last 59.16) - open/close 31691 (time 1774887786.05 total 477.44 last 55.16) - open/close 32328 (time 1774887796.05 total 487.44 last 63.70) - open/close 32859 (time 1774887806.06 total 497.45 last 53.06) - open/close 33261 (time 1774887816.06 total 507.45 last 40.18) - open/close 33924 (time 1774887826.07 total 517.47 last 66.22) - open/close 34605 (time 1774887836.09 total 527.48 last 68.02) - open/close 35244 (time 1774887846.09 total 537.48 last 63.89) - open/close 35922 (time 1774887856.10 total 547.49 last 67.73) - open/close 36689 (time 1774887866.11 total 557.50 last 76.63) - open/close 37332 (time 1774887876.11 total 567.50 last 64.27) - open/close 37989 (time 1774887886.12 total 577.51 last 65.67) - open/close 38642 (time 1774887896.13 total 587.52 last 65.25) - open/close 39226 (time 1774887906.15 total 597.54 last 58.26) - open/close 39778 (time 1774887916.16 total 607.55 last 55.15) - open/close 40000 (time 1774887920.15 total 611.54 last 55.58) - open/close 40567 (time 1774887930.16 total 621.55 last 56.64) - open/close 41077 (time 1774887940.18 total 631.57 last 50.92) - open/close 41656 (time 1774887950.18 total 641.57 last 57.88) - open/close 42270 (time 1774887960.19 total 651.58 last 61.38) - open/close 42781 (time 1774887970.19 total 661.58 last 51.09) - open/close 43445 (time 1774887980.19 total 671.58 last 66.37) - open/close 44153 (time 1774887990.21 total 681.60 last 70.71) - open/close 44711 (time 1774888000.21 total 691.61 last 55.76) - open/close 45282 (time 1774888010.22 total 701.61 last 57.06) - open/close 45842 (time 1774888020.25 total 711.64 last 55.87) - open/close 46439 (time 1774888030.25 total 721.65 last 59.64) - open/close 47063 (time 1774888040.26 total 731.65 last 62.39) - open/close 47630 (time 1774888050.27 total 741.66 last 56.64) - open/close 48241 (time 1774888060.27 total 751.66 last 61.08) - open/close 48859 (time 1774888070.28 total 761.67 last 61.74) - open/close 49500 (time 1774888080.29 total 771.68 last 64.02) - open/close 50000 (time 1774888088.15 total 779.54 last 63.63) - open/close 50573 (time 1774888098.15 total 789.54 last 57.29) - open/close 51175 (time 1774888108.17 total 799.56 last 60.07) - open/close 51679 (time 1774888118.18 total 809.57 last 50.37) - open/close 52249 (time 1774888128.20 total 819.59 last 56.89) - open/close 52764 (time 1774888138.21 total 829.60 last 51.45) - open/close 53459 (time 1774888148.22 total 839.61 last 69.45) - open/close 54090 (time 1774888158.22 total 849.61 last 63.07) - open/close 54696 (time 1774888168.24 total 859.63 last 60.47) - open/close 55309 (time 1774888178.25 total 869.64 last 61.23) - open/close 55820 (time 1774888188.26 total 879.65 last 51.05) - open/close 56417 (time 1774888198.29 total 889.68 last 59.53) - open/close 56946 (time 1774888208.30 total 899.69 last 52.87) - open/close 57693 (time 1774888218.34 total 909.73 last 74.40) - open/close 58202 (time 1774888228.35 total 919.74 last 50.85) - open/close 58821 (time 1774888238.36 total 929.75 last 61.83) - open/close 59380 (time 1774888248.38 total 939.77 last 55.80) - open/close 59927 (time 1774888258.38 total 949.77 last 54.69) - open/close 60552 (time 1774888268.41 total 959.80 last 62.31) - open/close 61155 (time 1774888278.41 total 969.80 last 60.29) - open/close 61665 (time 1774888288.45 total 979.84 last 50.81) - open/close 62297 (time 1774888298.47 total 989.86 last 63.04) - open/close 62913 (time 1774888308.50 total 999.89 last 61.42) - open/close 63466 (time 1774888318.52 total 1009.91 last 55.19) - open/close 64055 (time 1774888328.55 total 1019.94 last 58.74) - open/close 64603 (time 1774888338.58 total 1029.97 last 54.63) total: 64768 open/close in 1032.70 seconds: 62.72 ops/second - /unlink 552 (time 1774888357.25 total 10.02 last 55.09) - /unlink 1097 (time 1774888367.25 total 20.02 last 54.48) - /unlink 1597 (time 1774888377.28 total 30.06 last 49.82) - /unlink 2067 (time 1774888387.32 total 40.09 last 46.84) - /unlink 2644 (time 1774888397.34 total 50.11 last 57.58) - /unlink 3310 (time 1774888407.34 total 60.11 last 66.60) - /unlink 3842 (time 1774888417.36 total 70.13 last 53.11) - /unlink 4265 (time 1774888427.37 total 80.15 last 42.23) - /unlink 4700 (time 1774888437.38 total 90.15 last 43.48) - /unlink 5130 (time 1774888447.38 total 100.16 last 42.98) - /unlink 5628 (time 1774888457.41 total 110.18 last 49.68) - /unlink 6103 (time 1774888467.41 total 120.19 last 47.47) - /unlink 6685 (time 1774888477.43 total 130.20 last 58.11) - /unlink 7275 (time 1774888487.44 total 140.21 last 58.96) - /unlink 7923 (time 1774888497.44 total 150.22 last 64.77) - /unlink 8458 (time 1774888507.46 total 160.23 last 53.41) - /unlink 8986 (time 1774888517.47 total 170.24 last 52.77) - /unlink 9599 (time 1774888527.47 total 180.24 last 61.29) - /unlink 10000 (time 1774888534.11 total 186.89 last 60.33) - /unlink 10465 (time 1774888544.13 total 196.90 last 46.44) - /unlink 11062 (time 1774888554.13 total 206.91 last 59.65) - /unlink 11614 (time 1774888564.15 total 216.93 last 55.10) - /unlink 12106 (time 1774888574.20 total 226.98 last 48.95) - /unlink 12657 (time 1774888584.24 total 237.01 last 54.91) - /unlink 13210 (time 1774888594.26 total 247.03 last 55.19) - /unlink 13754 (time 1774888604.27 total 257.05 last 54.34) - /unlink 14439 (time 1774888614.33 total 267.11 last 68.06) - /unlink 15082 (time 1774888624.34 total 277.12 last 64.25) - /unlink 15646 (time 1774888634.37 total 287.14 last 56.25) - /unlink 16282 (time 1774888644.40 total 297.18 last 63.39) - /unlink 16936 (time 1774888654.41 total 307.18 last 65.37) - /unlink 17560 (time 1774888664.41 total 317.19 last 62.38) - /unlink 18209 (time 1774888674.42 total 327.20 last 64.83) - /unlink 18850 (time 1774888684.44 total 337.21 last 64.00) - /unlink 19540 (time 1774888694.46 total 347.23 last 68.84) - /unlink 20000 (time 1774888701.42 total 354.20 last 66.06) - /unlink 20596 (time 1774888711.43 total 364.21 last 59.55) - /unlink 21164 (time 1774888721.45 total 374.22 last 56.72) - /unlink 21721 (time 1774888731.45 total 384.23 last 55.67) - /unlink 22195 (time 1774888741.46 total 394.23 last 47.37) - /unlink 22708 (time 1774888751.47 total 404.25 last 51.23) - /unlink 23211 (time 1774888761.48 total 414.25 last 50.27) - /unlink 23819 (time 1774888771.48 total 424.26 last 60.75) - /unlink 24450 (time 1774888781.49 total 434.27 last 63.04) - /unlink 25171 (time 1774888791.50 total 444.27 last 72.08) - /unlink 25801 (time 1774888801.51 total 454.28 last 62.94) - /unlink 26435 (time 1774888811.51 total 464.29 last 63.36) - /unlink 27105 (time 1774888821.52 total 474.30 last 66.93) - /unlink 27709 (time 1774888831.53 total 484.31 last 60.33) - /unlink 28389 (time 1774888841.54 total 494.32 last 67.96) - /unlink 29027 (time 1774888851.55 total 504.33 last 63.72) - /unlink 29703 (time 1774888861.55 total 514.33 last 67.60) - /unlink 30000 (time 1774888865.75 total 518.53 last 70.74) - /unlink 30646 (time 1774888875.76 total 528.54 last 64.53) - /unlink 31313 (time 1774888885.79 total 538.57 last 66.50) - /unlink 32004 (time 1774888895.79 total 548.57 last 69.10) - /unlink 32678 (time 1774888905.80 total 558.57 last 67.37) - /unlink 33352 (time 1774888915.81 total 568.58 last 67.34) - /unlink 33847 (time 1774888925.81 total 578.59 last 49.48) - /unlink 34444 (time 1774888935.82 total 588.60 last 59.64) - /unlink 35210 (time 1774888945.83 total 598.60 last 76.55) - /unlink 35971 (time 1774888955.83 total 608.61 last 76.05) - /unlink 36736 (time 1774888965.84 total 618.61 last 76.46) - /unlink 37468 (time 1774888975.85 total 628.62 last 73.13) - /unlink 38142 (time 1774888985.86 total 638.63 last 67.35) - /unlink 38850 (time 1774888995.86 total 648.64 last 70.74) - /unlink 39522 (time 1774889005.88 total 658.65 last 67.12) - /unlink 40000 (time 1774889012.95 total 665.73 last 67.54) - /unlink 40638 (time 1774889022.97 total 675.74 last 63.71) - /unlink 41229 (time 1774889032.99 total 685.76 last 58.99) - /unlink 41848 (time 1774889043.00 total 695.77 last 61.83) - /unlink 42483 (time 1774889053.00 total 705.78 last 63.46) - /unlink 43189 (time 1774889063.00 total 715.78 last 70.60) - /unlink 43998 (time 1774889073.01 total 725.78 last 80.89) - /unlink 44731 (time 1774889083.01 total 735.78 last 73.30) - /unlink 45442 (time 1774889093.01 total 745.79 last 71.05) - /unlink 46101 (time 1774889103.02 total 755.80 last 65.84) - /unlink 46750 (time 1774889113.03 total 765.80 last 64.87) - /unlink 47464 (time 1774889123.03 total 775.81 last 71.37) - /unlink 48155 (time 1774889133.04 total 785.81 last 69.07) - /unlink 48786 (time 1774889143.04 total 795.81 last 63.09) - /unlink 49461 (time 1774889153.05 total 805.82 last 67.43) - /unlink 50000 (time 1774889159.55 total 812.33 last 82.88) - /unlink 50721 (time 1774889169.55 total 822.33 last 72.09) - /unlink 51420 (time 1774889179.56 total 832.34 last 69.84) - /unlink 52117 (time 1774889189.56 total 842.34 last 69.70) - /unlink 52813 (time 1774889199.56 total 852.34 last 69.58) - /unlink 53524 (time 1774889209.58 total 862.35 last 71.00) - /unlink 54302 (time 1774889219.58 total 872.35 last 77.79) - /unlink 54944 (time 1774889229.59 total 882.37 last 64.12) - /unlink 55657 (time 1774889239.60 total 892.38 last 71.21) - /unlink 56312 (time 1774889249.61 total 902.39 last 65.45) - /unlink 56977 (time 1774889259.62 total 912.39 last 66.45) - /unlink 57678 (time 1774889269.62 total 922.40 last 70.08) - /unlink 58414 (time 1774889279.63 total 932.40 last 73.56) - /unlink 59170 (time 1774889289.64 total 942.41 last 75.53) - /unlink 59903 (time 1774889299.64 total 952.41 last 73.28) - /unlink 60653 (time 1774889309.64 total 962.41 last 75.00) - /unlink 61316 (time 1774889319.64 total 972.42 last 66.29) - /unlink 62104 (time 1774889329.65 total 982.43 last 78.73) - /unlink 62900 (time 1774889339.65 total 992.43 last 79.59) - /unlink 63662 (time 1774889349.65 total 1002.43 last 76.18) - /unlink 64487 (time 1774889359.66 total 1012.44 last 82.44) total: 64768 /unlink in 1015.37 seconds: 63.79 ops/second Waiting for MDT destroys to complete oleg159-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 - open/close 759 (time 1774889386.69 total 10.00 last 75.88) - open/close 1596 (time 1774889396.70 total 20.01 last 83.64) - open/close 2369 (time 1774889406.70 total 30.01 last 77.27) - open/close 3293 (time 1774889416.70 total 40.02 last 92.37) - open/close 4145 (time 1774889426.70 total 50.02 last 85.19) - open/close 5167 (time 1774889436.71 total 60.02 last 102.19) - open/close 5897 (time 1774889446.71 total 70.02 last 72.99) - open/close 6755 (time 1774889456.72 total 80.03 last 85.69) - open/close 7536 (time 1774889466.72 total 90.04 last 78.07) - open/close 8517 (time 1774889476.72 total 100.04 last 98.10) - open/close 9357 (time 1774889486.73 total 110.04 last 83.94) - open/close 10000 (time 1774889494.73 total 118.04 last 80.39) - open/close 10688 (time 1774889504.74 total 128.05 last 68.71) - open/close 11172 (time 1774889514.75 total 138.07 last 48.34) - open/close 11835 (time 1774889524.76 total 148.07 last 66.27) - open/close 12494 (time 1774889534.76 total 158.08 last 65.86) - open/close 13140 (time 1774889544.78 total 168.09 last 64.52) - open/close 13896 (time 1774889554.78 total 178.10 last 75.54) - open/close 14673 (time 1774889564.80 total 188.11 last 77.62) - open/close 15488 (time 1774889574.80 total 198.11 last 81.45) - open/close 16366 (time 1774889584.81 total 208.12 last 87.73) - open/close 17279 (time 1774889594.82 total 218.13 last 91.19) - open/close 18119 (time 1774889604.84 total 228.15 last 83.84) - open/close 18996 (time 1774889614.85 total 238.16 last 87.60) - open/close 19846 (time 1774889624.86 total 248.17 last 84.95) - open/close 20727 (time 1774889634.88 total 258.19 last 87.93) - open/close 21675 (time 1774889644.88 total 268.19 last 94.77) - open/close 22551 (time 1774889654.89 total 278.20 last 87.56) - open/close 23439 (time 1774889664.90 total 288.21 last 88.67) - open/close 24349 (time 1774889674.91 total 298.22 last 90.93) - open/close 25225 (time 1774889684.92 total 308.23 last 87.54) - open/close 26173 (time 1774889694.93 total 318.24 last 94.66) - open/close 26947 (time 1774889704.94 total 328.25 last 77.34) - open/close 27817 (time 1774889714.95 total 338.26 last 86.94) - open/close 28627 (time 1774889724.97 total 348.28 last 80.83) - open/close 29591 (time 1774889734.98 total 358.29 last 96.31) - open/close 30000 (time 1774889739.97 total 363.28 last 81.94) - open/close 30840 (time 1774889749.97 total 373.28 last 83.97) - open/close 31800 (time 1774889759.97 total 383.29 last 95.98) - open/close 32682 (time 1774889769.98 total 393.29 last 88.16) - open/close 33469 (time 1774889779.98 total 403.30 last 78.64) - open/close 34231 (time 1774889789.99 total 413.31 last 76.14) - open/close 35196 (time 1774889800.00 total 423.31 last 96.46) - open/close 36135 (time 1774889810.00 total 433.32 last 93.85) - open/close 37037 (time 1774889820.01 total 443.33 last 90.09) - open/close 37819 (time 1774889830.03 total 453.34 last 78.10) - open/close 38712 (time 1774889840.04 total 463.36 last 89.16) - open/close 39633 (time 1774889850.05 total 473.36 last 92.08) - open/close 40000 (time 1774889853.98 total 477.30 last 93.22) - open/close 40978 (time 1774889863.98 total 487.30 last 97.80) - open/close 41963 (time 1774889873.99 total 497.30 last 98.47) - open/close 42952 (time 1774889884.00 total 507.32 last 98.74) - open/close 44060 (time 1774889894.01 total 517.32 last 110.76) - open/close 45177 (time 1774889904.01 total 527.33 last 111.60) - open/close 46392 (time 1774889914.02 total 537.33 last 121.46) - open/close 47604 (time 1774889924.02 total 547.34 last 121.15) - open/close 48705 (time 1774889934.02 total 557.34 last 110.08) - open/close 49794 (time 1774889944.03 total 567.35 last 108.79) - open/close 50919 (time 1774889954.04 total 577.35 last 112.46) - open/close 51954 (time 1774889964.04 total 587.35 last 103.47) - open/close 52991 (time 1774889974.04 total 597.35 last 103.70) - open/close 54113 (time 1774889984.04 total 607.35 last 112.18) - open/close 55340 (time 1774889994.05 total 617.36 last 122.63) - open/close 56475 (time 1774890004.06 total 627.37 last 113.37) - open/close 57697 (time 1774890014.06 total 637.37 last 122.19) - open/close 58923 (time 1774890024.07 total 647.38 last 122.53) - open/close 59970 (time 1774890034.07 total 657.38 last 104.64) - open/close 61121 (time 1774890044.07 total 667.39 last 115.09) - open/close 62243 (time 1774890054.07 total 677.39 last 112.19) - open/close 63335 (time 1774890064.08 total 687.39 last 109.12) - open/close 64270 (time 1774890074.08 total 697.40 last 93.47) total: 64768 open/close in 702.09 seconds: 92.25 ops/second - /unlink 1117 (time 1774890091.67 total 10.00 last 111.69) - /unlink 2252 (time 1774890101.68 total 20.00 last 113.46) - /unlink 3204 (time 1774890111.68 total 30.01 last 95.14) - /unlink 4316 (time 1774890121.68 total 40.01 last 111.18) - /unlink 5387 (time 1774890131.69 total 50.02 last 107.02) - /unlink 6555 (time 1774890141.69 total 60.02 last 116.77) - /unlink 7586 (time 1774890151.71 total 70.04 last 102.96) - /unlink 8805 (time 1774890161.71 total 80.04 last 121.85) - /unlink 9766 (time 1774890171.73 total 90.06 last 95.97) - /unlink 10000 (time 1774890173.85 total 92.18 last 110.33) - /unlink 10975 (time 1774890183.85 total 102.18 last 97.48) - /unlink 11943 (time 1774890193.85 total 112.18 last 96.75) - /unlink 13001 (time 1774890203.86 total 122.19 last 105.78) - /unlink 14207 (time 1774890213.86 total 132.19 last 120.58) - /unlink 15341 (time 1774890223.87 total 142.19 last 113.31) - /unlink 16341 (time 1774890233.87 total 152.20 last 99.98) - /unlink 17418 (time 1774890243.87 total 162.20 last 107.69) - /unlink 18537 (time 1774890253.88 total 172.21 last 111.81) - /unlink 19579 (time 1774890263.88 total 182.21 last 104.14) - /unlink 20000 (time 1774890267.66 total 185.99 last 111.38) - /unlink 21081 (time 1774890277.67 total 195.99 last 108.07) - /unlink 22155 (time 1774890287.67 total 206.00 last 107.39) - /unlink 23196 (time 1774890297.67 total 216.00 last 104.03) - /unlink 24282 (time 1774890307.68 total 226.01 last 108.57) - /unlink 25426 (time 1774890317.68 total 236.01 last 114.36) - /unlink 26557 (time 1774890327.69 total 246.02 last 113.03) - /unlink 27721 (time 1774890337.69 total 256.02 last 116.38) - /unlink 28778 (time 1774890347.69 total 266.02 last 105.68) - /unlink 29703 (time 1774890357.70 total 276.03 last 92.43) - /unlink 30000 (time 1774890361.13 total 279.46 last 86.56) - /unlink 31029 (time 1774890371.14 total 289.47 last 102.82) - /unlink 32081 (time 1774890381.14 total 299.47 last 105.14) - /unlink 33274 (time 1774890391.14 total 309.47 last 119.26) - /unlink 34349 (time 1774890401.15 total 319.48 last 107.44) - /unlink 35402 (time 1774890411.16 total 329.49 last 105.20) - /unlink 36444 (time 1774890421.17 total 339.50 last 104.13) - /unlink 37504 (time 1774890431.17 total 349.50 last 105.96) - /unlink 38681 (time 1774890441.17 total 359.50 last 117.70) - /unlink 40000 (time 1774890448.86 total 367.19 last 171.48) - /unlink 42014 (time 1774890458.86 total 377.19 last 201.38) - /unlink 43973 (time 1774890468.86 total 387.19 last 195.88) - /unlink 46025 (time 1774890478.87 total 397.20 last 205.14) - /unlink 48080 (time 1774890488.87 total 407.20 last 205.47) - /unlink 50000 (time 1774890498.84 total 417.17 last 192.51) - /unlink 52073 (time 1774890508.85 total 427.18 last 207.20) - /unlink 54124 (time 1774890518.85 total 437.18 last 205.05) - /unlink 56048 (time 1774890528.85 total 447.18 last 192.33) - /unlink 58008 (time 1774890538.86 total 457.19 last 195.88) - /unlink 59853 (time 1774890548.86 total 467.19 last 184.50) - /unlink 61868 (time 1774890558.87 total 477.20 last 201.36) - /unlink 63977 (time 1774890568.87 total 487.20 last 210.89) total: 64768 /unlink in 491.51 seconds: 131.77 ops/second Waiting for MDT destroys to complete oleg159-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 - open/close 1043 (time 1774890597.55 total 10.00 last 104.26) - open/close 2121 (time 1774890607.55 total 20.01 last 107.76) - open/close 3299 (time 1774890617.56 total 30.01 last 117.74) - open/close 4491 (time 1774890627.56 total 40.02 last 119.15) - open/close 5358 (time 1774890637.57 total 50.02 last 86.64) - open/close 6442 (time 1774890647.57 total 60.03 last 108.34) - open/close 7488 (time 1774890657.58 total 70.03 last 104.57) - open/close 8615 (time 1774890667.58 total 80.04 last 112.65) - open/close 9859 (time 1774890677.59 total 90.04 last 124.36) - open/close 10955 (time 1774890687.59 total 100.04 last 109.55) - open/close 12090 (time 1774890697.59 total 110.05 last 113.49) - open/close 13267 (time 1774890707.59 total 120.05 last 117.68) - open/close 14545 (time 1774890717.60 total 130.05 last 127.71) - open/close 15741 (time 1774890727.61 total 140.06 last 119.50) - open/close 16991 (time 1774890737.61 total 150.07 last 124.95) - open/close 18155 (time 1774890747.62 total 160.07 last 116.33) - open/close 19240 (time 1774890757.62 total 170.07 last 108.50) - open/close 20000 (time 1774890764.75 total 177.21 last 106.55) - open/close 21074 (time 1774890774.76 total 187.21 last 107.35) - open/close 22223 (time 1774890784.76 total 197.21 last 114.88) - open/close 23450 (time 1774890794.77 total 207.22 last 122.62) - open/close 24530 (time 1774890804.77 total 217.23 last 107.92) - open/close 25592 (time 1774890814.78 total 227.23 last 106.16) - open/close 26702 (time 1774890824.79 total 237.24 last 110.87) - open/close 27762 (time 1774890834.79 total 247.24 last 105.99) - open/close 28858 (time 1774890844.79 total 257.24 last 109.58) - open/close 30000 (time 1774890854.16 total 266.61 last 121.94) - open/close 31141 (time 1774890864.16 total 276.61 last 114.09) - open/close 32234 (time 1774890874.16 total 286.61 last 109.30) - open/close 33423 (time 1774890884.16 total 296.61 last 118.89) - open/close 34571 (time 1774890894.16 total 306.62 last 114.73) - open/close 35852 (time 1774890904.17 total 316.62 last 128.01) - open/close 37149 (time 1774890914.17 total 326.63 last 129.66) - open/close 38447 (time 1774890924.18 total 336.63 last 129.74) - open/close 39770 (time 1774890934.18 total 346.64 last 132.23) - open/close 41027 (time 1774890944.19 total 356.64 last 125.66) - open/close 42470 (time 1774890954.19 total 366.64 last 144.27) - open/close 43772 (time 1774890964.19 total 376.65 last 130.15) - open/close 45019 (time 1774890974.20 total 386.65 last 124.62) - open/close 46316 (time 1774890984.20 total 396.66 last 129.66) - open/close 47721 (time 1774890994.21 total 406.66 last 140.38) - open/close 49073 (time 1774891004.21 total 416.67 last 135.18) - open/close 50000 (time 1774891009.60 total 422.05 last 172.08) - open/close 51506 (time 1774891019.60 total 432.06 last 150.52) - open/close 52951 (time 1774891029.61 total 442.06 last 144.44) - open/close 54424 (time 1774891039.61 total 452.07 last 147.22) - open/close 55963 (time 1774891049.61 total 462.07 last 153.89) - open/close 57242 (time 1774891059.61 total 472.07 last 127.89) - open/close 58457 (time 1774891069.62 total 482.07 last 121.47) - open/close 59947 (time 1774891079.62 total 492.07 last 148.98) - open/close 61335 (time 1774891089.63 total 502.08 last 138.68) - open/close 62550 (time 1774891099.63 total 512.08 last 121.48) - open/close 63782 (time 1774891109.64 total 522.09 last 123.11) total: 64768 open/close in 529.11 seconds: 122.41 ops/second - /unlink 2659 (time 1774891128.72 total 10.00 last 265.87) - /unlink 5148 (time 1774891138.72 total 20.00 last 248.88) - /unlink 7632 (time 1774891148.72 total 30.00 last 248.33) - /unlink 10000 (time 1774891158.15 total 39.43 last 251.31) - /unlink 12116 (time 1774891168.15 total 49.43 last 211.56) - /unlink 14618 (time 1774891178.15 total 59.43 last 250.15) - /unlink 17238 (time 1774891188.15 total 69.43 last 261.95) - /unlink 19618 (time 1774891198.15 total 79.43 last 237.99) - /unlink 21830 (time 1774891208.15 total 89.43 last 221.19) - /unlink 24040 (time 1774891218.16 total 99.44 last 220.91) - /unlink 26561 (time 1774891228.16 total 109.44 last 252.09) - /unlink 28751 (time 1774891238.16 total 119.44 last 218.97) - /unlink 30000 (time 1774891243.53 total 124.81 last 232.47) - /unlink 32349 (time 1774891253.54 total 134.82 last 234.81) - /unlink 34468 (time 1774891263.54 total 144.82 last 211.88) - /unlink 37087 (time 1774891273.54 total 154.82 last 261.83) - /unlink 39713 (time 1774891283.54 total 164.82 last 262.51) - /unlink 41927 (time 1774891293.55 total 174.83 last 221.30) - /unlink 43940 (time 1774891303.55 total 184.83 last 201.25) - /unlink 46098 (time 1774891313.55 total 194.83 last 215.78) - /unlink 48088 (time 1774891323.55 total 204.83 last 198.94) - /unlink 50000 (time 1774891332.17 total 213.45 last 221.87) - /unlink 52188 (time 1774891342.17 total 223.45 last 218.80) - /unlink 54607 (time 1774891352.17 total 233.45 last 241.88) - /unlink 59541 (time 1774891362.17 total 243.45 last 493.35) - /unlink 62881 (time 1774891372.18 total 253.46 last 333.85) total: 64768 /unlink in 260.45 seconds: 248.68 ops/second Waiting for MDT destroys to complete oleg159-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 - open/close 1783 (time 1774891395.98 total 10.00 last 178.27) - open/close 3227 (time 1774891405.98 total 20.01 last 144.32) - open/close 4638 (time 1774891415.99 total 30.01 last 141.01) - open/close 6287 (time 1774891426.00 total 40.02 last 164.77) - open/close 7687 (time 1774891436.00 total 50.02 last 139.97) - open/close 9389 (time 1774891446.00 total 60.03 last 170.12) - open/close 10000 (time 1774891449.86 total 63.89 last 158.37) - open/close 11691 (time 1774891459.87 total 73.89 last 169.00) - open/close 13884 (time 1774891469.87 total 83.89 last 219.28) - open/close 15837 (time 1774891479.87 total 93.90 last 195.21) - open/close 17675 (time 1774891489.87 total 103.90 last 183.79) - open/close 19332 (time 1774891499.87 total 113.90 last 165.67) - open/close 20000 (time 1774891503.31 total 117.33 last 194.65) - open/close 21972 (time 1774891513.31 total 127.33 last 197.16) - open/close 23911 (time 1774891523.31 total 137.34 last 193.87) - open/close 25528 (time 1774891533.31 total 147.34 last 161.67) - open/close 27417 (time 1774891543.31 total 157.34 last 188.89) - open/close 29645 (time 1774891553.32 total 167.34 last 222.73) - open/close 32324 (time 1774891563.32 total 177.34 last 267.85) - open/close 34654 (time 1774891573.32 total 187.35 last 232.91) - open/close 37132 (time 1774891583.33 total 197.35 last 247.63) - open/close 39174 (time 1774891593.33 total 207.36 last 204.17) - open/close 40000 (time 1774891597.87 total 211.89 last 182.06) - open/close 41728 (time 1774891607.88 total 221.90 last 172.61) - open/close 43547 (time 1774891617.88 total 231.90 last 181.90) - open/close 45533 (time 1774891627.88 total 241.91 last 198.56) - open/close 47946 (time 1774891637.88 total 251.91 last 241.27) - open/close 50000 (time 1774891647.35 total 261.38 last 216.84) - open/close 52552 (time 1774891657.36 total 271.38 last 255.17) - open/close 55581 (time 1774891667.36 total 281.38 last 302.83) - open/close 58650 (time 1774891677.36 total 291.38 last 306.90) - open/close 60000 (time 1774891681.76 total 295.78 last 306.75) - open/close 63003 (time 1774891691.76 total 305.79 last 300.26) total: 64768 open/close in 311.52 seconds: 207.91 ops/second - /unlink 4972 (time 1774891709.20 total 10.00 last 497.10) - /unlink 10000 (time 1774891719.19 total 19.99 last 503.43) - /unlink 15210 (time 1774891729.19 total 29.99 last 520.91) - /unlink 20000 (time 1774891737.88 total 38.68 last 551.51) - /unlink 25526 (time 1774891747.88 total 48.68 last 552.54) - /unlink 30000 (time 1774891754.30 total 55.09 last 697.34) - /unlink 36903 (time 1774891764.30 total 65.09 last 690.26) - /unlink 40000 (time 1774891769.51 total 70.31 last 593.96) - /unlink 46451 (time 1774891779.51 total 80.31 last 645.10) - /unlink 50000 (time 1774891784.99 total 85.79 last 647.56) - /unlink 57848 (time 1774891794.99 total 95.79 last 784.79) - /unlink 60000 (time 1774891797.58 total 98.37 last 832.54) total: 64768 /unlink in 104.47 seconds: 619.99 ops/second Waiting for MDT destroys to complete oleg159-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 - open/close 3807 (time 1774891822.19 total 10.00 last 380.64) - open/close 7699 (time 1774891832.19 total 20.00 last 389.13) - open/close 10000 (time 1774891838.08 total 25.89 last 390.77) - open/close 13562 (time 1774891848.08 total 35.89 last 356.19) - open/close 17336 (time 1774891858.08 total 45.90 last 377.12) - open/close 20000 (time 1774891865.60 total 53.41 last 354.50) - open/close 23771 (time 1774891875.60 total 63.42 last 377.06) - open/close 27773 (time 1774891885.60 total 73.42 last 400.09) - open/close 30000 (time 1774891890.98 total 78.79 last 414.28) - open/close 34617 (time 1774891900.98 total 88.79 last 461.64) - open/close 39043 (time 1774891910.98 total 98.80 last 442.57) - open/close 40000 (time 1774891913.12 total 100.94 last 446.36) - open/close 44453 (time 1774891923.13 total 110.94 last 445.26) - open/close 48931 (time 1774891933.13 total 120.94 last 447.76) - open/close 50000 (time 1774891935.44 total 123.25 last 462.58) - open/close 53874 (time 1774891945.44 total 133.26 last 387.30) - open/close 58008 (time 1774891955.44 total 143.26 last 413.30) - open/close 60000 (time 1774891959.68 total 147.49 last 470.46) - open/close 64092 (time 1774891969.68 total 157.49 last 409.12) total: 64768 open/close in 159.27 seconds: 406.65 ops/second - /unlink 7277 (time 1774891982.86 total 10.00 last 727.68) - /unlink 10000 (time 1774891986.99 total 14.12 last 660.65) - /unlink 18027 (time 1774891996.99 total 24.12 last 802.68) - /unlink 20000 (time 1774891999.53 total 26.67 last 774.73) - /unlink 27512 (time 1774892009.53 total 36.67 last 751.17) - /unlink 30000 (time 1774892013.27 total 40.41 last 665.05) - /unlink 37546 (time 1774892023.28 total 50.41 last 754.48) - /unlink 40000 (time 1774892026.70 total 53.83 last 717.47) - /unlink 48497 (time 1774892036.70 total 63.83 last 849.68) - /unlink 57553 (time 1774892046.70 total 73.83 last 905.57) - /unlink 60000 (time 1774892049.51 total 76.65 last 869.18) total: 64768 /unlink in 82.51 seconds: 784.96 ops/second Waiting for MDT destroys to complete oleg159-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2