== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:06:22 (1767319582) Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 oleg333-server: oleg333-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg333-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg333-server: detected 4 online CPUs by sysfs oleg333-server: MODOPTS_LIBCFS= oleg333-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format ost1: /dev/vde Format mds2: /dev/vdd Format ost2: /dev/vdf Format mds1: /dev/vdc start mds service on oleg333-server pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 start mds service on oleg333-server pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 721 (time 1767319840.34 total 10.00 last 72.07) - open/close 1504 (time 1767319850.34 total 20.01 last 78.24) - open/close 2212 (time 1767319860.35 total 30.02 last 70.73) - open/close 2980 (time 1767319870.37 total 40.04 last 76.68) - open/close 3849 (time 1767319880.38 total 50.04 last 86.85) - open/close 4645 (time 1767319890.38 total 60.05 last 79.55) - open/close 5356 (time 1767319900.39 total 70.06 last 71.01) - open/close 6158 (time 1767319910.40 total 80.07 last 80.12) - open/close 6843 (time 1767319920.42 total 90.09 last 68.39) - open/close 7603 (time 1767319930.42 total 100.09 last 75.98) - open/close 8272 (time 1767319940.42 total 110.09 last 66.88) - open/close 8868 (time 1767319950.43 total 120.10 last 59.56) - open/close 9430 (time 1767319960.44 total 130.11 last 56.16) - open/close 10000 (time 1767319968.38 total 138.04 last 71.82) - open/close 10694 (time 1767319978.38 total 148.04 last 69.39) - open/close 11428 (time 1767319988.38 total 158.04 last 73.40) - open/close 12213 (time 1767319998.39 total 168.05 last 78.43) - open/close 12877 (time 1767320008.39 total 178.06 last 66.39) - open/close 13546 (time 1767320018.39 total 188.06 last 66.90) - open/close 14427 (time 1767320028.40 total 198.07 last 88.02) - open/close 15286 (time 1767320038.41 total 208.08 last 85.80) - open/close 16013 (time 1767320048.42 total 218.08 last 72.67) - open/close 16793 (time 1767320058.42 total 228.09 last 77.95) - open/close 17712 (time 1767320068.42 total 238.09 last 91.88) - open/close 18470 (time 1767320078.43 total 248.10 last 75.75) - open/close 19292 (time 1767320088.43 total 258.10 last 82.19) - open/close 20000 (time 1767320097.03 total 266.70 last 82.31) - open/close 20719 (time 1767320107.04 total 276.71 last 71.85) - open/close 21430 (time 1767320117.05 total 286.72 last 71.03) - open/close 22068 (time 1767320127.06 total 296.73 last 63.74) - open/close 22755 (time 1767320137.06 total 306.73 last 68.68) - open/close 23582 (time 1767320147.07 total 316.74 last 82.64) - open/close 24402 (time 1767320157.07 total 326.74 last 81.99) - open/close 25199 (time 1767320167.08 total 336.75 last 79.63) - open/close 26041 (time 1767320177.09 total 346.75 last 84.16) - open/close 26994 (time 1767320187.09 total 356.76 last 95.27) - open/close 27816 (time 1767320197.10 total 366.76 last 82.13) - open/close 28524 (time 1767320207.10 total 376.77 last 70.78) - open/close 29277 (time 1767320217.11 total 386.78 last 75.23) - open/close 30000 (time 1767320225.58 total 395.24 last 85.40) - open/close 30714 (time 1767320235.60 total 405.26 last 71.26) - open/close 31381 (time 1767320245.60 total 415.27 last 66.64) - open/close 32064 (time 1767320255.62 total 425.28 last 68.23) - open/close 32984 (time 1767320265.63 total 435.29 last 91.91) - open/close 33816 (time 1767320275.63 total 445.29 last 83.18) - open/close 34529 (time 1767320285.64 total 455.30 last 71.24) - open/close 35258 (time 1767320295.64 total 465.31 last 72.85) - open/close 36047 (time 1767320305.65 total 475.32 last 78.83) - open/close 36780 (time 1767320315.66 total 485.33 last 73.24) - open/close 37400 (time 1767320325.67 total 495.34 last 61.92) - open/close 38132 (time 1767320335.68 total 505.34 last 73.18) - open/close 38868 (time 1767320345.69 total 515.35 last 73.53) - open/close 39557 (time 1767320355.69 total 525.35 last 68.90) - open/close 40000 (time 1767320360.48 total 530.14 last 92.47) - open/close 40706 (time 1767320370.48 total 540.15 last 70.59) - open/close 41500 (time 1767320380.48 total 550.15 last 79.36) - open/close 42276 (time 1767320390.49 total 560.16 last 77.55) - open/close 42995 (time 1767320400.49 total 570.16 last 71.87) - open/close 43663 (time 1767320410.50 total 580.17 last 66.77) - open/close 44347 (time 1767320420.51 total 590.17 last 68.34) - open/close 44880 (time 1767320430.52 total 600.19 last 53.22) - open/close 45498 (time 1767320440.54 total 610.21 last 61.70) - open/close 46291 (time 1767320450.54 total 620.21 last 79.27) - open/close 47017 (time 1767320460.56 total 630.22 last 72.51) - open/close 47739 (time 1767320470.57 total 640.24 last 72.10) - open/close 48473 (time 1767320480.58 total 650.25 last 73.32) - open/close 49196 (time 1767320490.59 total 660.25 last 72.25) - open/close 49904 (time 1767320500.60 total 670.27 last 70.71) - open/close 50614 (time 1767320510.60 total 680.27 last 70.99) - open/close 51325 (time 1767320520.61 total 690.28 last 71.03) - open/close 52071 (time 1767320530.62 total 700.28 last 74.56) - open/close 52854 (time 1767320540.62 total 710.29 last 78.26) - open/close 53524 (time 1767320550.63 total 720.30 last 66.95) - open/close 54246 (time 1767320560.64 total 730.31 last 72.13) - open/close 55011 (time 1767320570.65 total 740.31 last 76.46) - open/close 55597 (time 1767320580.66 total 750.32 last 58.54) - open/close 56361 (time 1767320590.66 total 760.32 last 76.39) - open/close 57074 (time 1767320600.67 total 770.34 last 71.21) - open/close 57787 (time 1767320610.68 total 780.34 last 71.26) - open/close 58450 (time 1767320620.68 total 790.35 last 66.28) - open/close 59215 (time 1767320630.68 total 800.35 last 76.48) - open/close 59977 (time 1767320640.69 total 810.35 last 76.17) - open/close 60789 (time 1767320650.69 total 820.35 last 81.20) - open/close 61583 (time 1767320660.69 total 830.35 last 79.39) - open/close 62560 (time 1767320670.70 total 840.36 last 97.63) - open/close 63364 (time 1767320680.70 total 850.37 last 80.32) - open/close 64283 (time 1767320690.71 total 860.37 last 91.87) total: 64768 open/close in 865.82 seconds: 74.81 ops/second - /unlink 777 (time 1767320710.75 total 10.00 last 77.66) - /unlink 1560 (time 1767320720.75 total 20.01 last 78.30) - /unlink 2278 (time 1767320730.77 total 30.02 last 71.68) - /unlink 3007 (time 1767320740.79 total 40.04 last 72.78) - /unlink 3757 (time 1767320750.79 total 50.04 last 74.96) - /unlink 4488 (time 1767320760.80 total 60.05 last 73.08) - /unlink 5190 (time 1767320770.82 total 70.07 last 70.06) - /unlink 5973 (time 1767320780.82 total 80.07 last 78.29) - /unlink 6681 (time 1767320790.82 total 90.07 last 70.77) - /unlink 7368 (time 1767320800.83 total 100.08 last 68.66) - /unlink 8186 (time 1767320810.85 total 110.10 last 81.61) - /unlink 8903 (time 1767320820.86 total 120.11 last 71.63) - /unlink 9640 (time 1767320830.86 total 130.11 last 73.68) - /unlink 10000 (time 1767320836.67 total 135.92 last 61.99) - /unlink 10680 (time 1767320846.68 total 145.93 last 67.96) - /unlink 11470 (time 1767320856.68 total 155.93 last 78.95) - /unlink 12219 (time 1767320866.69 total 165.94 last 74.83) - /unlink 13044 (time 1767320876.72 total 175.97 last 82.28) - /unlink 13861 (time 1767320886.73 total 185.98 last 81.63) - /unlink 14523 (time 1767320896.73 total 195.98 last 66.18) - /unlink 15204 (time 1767320906.74 total 205.99 last 68.06) - /unlink 16053 (time 1767320916.74 total 215.99 last 84.85) - /unlink 16856 (time 1767320926.75 total 226.00 last 80.26) - /unlink 17666 (time 1767320936.75 total 236.00 last 80.94) - /unlink 18422 (time 1767320946.76 total 246.01 last 75.52) - /unlink 19278 (time 1767320956.77 total 256.02 last 85.52) - /unlink 20000 (time 1767320966.61 total 265.86 last 73.43) - /unlink 20747 (time 1767320976.62 total 275.87 last 74.56) - /unlink 21380 (time 1767320986.63 total 285.88 last 63.28) - /unlink 22160 (time 1767320996.63 total 295.88 last 77.95) - /unlink 22981 (time 1767321006.65 total 305.90 last 81.97) - /unlink 23777 (time 1767321016.65 total 315.90 last 79.57) - /unlink 24443 (time 1767321026.68 total 325.93 last 66.40) - /unlink 25207 (time 1767321036.69 total 335.94 last 76.38) - /unlink 25910 (time 1767321046.71 total 345.96 last 70.16) - /unlink 26664 (time 1767321056.71 total 355.96 last 75.34) - /unlink 27389 (time 1767321066.72 total 365.97 last 72.49) - /unlink 28052 (time 1767321076.74 total 375.99 last 66.16) - /unlink 28784 (time 1767321086.74 total 385.99 last 73.18) - /unlink 29503 (time 1767321096.75 total 396.00 last 71.80) - /unlink 30000 (time 1767321103.75 total 403.00 last 71.03) - /unlink 30654 (time 1767321113.76 total 413.01 last 65.34) - /unlink 31351 (time 1767321123.77 total 423.02 last 69.63) - /unlink 32111 (time 1767321133.77 total 433.02 last 75.98) - /unlink 32889 (time 1767321143.77 total 443.02 last 77.80) - /unlink 33647 (time 1767321153.78 total 453.03 last 75.80) - /unlink 34436 (time 1767321163.78 total 463.03 last 78.85) - /unlink 35350 (time 1767321173.79 total 473.04 last 91.35) - /unlink 36181 (time 1767321183.80 total 483.05 last 82.99) - /unlink 36851 (time 1767321193.81 total 493.06 last 66.95) - /unlink 37620 (time 1767321203.84 total 503.09 last 76.66) - /unlink 38486 (time 1767321213.85 total 513.10 last 86.48) - /unlink 39425 (time 1767321223.86 total 523.11 last 93.87) - /unlink 40000 (time 1767321230.79 total 530.04 last 82.96) - /unlink 40866 (time 1767321240.79 total 540.04 last 86.54) - /unlink 41662 (time 1767321250.80 total 550.05 last 79.59) - /unlink 42342 (time 1767321260.80 total 560.05 last 67.97) - /unlink 43109 (time 1767321270.81 total 570.06 last 76.65) - /unlink 43961 (time 1767321280.81 total 580.06 last 85.13) - /unlink 44831 (time 1767321290.82 total 590.07 last 86.93) - /unlink 45729 (time 1767321300.83 total 600.08 last 89.76) - /unlink 46544 (time 1767321310.83 total 610.08 last 81.48) - /unlink 47510 (time 1767321320.84 total 620.09 last 96.49) - /unlink 48268 (time 1767321330.85 total 630.10 last 75.76) - /unlink 49146 (time 1767321340.85 total 640.10 last 87.76) - /unlink 50000 (time 1767321350.64 total 649.89 last 87.28) - /unlink 50829 (time 1767321360.66 total 659.91 last 82.73) - /unlink 51678 (time 1767321370.66 total 669.91 last 84.86) - /unlink 52546 (time 1767321380.67 total 679.92 last 86.73) - /unlink 53463 (time 1767321390.67 total 689.92 last 91.66) - /unlink 54371 (time 1767321400.70 total 699.95 last 90.59) - /unlink 55120 (time 1767321410.71 total 709.96 last 74.77) - /unlink 55925 (time 1767321420.72 total 719.97 last 80.41) - /unlink 56685 (time 1767321430.74 total 729.99 last 75.89) - /unlink 57443 (time 1767321440.74 total 739.99 last 75.79) - /unlink 58189 (time 1767321450.74 total 750.00 last 74.56) - /unlink 59060 (time 1767321460.75 total 760.00 last 87.09) - /unlink 59969 (time 1767321470.75 total 770.00 last 90.89) - /unlink 60771 (time 1767321480.75 total 780.00 last 80.20) - /unlink 61501 (time 1767321490.76 total 790.01 last 72.92) - /unlink 62286 (time 1767321500.77 total 800.02 last 78.41) - /unlink 63069 (time 1767321510.78 total 810.03 last 78.25) - /unlink 63683 (time 1767321520.78 total 820.03 last 61.38) - /unlink 64348 (time 1767321530.78 total 830.03 last 66.46) total: 64768 /unlink in 835.07 seconds: 77.56 ops/second Waiting for MDT destroys to complete oleg333-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 - open/close 758 (time 1767321558.16 total 10.00 last 75.79) - open/close 1610 (time 1767321568.16 total 20.01 last 85.15) - open/close 2471 (time 1767321578.17 total 30.01 last 86.04) - open/close 3241 (time 1767321588.18 total 40.03 last 76.90) - open/close 4071 (time 1767321598.20 total 50.04 last 82.88) - open/close 4805 (time 1767321608.20 total 60.04 last 73.37) - open/close 5650 (time 1767321618.21 total 70.06 last 84.41) - open/close 6400 (time 1767321628.22 total 80.06 last 74.95) - open/close 7159 (time 1767321638.22 total 90.07 last 75.86) - open/close 7940 (time 1767321648.23 total 100.07 last 78.07) - open/close 8774 (time 1767321658.23 total 110.07 last 83.38) - open/close 9620 (time 1767321668.24 total 120.08 last 84.55) - open/close 10000 (time 1767321672.37 total 124.22 last 91.84) - open/close 10834 (time 1767321682.37 total 134.22 last 83.40) - open/close 11712 (time 1767321692.38 total 144.22 last 87.73) - open/close 12560 (time 1767321702.40 total 154.24 last 84.66) - open/close 13442 (time 1767321712.41 total 164.25 last 88.13) - open/close 14206 (time 1767321722.42 total 174.26 last 76.32) - open/close 15029 (time 1767321732.42 total 184.26 last 82.29) - open/close 15883 (time 1767321742.42 total 194.27 last 85.35) - open/close 16709 (time 1767321752.43 total 204.27 last 82.56) - open/close 17467 (time 1767321762.44 total 214.28 last 75.70) - open/close 18240 (time 1767321772.46 total 224.30 last 77.16) - open/close 19017 (time 1767321782.46 total 234.31 last 77.68) - open/close 19852 (time 1767321792.47 total 244.31 last 83.46) - open/close 20724 (time 1767321802.48 total 254.32 last 87.13) - open/close 21609 (time 1767321812.48 total 264.32 last 88.46) - open/close 22422 (time 1767321822.49 total 274.33 last 81.25) - open/close 23325 (time 1767321832.49 total 284.33 last 90.25) - open/close 24203 (time 1767321842.50 total 294.34 last 87.75) - open/close 25097 (time 1767321852.50 total 304.35 last 89.35) - open/close 25982 (time 1767321862.51 total 314.36 last 88.41) - open/close 26787 (time 1767321872.52 total 324.36 last 80.45) - open/close 27584 (time 1767321882.53 total 334.37 last 79.62) - open/close 28452 (time 1767321892.54 total 344.38 last 86.74) - open/close 29270 (time 1767321902.55 total 354.39 last 81.70) - open/close 30000 (time 1767321910.92 total 362.76 last 87.21) - open/close 30942 (time 1767321920.92 total 372.77 last 94.17) - open/close 31751 (time 1767321930.93 total 382.78 last 80.81) - open/close 32454 (time 1767321940.94 total 392.78 last 70.25) - open/close 33231 (time 1767321950.95 total 402.79 last 77.64) - open/close 33979 (time 1767321960.95 total 412.80 last 74.77) - open/close 34752 (time 1767321970.97 total 422.81 last 77.18) - open/close 35517 (time 1767321980.98 total 432.82 last 76.43) - open/close 36257 (time 1767321990.99 total 442.83 last 73.92) - open/close 36912 (time 1767322001.00 total 452.85 last 65.40) - open/close 37651 (time 1767322011.01 total 462.85 last 73.86) - open/close 38476 (time 1767322021.01 total 472.86 last 82.46) - open/close 39322 (time 1767322031.02 total 482.86 last 84.58) - open/close 40000 (time 1767322038.96 total 490.80 last 85.35) - open/close 40779 (time 1767322048.98 total 500.82 last 77.78) - open/close 41652 (time 1767322058.98 total 510.83 last 87.23) - open/close 42527 (time 1767322068.98 total 520.83 last 87.50) - open/close 43397 (time 1767322078.99 total 530.84 last 86.92) - open/close 44193 (time 1767322089.00 total 540.84 last 79.56) - open/close 45046 (time 1767322099.00 total 550.85 last 85.24) - open/close 45843 (time 1767322109.02 total 560.87 last 79.54) - open/close 46546 (time 1767322119.03 total 570.87 last 70.26) - open/close 47341 (time 1767322129.03 total 580.87 last 79.50) - open/close 48126 (time 1767322139.03 total 590.87 last 78.50) - open/close 49006 (time 1767322149.04 total 600.88 last 87.95) - open/close 49790 (time 1767322159.04 total 610.88 last 78.36) - open/close 50000 (time 1767322162.08 total 613.92 last 69.08) - open/close 50778 (time 1767322172.09 total 623.93 last 77.73) - open/close 51609 (time 1767322182.09 total 633.94 last 83.08) - open/close 52431 (time 1767322192.11 total 643.95 last 82.08) - open/close 53292 (time 1767322202.11 total 653.95 last 86.09) - open/close 54095 (time 1767322212.11 total 663.96 last 80.27) - open/close 54849 (time 1767322222.12 total 673.96 last 75.36) - open/close 55619 (time 1767322232.12 total 683.96 last 76.97) - open/close 56181 (time 1767322242.13 total 693.98 last 56.14) - open/close 56909 (time 1767322252.14 total 703.98 last 72.75) - open/close 57695 (time 1767322262.15 total 713.99 last 78.53) - open/close 58575 (time 1767322272.16 total 724.00 last 87.92) - open/close 59368 (time 1767322282.16 total 734.01 last 79.26) - open/close 60000 (time 1767322290.05 total 741.89 last 80.17) - open/close 60770 (time 1767322300.05 total 751.90 last 76.95) - open/close 61628 (time 1767322310.07 total 761.91 last 85.68) - open/close 62413 (time 1767322320.08 total 771.92 last 78.38) - open/close 63302 (time 1767322330.09 total 781.93 last 88.82) - open/close 64124 (time 1767322340.10 total 791.94 last 82.15) total: 64768 open/close in 800.37 seconds: 80.92 ops/second - /unlink 713 (time 1767322362.11 total 10.02 last 71.18) - /unlink 1394 (time 1767322372.12 total 20.03 last 68.04) - /unlink 2234 (time 1767322382.12 total 30.03 last 83.94) - /unlink 2881 (time 1767322392.14 total 40.05 last 64.59) - /unlink 3758 (time 1767322402.15 total 50.05 last 87.65) - /unlink 4611 (time 1767322412.16 total 60.07 last 85.20) - /unlink 5452 (time 1767322422.16 total 70.07 last 84.09) - /unlink 6241 (time 1767322432.17 total 80.07 last 78.85) - /unlink 7067 (time 1767322442.17 total 90.08 last 82.60) - /unlink 7858 (time 1767322452.18 total 100.09 last 79.00) - /unlink 8671 (time 1767322462.18 total 110.09 last 81.30) - /unlink 9502 (time 1767322472.19 total 120.10 last 83.01) - /unlink 10000 (time 1767322479.21 total 127.12 last 70.93) - /unlink 10653 (time 1767322489.22 total 137.13 last 65.26) - /unlink 11449 (time 1767322499.23 total 147.14 last 79.47) - /unlink 12199 (time 1767322509.24 total 157.15 last 74.96) - /unlink 12855 (time 1767322519.25 total 167.16 last 65.55) - /unlink 13586 (time 1767322529.25 total 177.16 last 73.06) - /unlink 14358 (time 1767322539.25 total 187.16 last 77.19) - /unlink 15176 (time 1767322549.26 total 197.17 last 81.74) - /unlink 16025 (time 1767322559.27 total 207.17 last 84.85) - /unlink 16860 (time 1767322569.27 total 217.18 last 83.45) - /unlink 17728 (time 1767322579.27 total 227.18 last 86.79) - /unlink 18571 (time 1767322589.29 total 237.20 last 84.19) - /unlink 19389 (time 1767322599.29 total 247.20 last 81.78) - /unlink 20000 (time 1767322605.91 total 253.82 last 92.25) - /unlink 20826 (time 1767322615.93 total 263.84 last 82.43) - /unlink 21556 (time 1767322625.94 total 273.85 last 72.94) - /unlink 22317 (time 1767322635.95 total 283.85 last 76.07) - /unlink 23075 (time 1767322645.96 total 293.86 last 75.73) - /unlink 23894 (time 1767322655.96 total 303.87 last 81.85) - /unlink 24736 (time 1767322665.96 total 313.87 last 84.20) - /unlink 25518 (time 1767322675.96 total 323.87 last 78.18) - /unlink 26314 (time 1767322685.97 total 333.88 last 79.58) - /unlink 27188 (time 1767322695.97 total 343.88 last 87.35) - /unlink 28082 (time 1767322705.97 total 353.88 last 89.38) - /unlink 29006 (time 1767322715.98 total 363.88 last 92.40) - /unlink 29919 (time 1767322725.99 total 373.90 last 91.18) - /unlink 30730 (time 1767322736.00 total 383.91 last 80.97) - /unlink 31569 (time 1767322746.01 total 393.91 last 83.89) - /unlink 32538 (time 1767322756.01 total 403.92 last 96.85) - /unlink 33351 (time 1767322766.02 total 413.93 last 81.23) - /unlink 34215 (time 1767322776.03 total 423.94 last 86.31) - /unlink 35236 (time 1767322786.03 total 433.94 last 102.09) - /unlink 36102 (time 1767322796.04 total 443.95 last 86.48) - /unlink 37044 (time 1767322806.05 total 453.96 last 94.12) - /unlink 37987 (time 1767322816.05 total 463.96 last 94.30) - /unlink 38880 (time 1767322826.06 total 473.97 last 89.25) - /unlink 39696 (time 1767322836.07 total 483.98 last 81.51) - /unlink 40000 (time 1767322839.71 total 487.62 last 83.54) - /unlink 40954 (time 1767322849.71 total 497.62 last 95.38) - /unlink 41657 (time 1767322859.71 total 507.62 last 70.29) - /unlink 42456 (time 1767322869.72 total 517.62 last 79.87) - /unlink 43459 (time 1767322879.73 total 527.63 last 100.21) - /unlink 44473 (time 1767322889.73 total 537.64 last 101.32) - /unlink 45573 (time 1767322899.74 total 547.65 last 109.92) - /unlink 46540 (time 1767322909.74 total 557.65 last 96.67) - /unlink 47527 (time 1767322919.75 total 567.66 last 98.67) - /unlink 48581 (time 1767322929.77 total 577.68 last 105.14) - /unlink 49545 (time 1767322939.80 total 587.70 last 96.16) - /unlink 50000 (time 1767322944.69 total 592.60 last 92.87) - /unlink 50824 (time 1767322954.70 total 602.61 last 82.39) - /unlink 51774 (time 1767322964.70 total 612.61 last 94.96) - /unlink 52841 (time 1767322974.70 total 622.61 last 106.66) - /unlink 53912 (time 1767322984.71 total 632.61 last 107.09) - /unlink 54953 (time 1767322994.71 total 642.62 last 104.07) - /unlink 56022 (time 1767323004.71 total 652.62 last 106.87) - /unlink 56912 (time 1767323014.71 total 662.62 last 88.98) - /unlink 57902 (time 1767323024.71 total 672.62 last 98.99) - /unlink 58901 (time 1767323034.72 total 682.63 last 99.87) - /unlink 59869 (time 1767323044.73 total 692.64 last 96.67) - /unlink 60822 (time 1767323054.75 total 702.66 last 95.08) - /unlink 61817 (time 1767323064.76 total 712.67 last 99.44) - /unlink 62956 (time 1767323074.77 total 722.68 last 113.83) - /unlink 63855 (time 1767323084.77 total 732.68 last 89.88) total: 64768 /unlink in 741.99 seconds: 87.29 ops/second Waiting for MDT destroys to complete oleg333-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 - open/close 929 (time 1767323117.46 total 10.00 last 92.88) - open/close 1936 (time 1767323127.46 total 20.01 last 100.67) - open/close 2858 (time 1767323137.46 total 30.01 last 92.18) - open/close 3679 (time 1767323147.47 total 40.02 last 82.03) - open/close 4662 (time 1767323157.47 total 50.02 last 98.28) - open/close 5584 (time 1767323167.48 total 60.03 last 92.10) - open/close 6491 (time 1767323177.49 total 70.04 last 90.64) - open/close 7400 (time 1767323187.50 total 80.04 last 90.85) - open/close 8335 (time 1767323197.50 total 90.05 last 93.47) - open/close 9258 (time 1767323207.52 total 100.06 last 92.13) - open/close 10000 (time 1767323214.89 total 107.43 last 100.69) - open/close 11003 (time 1767323224.89 total 117.44 last 100.26) - open/close 12009 (time 1767323234.89 total 127.44 last 100.59) - open/close 12975 (time 1767323244.90 total 137.45 last 96.49) - open/close 14039 (time 1767323254.91 total 147.46 last 106.35) - open/close 15017 (time 1767323264.92 total 157.46 last 97.74) - open/close 16088 (time 1767323274.92 total 167.46 last 107.07) - open/close 17088 (time 1767323284.93 total 177.47 last 99.92) - open/close 18014 (time 1767323294.94 total 187.48 last 92.49) - open/close 18989 (time 1767323304.95 total 197.50 last 97.39) - open/close 19979 (time 1767323314.95 total 207.50 last 99.00) - open/close 20978 (time 1767323324.96 total 217.50 last 99.84) - open/close 21938 (time 1767323334.96 total 227.51 last 95.91) - open/close 22882 (time 1767323344.97 total 237.52 last 94.35) - open/close 23882 (time 1767323354.98 total 247.53 last 99.89) - open/close 24887 (time 1767323364.99 total 257.53 last 100.46) - open/close 25925 (time 1767323374.99 total 267.53 last 103.78) - open/close 26994 (time 1767323384.99 total 277.53 last 106.89) - open/close 27964 (time 1767323394.99 total 287.53 last 96.99) - open/close 29007 (time 1767323405.00 total 297.54 last 104.22) - open/close 30000 (time 1767323413.80 total 306.35 last 112.79) - open/close 31019 (time 1767323423.80 total 316.35 last 101.88) - open/close 32085 (time 1767323433.82 total 326.37 last 106.40) - open/close 33110 (time 1767323443.82 total 336.37 last 102.50) - open/close 34128 (time 1767323453.83 total 346.37 last 101.73) - open/close 35216 (time 1767323463.84 total 356.38 last 108.69) - open/close 36357 (time 1767323473.84 total 366.39 last 114.05) - open/close 37409 (time 1767323483.84 total 376.39 last 105.19) - open/close 38572 (time 1767323493.85 total 386.39 last 116.23) - open/close 39608 (time 1767323503.85 total 396.40 last 103.60) - open/close 40000 (time 1767323507.59 total 400.14 last 104.68) - open/close 41116 (time 1767323517.60 total 410.15 last 111.53) - open/close 42247 (time 1767323527.60 total 420.15 last 113.09) - open/close 43307 (time 1767323537.61 total 430.15 last 105.93) - open/close 44390 (time 1767323547.61 total 440.16 last 108.24) - open/close 45546 (time 1767323557.62 total 450.17 last 115.53) - open/close 46650 (time 1767323567.63 total 460.17 last 110.34) - open/close 47707 (time 1767323577.63 total 470.18 last 105.60) - open/close 48899 (time 1767323587.64 total 480.19 last 119.11) - open/close 50000 (time 1767323596.56 total 489.11 last 123.44) - open/close 51125 (time 1767323606.56 total 499.11 last 112.49) - open/close 52229 (time 1767323616.56 total 509.11 last 110.39) - open/close 53451 (time 1767323626.57 total 519.11 last 122.18) - open/close 54653 (time 1767323636.57 total 529.11 last 120.19) - open/close 55913 (time 1767323646.57 total 539.11 last 125.96) - open/close 57147 (time 1767323656.57 total 549.12 last 123.38) - open/close 58240 (time 1767323666.58 total 559.12 last 109.25) - open/close 59352 (time 1767323676.59 total 569.13 last 111.09) - open/close 60000 (time 1767323681.89 total 574.44 last 122.08) - open/close 61305 (time 1767323691.90 total 584.44 last 130.46) - open/close 62655 (time 1767323701.90 total 594.44 last 134.96) - open/close 63794 (time 1767323711.90 total 604.45 last 113.89) total: 64768 open/close in 612.27 seconds: 105.78 ops/second - /unlink 1373 (time 1767323731.66 total 10.00 last 137.27) - /unlink 2850 (time 1767323741.67 total 20.01 last 147.63) - /unlink 4812 (time 1767323751.67 total 30.01 last 196.15) - /unlink 7301 (time 1767323761.67 total 40.01 last 248.86) - /unlink 10000 (time 1767323771.62 total 49.96 last 271.33) - /unlink 12427 (time 1767323781.62 total 59.96 last 242.67) - /unlink 14724 (time 1767323791.62 total 69.96 last 229.60) - /unlink 17144 (time 1767323801.63 total 79.96 last 241.97) - /unlink 19896 (time 1767323811.63 total 89.96 last 275.20) - /unlink 22862 (time 1767323821.63 total 99.97 last 296.52) - /unlink 25796 (time 1767323831.63 total 109.97 last 293.37) - /unlink 28448 (time 1767323841.63 total 119.97 last 265.07) - /unlink 30000 (time 1767323848.12 total 126.46 last 239.24) - /unlink 33132 (time 1767323858.12 total 136.46 last 313.18) - /unlink 36305 (time 1767323868.12 total 146.46 last 317.27) - /unlink 38488 (time 1767323878.13 total 156.46 last 218.24) - /unlink 40000 (time 1767323885.43 total 163.77 last 207.01) - /unlink 42174 (time 1767323895.43 total 173.77 last 217.33) - /unlink 44817 (time 1767323905.43 total 183.77 last 264.29) - /unlink 47773 (time 1767323915.44 total 193.77 last 295.53) - /unlink 50000 (time 1767323923.98 total 202.31 last 260.77) - /unlink 52434 (time 1767323933.98 total 212.31 last 243.39) - /unlink 55344 (time 1767323943.98 total 222.31 last 291.00) - /unlink 58583 (time 1767323953.98 total 232.32 last 323.87) - /unlink 60000 (time 1767323958.48 total 236.82 last 314.60) - /unlink 62800 (time 1767323968.49 total 246.83 last 279.80) total: 64768 /unlink in 253.70 seconds: 255.30 ops/second Waiting for MDT destroys to complete oleg333-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 - open/close 1399 (time 1767323995.41 total 10.01 last 139.81) - open/close 2721 (time 1767324005.42 total 20.01 last 132.16) - open/close 4291 (time 1767324015.42 total 30.01 last 156.95) - open/close 5698 (time 1767324025.42 total 40.02 last 140.67) - open/close 7063 (time 1767324035.42 total 50.02 last 136.47) - open/close 8481 (time 1767324045.43 total 60.02 last 141.78) - open/close 10000 (time 1767324053.16 total 67.75 last 196.43) - open/close 12233 (time 1767324063.16 total 77.75 last 223.26) - open/close 14069 (time 1767324073.16 total 87.75 last 183.59) - open/close 15808 (time 1767324083.17 total 97.76 last 173.80) - open/close 17211 (time 1767324093.17 total 107.77 last 140.21) - open/close 18832 (time 1767324103.18 total 117.77 last 162.07) - open/close 20000 (time 1767324108.61 total 123.20 last 215.05) - open/close 21895 (time 1767324118.61 total 133.20 last 189.48) - open/close 23294 (time 1767324128.61 total 143.21 last 139.82) - open/close 24815 (time 1767324138.61 total 153.21 last 152.09) - open/close 26740 (time 1767324148.61 total 163.21 last 192.49) - open/close 28563 (time 1767324158.62 total 173.21 last 182.26) - open/close 30000 (time 1767324167.44 total 182.03 last 162.83) - open/close 31365 (time 1767324177.44 total 192.04 last 136.48) - open/close 32802 (time 1767324187.46 total 202.05 last 143.52) - open/close 34495 (time 1767324197.46 total 212.05 last 169.27) - open/close 36077 (time 1767324207.46 total 222.05 last 158.14) - open/close 37614 (time 1767324217.47 total 232.06 last 153.62) - open/close 39247 (time 1767324227.47 total 242.06 last 163.29) - open/close 40000 (time 1767324231.97 total 246.56 last 167.33) - open/close 42233 (time 1767324241.97 total 256.57 last 223.18) - open/close 44127 (time 1767324251.98 total 266.57 last 189.30) - open/close 45883 (time 1767324261.98 total 276.58 last 175.50) - open/close 47303 (time 1767324271.99 total 286.58 last 141.91) - open/close 49142 (time 1767324281.99 total 296.58 last 183.89) - open/close 50000 (time 1767324287.21 total 301.80 last 164.49) - open/close 51530 (time 1767324297.21 total 311.80 last 153.00) - open/close 53371 (time 1767324307.21 total 321.80 last 184.03) - open/close 55005 (time 1767324317.21 total 331.80 last 163.38) - open/close 57049 (time 1767324327.21 total 341.80 last 204.40) - open/close 59031 (time 1767324337.22 total 351.81 last 198.02) - open/close 60000 (time 1767324342.53 total 357.12 last 182.63) - open/close 61494 (time 1767324352.53 total 367.13 last 149.32) - open/close 62932 (time 1767324362.53 total 377.13 last 143.78) - open/close 64653 (time 1767324372.54 total 387.13 last 172.04) total: 64768 open/close in 387.87 seconds: 166.98 ops/second - /unlink 2499 (time 1767324385.18 total 10.00 last 249.90) - /unlink 4964 (time 1767324395.18 total 20.00 last 246.48) - /unlink 7358 (time 1767324405.19 total 30.00 last 239.32) - /unlink 9933 (time 1767324415.20 total 40.01 last 257.26) - /unlink 13067 (time 1767324425.20 total 50.01 last 313.39) - /unlink 15962 (time 1767324435.20 total 60.02 last 289.37) - /unlink 19486 (time 1767324445.20 total 70.02 last 352.31) - /unlink 22901 (time 1767324455.21 total 80.02 last 341.41) - /unlink 26755 (time 1767324465.21 total 90.02 last 385.39) - /unlink 30000 (time 1767324474.23 total 99.05 last 359.63) - /unlink 34190 (time 1767324484.23 total 109.05 last 418.98) - /unlink 37550 (time 1767324494.23 total 119.05 last 335.87) - /unlink 40000 (time 1767324502.37 total 127.19 last 301.16) - /unlink 44066 (time 1767324512.37 total 137.19 last 406.58) - /unlink 47324 (time 1767324522.37 total 147.19 last 325.73) - /unlink 50000 (time 1767324530.16 total 154.98 last 343.58) - /unlink 53225 (time 1767324540.16 total 164.98 last 322.49) - /unlink 56530 (time 1767324550.16 total 174.98 last 330.43) - /unlink 59346 (time 1767324560.16 total 184.98 last 281.54) - /unlink 60000 (time 1767324562.40 total 187.22 last 292.77) - /unlink 63953 (time 1767324572.40 total 197.22 last 395.30) total: 64768 /unlink in 200.44 seconds: 323.13 ops/second Waiting for MDT destroys to complete oleg333-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 - open/close 2198 (time 1767324595.13 total 10.00 last 219.77) - open/close 3555 (time 1767324605.13 total 20.00 last 135.67) - open/close 5553 (time 1767324615.14 total 30.01 last 199.77) - open/close 7373 (time 1767324625.14 total 40.01 last 181.99) - open/close 9422 (time 1767324635.14 total 50.01 last 204.87) - open/close 10000 (time 1767324637.50 total 52.37 last 244.97) - open/close 12224 (time 1767324647.50 total 62.37 last 222.33) - open/close 14642 (time 1767324657.50 total 72.37 last 241.77) - open/close 16382 (time 1767324667.50 total 82.37 last 173.96) - open/close 18505 (time 1767324677.51 total 92.38 last 212.26) - open/close 20000 (time 1767324685.47 total 100.34 last 187.79) - open/close 21639 (time 1767324695.47 total 110.34 last 163.84) - open/close 23141 (time 1767324705.47 total 120.34 last 150.18) - open/close 25384 (time 1767324715.48 total 130.35 last 224.04) - open/close 27796 (time 1767324725.48 total 140.35 last 241.18) - open/close 29941 (time 1767324735.49 total 150.36 last 214.46) - open/close 32082 (time 1767324745.49 total 160.36 last 213.94) - open/close 34576 (time 1767324755.50 total 170.36 last 249.35) - open/close 37706 (time 1767324765.50 total 180.37 last 312.92) - open/close 40000 (time 1767324774.03 total 188.90 last 268.86) - open/close 42710 (time 1767324784.03 total 198.90 last 270.97) - open/close 44758 (time 1767324794.03 total 208.90 last 204.76) - open/close 47477 (time 1767324804.04 total 218.91 last 271.74) - open/close 50000 (time 1767324812.13 total 227.00 last 311.80) - open/close 52279 (time 1767324822.13 total 237.00 last 227.85) - open/close 54659 (time 1767324832.14 total 247.01 last 237.92) - open/close 57541 (time 1767324842.14 total 257.01 last 288.19) - open/close 60000 (time 1767324850.30 total 265.17 last 301.06) - open/close 62477 (time 1767324860.30 total 275.17 last 247.70) total: 64768 open/close in 283.86 seconds: 228.17 ops/second - /unlink 5276 (time 1767324880.53 total 10.00 last 527.51) - /unlink 10000 (time 1767324889.95 total 19.42 last 501.59) - /unlink 15372 (time 1767324899.95 total 29.42 last 537.18) - /unlink 20000 (time 1767324907.16 total 36.63 last 641.92) - /unlink 24658 (time 1767324917.16 total 46.63 last 465.80) - /unlink 30000 (time 1767324926.28 total 55.75 last 585.95) - /unlink 36160 (time 1767324936.28 total 65.75 last 615.99) - /unlink 40000 (time 1767324941.89 total 71.36 last 684.25) - /unlink 47164 (time 1767324951.89 total 81.36 last 716.40) - /unlink 50000 (time 1767324955.72 total 85.19 last 740.00) - /unlink 57569 (time 1767324965.72 total 95.19 last 756.86) - /unlink 60000 (time 1767324968.98 total 98.45 last 746.86) total: 64768 /unlink in 104.39 seconds: 620.42 ops/second Waiting for MDT destroys to complete oleg333-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local oleg333-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg333-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2