== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:55:25 (1772470525) Stopping clients: oleg446-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg446-client.virtnet /mnt/lustre2 (opts:-f) oleg446-server: oleg446-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing load_modules_local oleg446-server: Loading modules from /home/green/git/lustre-release/lustre oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg446-server: detected 4 online CPUs by sysfs oleg446-server: MODOPTS_LIBCFS= oleg446-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg446-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg446-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg446-client.virtnet: -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 646 (time 1772470648.03 total 10.00 last 64.57) - open/close 1331 (time 1772470658.03 total 20.01 last 68.49) - open/close 2065 (time 1772470668.05 total 30.02 last 73.32) - open/close 2726 (time 1772470678.05 total 40.03 last 66.04) - open/close 3451 (time 1772470688.07 total 50.04 last 72.39) - open/close 4221 (time 1772470698.08 total 60.05 last 76.95) - open/close 4963 (time 1772470708.08 total 70.06 last 74.14) - open/close 5711 (time 1772470718.09 total 80.06 last 74.79) - open/close 6540 (time 1772470728.09 total 90.06 last 82.87) - open/close 7198 (time 1772470738.10 total 100.07 last 65.73) - open/close 7762 (time 1772470748.11 total 110.08 last 56.34) - open/close 8422 (time 1772470758.12 total 120.09 last 65.93) - open/close 9073 (time 1772470768.13 total 130.10 last 65.05) - open/close 9801 (time 1772470778.13 total 140.10 last 72.78) - open/close 10000 (time 1772470780.98 total 142.95 last 69.91) - open/close 10680 (time 1772470790.99 total 152.96 last 67.92) - open/close 11369 (time 1772470801.00 total 162.97 last 68.85) - open/close 11975 (time 1772470811.01 total 172.98 last 60.55) - open/close 12648 (time 1772470821.01 total 182.99 last 67.25) - open/close 13402 (time 1772470831.03 total 193.00 last 75.27) - open/close 14025 (time 1772470841.04 total 203.01 last 62.25) - open/close 14724 (time 1772470851.05 total 213.02 last 69.82) - open/close 15266 (time 1772470861.06 total 223.03 last 54.15) - open/close 15862 (time 1772470871.07 total 233.04 last 59.53) - open/close 16568 (time 1772470881.08 total 243.05 last 70.53) - open/close 17329 (time 1772470891.10 total 253.07 last 75.94) - open/close 18010 (time 1772470901.11 total 263.08 last 68.05) - open/close 18660 (time 1772470911.11 total 273.09 last 64.97) - open/close 19288 (time 1772470921.12 total 283.09 last 62.75) - open/close 19904 (time 1772470931.12 total 293.10 last 61.59) - open/close 20481 (time 1772470941.13 total 303.10 last 57.65) - open/close 21171 (time 1772470951.14 total 313.11 last 68.97) - open/close 21816 (time 1772470961.15 total 323.12 last 64.45) - open/close 22422 (time 1772470971.17 total 333.14 last 60.47) - open/close 23064 (time 1772470981.17 total 343.14 last 64.19) - open/close 23733 (time 1772470991.17 total 353.14 last 66.86) - open/close 24413 (time 1772471001.21 total 363.18 last 67.76) - open/close 25052 (time 1772471011.22 total 373.19 last 63.85) - open/close 25795 (time 1772471021.22 total 383.19 last 74.25) - open/close 26518 (time 1772471031.23 total 393.20 last 72.28) - open/close 27161 (time 1772471041.23 total 403.21 last 64.24) - open/close 27875 (time 1772471051.25 total 413.23 last 71.26) - open/close 28529 (time 1772471061.26 total 423.23 last 65.35) - open/close 29300 (time 1772471071.27 total 433.24 last 77.04) - open/close 30000 (time 1772471080.97 total 442.94 last 72.19) - open/close 30844 (time 1772471090.97 total 452.94 last 84.37) - open/close 31529 (time 1772471100.97 total 462.95 last 68.47) - open/close 32123 (time 1772471110.99 total 472.96 last 59.32) - open/close 32800 (time 1772471121.00 total 482.97 last 67.65) - open/close 33447 (time 1772471131.01 total 492.98 last 64.62) - open/close 34102 (time 1772471141.02 total 502.99 last 65.44) - open/close 34811 (time 1772471151.02 total 513.00 last 70.86) - open/close 35606 (time 1772471161.03 total 523.00 last 79.44) - open/close 36366 (time 1772471171.04 total 533.02 last 75.91) - open/close 37104 (time 1772471181.04 total 543.02 last 73.80) - open/close 37799 (time 1772471191.05 total 553.03 last 69.43) - open/close 38513 (time 1772471201.07 total 563.04 last 71.32) - open/close 39171 (time 1772471211.09 total 573.06 last 65.63) - open/close 39921 (time 1772471221.10 total 583.07 last 74.93) - open/close 40647 (time 1772471231.10 total 593.07 last 72.59) - open/close 41411 (time 1772471241.11 total 603.08 last 76.37) - open/close 42165 (time 1772471251.11 total 613.08 last 75.39) - open/close 42866 (time 1772471261.11 total 623.08 last 70.09) - open/close 43592 (time 1772471271.11 total 633.08 last 72.58) - open/close 44328 (time 1772471281.12 total 643.09 last 73.56) - open/close 45049 (time 1772471291.12 total 653.09 last 72.07) - open/close 45801 (time 1772471301.13 total 663.11 last 75.10) - open/close 46477 (time 1772471311.15 total 673.12 last 67.52) - open/close 47180 (time 1772471321.15 total 683.12 last 70.25) - open/close 47824 (time 1772471331.16 total 693.13 last 64.35) - open/close 48531 (time 1772471341.16 total 703.13 last 70.69) - open/close 49118 (time 1772471351.17 total 713.14 last 58.64) - open/close 49725 (time 1772471361.23 total 723.20 last 60.37) - open/close 50000 (time 1772471366.72 total 728.69 last 50.04) - open/close 50608 (time 1772471376.73 total 738.71 last 60.74) - open/close 51198 (time 1772471386.74 total 748.71 last 58.98) - open/close 51921 (time 1772471396.74 total 758.71 last 72.25) - open/close 52565 (time 1772471406.75 total 768.72 last 64.36) - open/close 53249 (time 1772471416.76 total 778.73 last 68.36) - open/close 53854 (time 1772471426.76 total 788.73 last 60.47) - open/close 54419 (time 1772471436.77 total 798.74 last 56.47) - open/close 55042 (time 1772471446.80 total 808.78 last 62.07) - open/close 55793 (time 1772471456.81 total 818.78 last 75.03) - open/close 56431 (time 1772471466.84 total 828.81 last 63.64) - open/close 57015 (time 1772471476.84 total 838.82 last 58.37) - open/close 57617 (time 1772471486.84 total 848.82 last 60.20) - open/close 58296 (time 1772471496.85 total 858.82 last 67.88) - open/close 58962 (time 1772471506.85 total 868.82 last 66.58) - open/close 59563 (time 1772471516.86 total 878.83 last 60.06) - open/close 60000 (time 1772471525.83 total 887.80 last 48.72) - open/close 60629 (time 1772471535.83 total 897.81 last 62.85) - open/close 61367 (time 1772471545.84 total 907.81 last 73.79) - open/close 61989 (time 1772471555.84 total 917.81 last 62.16) - open/close 62723 (time 1772471565.85 total 927.83 last 73.31) - open/close 63310 (time 1772471575.86 total 937.83 last 58.68) - open/close 63952 (time 1772471585.87 total 947.84 last 64.11) - open/close 64492 (time 1772471595.88 total 957.85 last 53.98) total: 64768 open/close in 963.34 seconds: 67.23 ops/second - /unlink 558 (time 1772471615.64 total 10.03 last 55.64) - /unlink 1189 (time 1772471625.65 total 20.04 last 63.03) - /unlink 1799 (time 1772471635.65 total 30.04 last 60.99) - /unlink 2388 (time 1772471645.66 total 40.06 last 58.81) - /unlink 3042 (time 1772471655.67 total 50.06 last 65.36) - /unlink 3647 (time 1772471665.67 total 60.07 last 60.48) - /unlink 4266 (time 1772471675.68 total 70.07 last 61.85) - /unlink 4947 (time 1772471685.69 total 80.09 last 68.01) - /unlink 5565 (time 1772471695.71 total 90.10 last 61.73) - /unlink 6280 (time 1772471705.73 total 100.12 last 71.31) - /unlink 6925 (time 1772471715.74 total 110.13 last 64.46) - /unlink 7493 (time 1772471725.74 total 120.13 last 56.79) - /unlink 8070 (time 1772471735.75 total 130.14 last 57.63) - /unlink 8702 (time 1772471745.76 total 140.15 last 63.17) - /unlink 9357 (time 1772471755.76 total 150.15 last 65.46) - /unlink 9977 (time 1772471765.77 total 160.16 last 61.94) - /unlink 10656 (time 1772471775.78 total 170.17 last 67.83) - /unlink 11253 (time 1772471785.79 total 180.19 last 59.62) - /unlink 11802 (time 1772471795.80 total 190.19 last 54.86) - /unlink 12517 (time 1772471805.81 total 200.20 last 71.48) - /unlink 13203 (time 1772471815.81 total 210.21 last 68.54) - /unlink 13858 (time 1772471825.82 total 220.21 last 65.46) - /unlink 14565 (time 1772471835.84 total 230.23 last 70.57) - /unlink 15169 (time 1772471845.85 total 240.25 last 60.30) - /unlink 15782 (time 1772471855.86 total 250.25 last 61.25) - /unlink 16339 (time 1772471865.87 total 260.26 last 55.65) - /unlink 16965 (time 1772471875.88 total 270.27 last 62.57) - /unlink 17559 (time 1772471885.89 total 280.28 last 59.32) - /unlink 18197 (time 1772471895.89 total 290.29 last 63.78) - /unlink 18773 (time 1772471905.90 total 300.29 last 57.56) - /unlink 19364 (time 1772471915.90 total 310.29 last 59.10) - /unlink 20000 (time 1772471925.77 total 320.16 last 64.46) - /unlink 20676 (time 1772471935.77 total 330.16 last 67.58) - /unlink 21256 (time 1772471945.78 total 340.17 last 57.93) - /unlink 21814 (time 1772471955.78 total 350.18 last 55.79) - /unlink 22407 (time 1772471965.79 total 360.18 last 59.27) - /unlink 22990 (time 1772471975.79 total 370.19 last 58.28) - /unlink 23709 (time 1772471985.80 total 380.19 last 71.88) - /unlink 24235 (time 1772471995.80 total 390.19 last 52.58) - /unlink 24800 (time 1772472005.80 total 400.19 last 56.50) - /unlink 25442 (time 1772472015.82 total 410.21 last 64.11) - /unlink 26013 (time 1772472025.83 total 420.22 last 57.02) - /unlink 26605 (time 1772472035.84 total 430.23 last 59.12) - /unlink 27261 (time 1772472045.85 total 440.24 last 65.55) - /unlink 27829 (time 1772472055.86 total 450.25 last 56.74) - /unlink 28439 (time 1772472065.86 total 460.25 last 60.99) - /unlink 29080 (time 1772472075.86 total 470.26 last 64.08) - /unlink 29643 (time 1772472085.89 total 480.28 last 56.17) - /unlink 30000 (time 1772472092.50 total 486.89 last 53.97) - /unlink 30700 (time 1772472102.55 total 496.94 last 69.66) - /unlink 31413 (time 1772472112.57 total 506.96 last 71.17) - /unlink 31954 (time 1772472122.59 total 516.98 last 53.98) - /unlink 32467 (time 1772472132.61 total 527.00 last 51.23) - /unlink 33034 (time 1772472142.61 total 537.00 last 56.69) - /unlink 33705 (time 1772472152.61 total 547.00 last 67.09) - /unlink 34351 (time 1772472162.62 total 557.01 last 64.53) - /unlink 35019 (time 1772472172.63 total 567.02 last 66.73) - /unlink 35801 (time 1772472182.63 total 577.02 last 78.20) - /unlink 36430 (time 1772472192.64 total 587.03 last 62.86) - /unlink 37125 (time 1772472202.68 total 597.07 last 69.24) - /unlink 37874 (time 1772472212.68 total 607.07 last 74.90) - /unlink 38562 (time 1772472222.68 total 617.08 last 68.75) - /unlink 39138 (time 1772472232.70 total 627.10 last 57.49) - /unlink 39872 (time 1772472242.71 total 637.10 last 73.39) - /unlink 40585 (time 1772472252.71 total 647.10 last 71.28) - /unlink 41262 (time 1772472262.72 total 657.11 last 67.65) - /unlink 42002 (time 1772472272.72 total 667.12 last 73.95) - /unlink 42727 (time 1772472282.79 total 677.18 last 72.05) - /unlink 43414 (time 1772472292.79 total 687.18 last 68.66) - /unlink 44047 (time 1772472302.80 total 697.19 last 63.27) - /unlink 44769 (time 1772472312.81 total 707.20 last 72.12) - /unlink 45505 (time 1772472322.82 total 717.21 last 73.49) - /unlink 46291 (time 1772472332.85 total 727.24 last 78.39) - /unlink 46984 (time 1772472342.86 total 737.25 last 69.25) - /unlink 47722 (time 1772472352.90 total 747.29 last 73.51) - /unlink 48511 (time 1772472362.90 total 757.30 last 78.83) - /unlink 49276 (time 1772472372.92 total 767.31 last 76.39) - /unlink 50000 (time 1772472382.45 total 776.84 last 75.94) - /unlink 50901 (time 1772472392.46 total 786.85 last 90.05) - /unlink 51643 (time 1772472402.47 total 796.86 last 74.11) - /unlink 52263 (time 1772472412.48 total 806.87 last 61.95) - /unlink 53049 (time 1772472422.48 total 816.87 last 78.59) - /unlink 53856 (time 1772472432.49 total 826.88 last 80.65) - /unlink 54639 (time 1772472442.49 total 836.88 last 78.27) - /unlink 55321 (time 1772472452.50 total 846.89 last 68.14) - /unlink 55930 (time 1772472462.51 total 856.90 last 60.81) - /unlink 56562 (time 1772472472.53 total 866.92 last 63.09) - /unlink 57268 (time 1772472482.54 total 876.93 last 70.55) - /unlink 58004 (time 1772472492.54 total 886.93 last 73.55) - /unlink 58699 (time 1772472502.54 total 896.94 last 69.50) - /unlink 59408 (time 1772472512.55 total 906.94 last 70.85) - /unlink 60000 (time 1772472519.35 total 913.74 last 87.06) - /unlink 60775 (time 1772472529.35 total 923.74 last 77.49) - /unlink 61440 (time 1772472539.35 total 933.74 last 66.50) - /unlink 62180 (time 1772472549.35 total 943.74 last 73.99) - /unlink 62958 (time 1772472559.36 total 953.75 last 77.76) - /unlink 63642 (time 1772472569.36 total 963.76 last 68.35) - /unlink 64251 (time 1772472579.37 total 973.76 last 60.85) total: 64768 /unlink in 982.19 seconds: 65.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 757 (time 1772472627.33 total 10.03 last 75.50) - open/close 1523 (time 1772472637.34 total 20.03 last 76.54) - open/close 2434 (time 1772472647.35 total 30.04 last 91.05) - open/close 3149 (time 1772472657.35 total 40.04 last 71.47) - open/close 3881 (time 1772472667.35 total 50.04 last 73.19) - open/close 4703 (time 1772472677.36 total 60.05 last 82.14) - open/close 5476 (time 1772472687.37 total 70.06 last 77.22) - open/close 6394 (time 1772472697.37 total 80.07 last 91.76) - open/close 7114 (time 1772472707.39 total 90.08 last 71.91) - open/close 7904 (time 1772472717.39 total 100.08 last 78.95) - open/close 8765 (time 1772472727.40 total 110.09 last 86.08) - open/close 9590 (time 1772472737.41 total 120.10 last 82.39) - open/close 10000 (time 1772472743.06 total 125.75 last 72.57) - open/close 10810 (time 1772472753.07 total 135.76 last 80.91) - open/close 11633 (time 1772472763.07 total 145.76 last 82.29) - open/close 12455 (time 1772472773.07 total 155.76 last 82.19) - open/close 13068 (time 1772472783.08 total 165.77 last 61.26) - open/close 13842 (time 1772472793.08 total 175.77 last 77.38) - open/close 14656 (time 1772472803.09 total 185.79 last 81.29) - open/close 15544 (time 1772472813.11 total 195.80 last 88.68) - open/close 16322 (time 1772472823.13 total 205.82 last 77.62) - open/close 17094 (time 1772472833.16 total 215.85 last 76.99) - open/close 17891 (time 1772472843.17 total 225.86 last 79.59) - open/close 18683 (time 1772472853.18 total 235.87 last 79.14) - open/close 19348 (time 1772472863.21 total 245.90 last 66.33) - open/close 20000 (time 1772472872.78 total 255.47 last 68.07) - open/close 20738 (time 1772472882.79 total 265.49 last 73.72) - open/close 21447 (time 1772472892.80 total 275.50 last 70.83) - open/close 22150 (time 1772472902.81 total 285.50 last 70.29) - open/close 22841 (time 1772472912.81 total 295.50 last 69.08) - open/close 23591 (time 1772472922.81 total 305.50 last 74.96) - open/close 24313 (time 1772472932.82 total 315.52 last 72.12) - open/close 24931 (time 1772472942.83 total 325.52 last 61.76) - open/close 25625 (time 1772472952.84 total 335.53 last 69.33) - open/close 26318 (time 1772472962.84 total 345.53 last 69.29) - open/close 26988 (time 1772472972.86 total 355.55 last 66.88) - open/close 27796 (time 1772472982.87 total 365.56 last 80.75) - open/close 28582 (time 1772472992.87 total 375.56 last 78.59) - open/close 29304 (time 1772473002.88 total 385.57 last 72.13) - open/close 29885 (time 1772473012.90 total 395.59 last 57.95) - open/close 30000 (time 1772473015.12 total 397.81 last 51.84) - open/close 30625 (time 1772473025.13 total 407.82 last 62.42) - open/close 31291 (time 1772473035.14 total 417.84 last 66.53) - open/close 32089 (time 1772473045.15 total 427.84 last 79.79) - open/close 32801 (time 1772473055.15 total 437.84 last 71.18) - open/close 33545 (time 1772473065.15 total 447.84 last 74.38) - open/close 34292 (time 1772473075.16 total 457.85 last 74.66) - open/close 35056 (time 1772473085.16 total 467.85 last 76.37) - open/close 35896 (time 1772473095.17 total 477.86 last 83.93) - open/close 36632 (time 1772473105.18 total 487.87 last 73.51) - open/close 37168 (time 1772473115.19 total 497.88 last 53.57) - open/close 37879 (time 1772473125.20 total 507.89 last 71.00) - open/close 38544 (time 1772473135.21 total 517.90 last 66.43) - open/close 39112 (time 1772473145.22 total 527.91 last 56.78) - open/close 39687 (time 1772473155.23 total 537.92 last 57.42) - open/close 40000 (time 1772473159.18 total 541.88 last 79.16) - open/close 40724 (time 1772473169.19 total 551.88 last 72.34) - open/close 41407 (time 1772473179.19 total 561.89 last 68.29) - open/close 42225 (time 1772473189.20 total 571.89 last 81.73) - open/close 42965 (time 1772473199.21 total 581.90 last 73.92) - open/close 43801 (time 1772473209.24 total 591.94 last 83.34) - open/close 44598 (time 1772473219.25 total 601.94 last 79.65) - open/close 45206 (time 1772473229.26 total 611.95 last 60.77) - open/close 45765 (time 1772473239.27 total 621.97 last 55.80) - open/close 46454 (time 1772473249.28 total 631.97 last 68.85) - open/close 47024 (time 1772473259.30 total 641.99 last 56.91) - open/close 47636 (time 1772473269.33 total 652.02 last 61.03) - open/close 48192 (time 1772473279.34 total 662.03 last 55.54) - open/close 48694 (time 1772473289.35 total 672.04 last 50.11) - open/close 49384 (time 1772473299.37 total 682.06 last 68.90) - open/close 50000 (time 1772473308.78 total 691.47 last 65.43) - open/close 50592 (time 1772473318.79 total 701.48 last 59.16) - open/close 51313 (time 1772473328.80 total 711.50 last 71.98) - open/close 52150 (time 1772473338.81 total 721.50 last 83.69) - open/close 52933 (time 1772473348.81 total 731.51 last 78.24) - open/close 53706 (time 1772473358.82 total 741.51 last 77.29) - open/close 54423 (time 1772473368.83 total 751.53 last 71.57) - open/close 55218 (time 1772473378.84 total 761.53 last 79.46) - open/close 56078 (time 1772473388.85 total 771.54 last 85.89) - open/close 56852 (time 1772473398.85 total 781.54 last 77.39) - open/close 57492 (time 1772473408.86 total 791.55 last 63.94) - open/close 58281 (time 1772473418.86 total 801.55 last 78.90) - open/close 59093 (time 1772473428.87 total 811.56 last 81.16) - open/close 59736 (time 1772473438.88 total 821.58 last 64.18) - open/close 60000 (time 1772473443.58 total 826.27 last 56.21) - open/close 60714 (time 1772473453.58 total 836.27 last 71.40) - open/close 61495 (time 1772473463.59 total 846.28 last 78.06) - open/close 62302 (time 1772473473.60 total 856.29 last 80.58) - open/close 63191 (time 1772473483.60 total 866.29 last 88.89) - open/close 63934 (time 1772473493.62 total 876.31 last 74.18) total: 64768 open/close in 885.26 seconds: 73.16 ops/second - /unlink 783 (time 1772473516.90 total 10.01 last 78.21) - /unlink 1686 (time 1772473526.94 total 20.05 last 89.97) - /unlink 2549 (time 1772473536.95 total 30.05 last 86.26) - /unlink 3438 (time 1772473546.96 total 40.06 last 88.82) - /unlink 4393 (time 1772473556.97 total 50.07 last 95.39) - /unlink 5226 (time 1772473566.97 total 60.08 last 83.24) - /unlink 5981 (time 1772473576.98 total 70.08 last 75.49) - /unlink 6900 (time 1772473586.98 total 80.09 last 91.87) - /unlink 7780 (time 1772473596.99 total 90.10 last 87.91) - /unlink 8716 (time 1772473607.00 total 100.11 last 93.48) - /unlink 9539 (time 1772473617.02 total 110.12 last 82.19) - /unlink 10000 (time 1772473622.56 total 115.67 last 83.08) - /unlink 10920 (time 1772473632.57 total 125.67 last 91.98) - /unlink 11725 (time 1772473642.57 total 135.68 last 80.48) - /unlink 12713 (time 1772473652.57 total 145.68 last 98.75) - /unlink 13584 (time 1772473662.58 total 155.68 last 87.09) - /unlink 14433 (time 1772473672.58 total 165.68 last 84.88) - /unlink 15320 (time 1772473682.58 total 175.69 last 88.66) - /unlink 16268 (time 1772473692.60 total 185.70 last 94.68) - /unlink 17281 (time 1772473702.60 total 195.71 last 101.21) - /unlink 18201 (time 1772473712.61 total 205.72 last 91.92) - /unlink 19181 (time 1772473722.62 total 215.73 last 97.91) - /unlink 20000 (time 1772473730.62 total 223.72 last 102.44) - /unlink 21116 (time 1772473740.62 total 233.73 last 111.56) - /unlink 22236 (time 1772473750.63 total 243.73 last 111.95) - /unlink 23401 (time 1772473760.63 total 253.74 last 116.44) - /unlink 24568 (time 1772473770.64 total 263.75 last 116.56) - /unlink 25497 (time 1772473780.65 total 273.75 last 92.87) - /unlink 26444 (time 1772473790.66 total 283.76 last 94.59) - /unlink 27457 (time 1772473800.66 total 293.77 last 101.26) - /unlink 28681 (time 1772473810.67 total 303.78 last 122.30) - /unlink 29830 (time 1772473820.67 total 313.78 last 114.87) - /unlink 30895 (time 1772473830.67 total 323.78 last 106.50) - /unlink 32081 (time 1772473840.68 total 333.79 last 118.53) - /unlink 33256 (time 1772473850.68 total 343.79 last 117.47) - /unlink 34382 (time 1772473860.69 total 353.79 last 112.53) - /unlink 35594 (time 1772473870.70 total 363.81 last 121.03) - /unlink 36795 (time 1772473880.71 total 373.81 last 120.03) - /unlink 37851 (time 1772473890.72 total 383.83 last 105.44) - /unlink 38891 (time 1772473900.72 total 393.83 last 103.99) - /unlink 40000 (time 1772473910.42 total 403.53 last 114.32) - /unlink 41166 (time 1772473920.43 total 413.54 last 116.54) - /unlink 42367 (time 1772473930.43 total 423.54 last 120.09) - /unlink 43391 (time 1772473940.43 total 433.54 last 102.38) - /unlink 44522 (time 1772473950.43 total 443.54 last 113.06) - /unlink 45605 (time 1772473960.44 total 453.55 last 108.26) - /unlink 46738 (time 1772473970.45 total 463.56 last 113.17) - /unlink 47931 (time 1772473980.46 total 473.57 last 119.16) - /unlink 49086 (time 1772473990.47 total 483.57 last 115.43) - /unlink 50000 (time 1772473999.12 total 492.23 last 105.61) - /unlink 51155 (time 1772474009.12 total 502.23 last 115.49) - /unlink 52396 (time 1772474019.13 total 512.23 last 124.04) - /unlink 53501 (time 1772474029.13 total 522.24 last 110.43) - /unlink 54537 (time 1772474039.13 total 532.24 last 103.58) - /unlink 55624 (time 1772474049.14 total 542.25 last 108.59) - /unlink 56630 (time 1772474059.15 total 552.26 last 100.53) - /unlink 57868 (time 1772474069.17 total 562.28 last 123.51) - /unlink 58898 (time 1772474079.18 total 572.28 last 102.97) - /unlink 60000 (time 1772474089.04 total 582.15 last 111.73) - /unlink 61035 (time 1772474099.04 total 592.15 last 103.48) - /unlink 62024 (time 1772474109.04 total 602.15 last 98.87) - /unlink 63093 (time 1772474119.05 total 612.15 last 106.88) - /unlink 64186 (time 1772474129.05 total 622.16 last 109.26) total: 64768 /unlink in 628.06 seconds: 103.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1165 (time 1772474170.30 total 10.00 last 116.49) - open/close 2227 (time 1772474180.31 total 20.01 last 106.09) - open/close 3374 (time 1772474190.32 total 30.02 last 114.65) - open/close 4532 (time 1772474200.33 total 40.02 last 115.71) - open/close 5729 (time 1772474210.33 total 50.03 last 119.64) - open/close 6948 (time 1772474220.33 total 60.03 last 121.85) - open/close 8110 (time 1772474230.34 total 70.04 last 116.14) - open/close 9160 (time 1772474240.35 total 80.05 last 104.91) - open/close 10000 (time 1772474247.91 total 87.60 last 111.15) - open/close 11133 (time 1772474257.91 total 97.61 last 113.25) - open/close 12264 (time 1772474267.92 total 107.61 last 113.03) - open/close 13420 (time 1772474277.93 total 117.62 last 115.49) - open/close 14373 (time 1772474287.93 total 127.62 last 95.29) - open/close 15442 (time 1772474297.93 total 137.63 last 106.83) - open/close 16683 (time 1772474307.94 total 147.64 last 124.03) - open/close 17828 (time 1772474317.95 total 157.65 last 114.38) - open/close 19056 (time 1772474327.96 total 167.65 last 122.71) - open/close 20000 (time 1772474336.23 total 175.93 last 114.14) - open/close 21083 (time 1772474346.23 total 185.93 last 108.26) - open/close 22166 (time 1772474356.24 total 195.94 last 108.21) - open/close 23247 (time 1772474366.24 total 205.94 last 108.09) - open/close 24368 (time 1772474376.25 total 215.95 last 112.00) - open/close 25367 (time 1772474386.26 total 225.96 last 99.78) - open/close 26488 (time 1772474396.27 total 235.96 last 112.03) - open/close 27638 (time 1772474406.27 total 245.97 last 114.99) - open/close 28760 (time 1772474416.27 total 255.97 last 112.15) - open/close 29964 (time 1772474426.27 total 265.97 last 120.38) - open/close 31186 (time 1772474436.27 total 275.97 last 122.19) - open/close 32383 (time 1772474446.28 total 285.97 last 119.68) - open/close 33611 (time 1772474456.28 total 295.98 last 122.74) - open/close 34756 (time 1772474466.29 total 305.99 last 114.38) - open/close 35984 (time 1772474476.30 total 316.00 last 122.69) - open/close 37159 (time 1772474486.31 total 326.01 last 117.34) - open/close 38263 (time 1772474496.32 total 336.02 last 110.35) - open/close 39458 (time 1772474506.32 total 346.02 last 119.50) - open/close 40000 (time 1772474510.95 total 350.64 last 117.09) - open/close 41303 (time 1772474520.95 total 360.65 last 130.28) - open/close 42484 (time 1772474530.96 total 370.65 last 118.01) - open/close 43718 (time 1772474540.96 total 380.66 last 123.37) - open/close 45413 (time 1772474550.96 total 390.66 last 169.47) - open/close 46815 (time 1772474560.96 total 400.66 last 140.16) - open/close 48617 (time 1772474570.97 total 410.67 last 180.07) - open/close 50000 (time 1772474579.47 total 419.17 last 162.70) - open/close 51466 (time 1772474589.47 total 429.17 last 146.60) - open/close 52846 (time 1772474599.48 total 439.17 last 137.92) - open/close 54221 (time 1772474609.48 total 449.18 last 137.44) - open/close 55407 (time 1772474619.49 total 459.19 last 118.51) - open/close 56658 (time 1772474629.50 total 469.19 last 125.01) - open/close 58180 (time 1772474639.50 total 479.19 last 152.20) - open/close 59883 (time 1772474649.50 total 489.19 last 170.27) - open/close 61108 (time 1772474659.50 total 499.20 last 122.49) - open/close 62369 (time 1772474669.50 total 509.20 last 126.07) - open/close 63558 (time 1772474679.50 total 519.20 last 118.90) total: 64768 open/close in 526.88 seconds: 122.93 ops/second - /unlink 2883 (time 1772474699.46 total 10.00 last 288.26) - /unlink 5614 (time 1772474709.46 total 20.00 last 273.08) - /unlink 8125 (time 1772474719.46 total 30.00 last 251.04) - /unlink 10000 (time 1772474726.93 total 37.48 last 250.94) - /unlink 13150 (time 1772474736.93 total 47.48 last 315.00) - /unlink 16342 (time 1772474746.94 total 57.48 last 319.13) - /unlink 18821 (time 1772474756.94 total 67.48 last 247.84) - /unlink 20000 (time 1772474761.81 total 72.35 last 242.08) - /unlink 22256 (time 1772474771.81 total 82.35 last 225.60) - /unlink 24456 (time 1772474781.81 total 92.35 last 219.96) - /unlink 27081 (time 1772474791.81 total 102.36 last 262.41) - /unlink 29401 (time 1772474801.82 total 112.36 last 231.91) - /unlink 30000 (time 1772474804.37 total 114.91 last 234.95) - /unlink 32320 (time 1772474814.37 total 124.91 last 231.94) - /unlink 34744 (time 1772474824.38 total 134.92 last 242.23) - /unlink 36838 (time 1772474834.38 total 144.93 last 209.24) - /unlink 39098 (time 1772474844.39 total 154.93 last 225.94) - /unlink 40000 (time 1772474848.44 total 158.98 last 222.67) - /unlink 42563 (time 1772474858.44 total 168.98 last 256.24) - /unlink 44909 (time 1772474868.44 total 178.99 last 234.52) - /unlink 47306 (time 1772474878.44 total 188.99 last 239.68) - /unlink 49806 (time 1772474888.45 total 198.99 last 249.93) - /unlink 52268 (time 1772474898.45 total 208.99 last 246.15) - /unlink 54726 (time 1772474908.45 total 218.99 last 245.78) - /unlink 57419 (time 1772474918.45 total 229.00 last 269.16) - /unlink 60000 (time 1772474928.37 total 238.92 last 260.19) - /unlink 62742 (time 1772474938.38 total 248.92 last 274.09) total: 64768 /unlink in 256.92 seconds: 252.09 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1324 (time 1772474977.54 total 10.00 last 132.34) - open/close 2968 (time 1772474987.54 total 20.01 last 164.28) - open/close 4459 (time 1772474997.55 total 30.02 last 148.95) - open/close 5829 (time 1772475007.56 total 40.03 last 136.94) - open/close 7056 (time 1772475017.56 total 50.03 last 122.70) - open/close 8402 (time 1772475027.57 total 60.04 last 134.41) - open/close 9847 (time 1772475037.58 total 70.04 last 144.43) - open/close 11522 (time 1772475047.59 total 80.05 last 167.36) - open/close 12948 (time 1772475057.59 total 90.05 last 142.59) - open/close 14468 (time 1772475067.59 total 100.06 last 151.94) - open/close 16179 (time 1772475077.59 total 110.06 last 171.06) - open/close 17777 (time 1772475087.60 total 120.07 last 159.67) - open/close 19399 (time 1772475097.60 total 130.07 last 162.19) - open/close 20000 (time 1772475102.16 total 134.62 last 131.93) - open/close 21220 (time 1772475112.16 total 144.63 last 121.93) - open/close 22781 (time 1772475122.17 total 154.63 last 156.05) - open/close 24400 (time 1772475132.17 total 164.63 last 161.90) - open/close 25868 (time 1772475142.17 total 174.64 last 146.70) - open/close 27242 (time 1772475152.18 total 184.64 last 137.36) - open/close 28904 (time 1772475162.18 total 194.65 last 166.10) - open/close 30000 (time 1772475170.04 total 202.51 last 139.41) - open/close 31473 (time 1772475180.04 total 212.51 last 147.29) - open/close 33270 (time 1772475190.05 total 222.52 last 179.65) - open/close 35198 (time 1772475200.05 total 232.52 last 192.80) - open/close 36917 (time 1772475210.05 total 242.52 last 171.85) - open/close 39322 (time 1772475220.09 total 252.56 last 239.56) - open/close 40000 (time 1772475223.10 total 255.57 last 225.03) - open/close 41751 (time 1772475233.11 total 265.57 last 175.05) - open/close 43812 (time 1772475243.11 total 275.58 last 206.05) - open/close 45853 (time 1772475253.11 total 285.58 last 204.02) - open/close 47847 (time 1772475263.12 total 295.58 last 199.31) - open/close 50000 (time 1772475271.91 total 304.37 last 244.94) - open/close 53161 (time 1772475281.91 total 314.38 last 316.01) - open/close 55737 (time 1772475291.96 total 324.43 last 256.37) - open/close 58002 (time 1772475302.04 total 334.51 last 224.68) - open/close 60000 (time 1772475311.76 total 344.23 last 205.48) - open/close 61963 (time 1772475321.76 total 354.23 last 196.25) - open/close 63789 (time 1772475331.76 total 364.23 last 182.59) total: 64768 open/close in 369.74 seconds: 175.17 ops/second - /unlink 2784 (time 1772475349.15 total 10.00 last 278.39) - /unlink 5644 (time 1772475359.15 total 20.00 last 285.98) - /unlink 9041 (time 1772475369.16 total 30.01 last 339.51) - /unlink 10000 (time 1772475372.67 total 33.52 last 273.16) - /unlink 12880 (time 1772475382.67 total 43.52 last 287.97) - /unlink 15833 (time 1772475392.67 total 53.52 last 295.23) - /unlink 18587 (time 1772475402.67 total 63.52 last 275.35) - /unlink 20000 (time 1772475407.44 total 68.29 last 296.44) - /unlink 22845 (time 1772475417.45 total 78.30 last 284.26) - /unlink 25734 (time 1772475427.45 total 88.30 last 288.85) - /unlink 29440 (time 1772475437.45 total 98.30 last 370.57) - /unlink 33580 (time 1772475447.45 total 108.30 last 413.98) - /unlink 38179 (time 1772475457.45 total 118.30 last 459.83) - /unlink 40000 (time 1772475462.27 total 123.11 last 378.49) - /unlink 43596 (time 1772475472.27 total 133.12 last 359.31) - /unlink 46043 (time 1772475482.28 total 143.12 last 244.65) - /unlink 49591 (time 1772475492.28 total 153.13 last 354.67) - /unlink 52606 (time 1772475502.28 total 163.13 last 301.40) - /unlink 56002 (time 1772475512.28 total 173.13 last 339.59) - /unlink 59048 (time 1772475522.29 total 183.13 last 304.52) - /unlink 60000 (time 1772475525.38 total 186.23 last 307.21) - /unlink 62629 (time 1772475535.39 total 196.23 last 262.86) total: 64768 /unlink in 203.03 seconds: 319.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1987 (time 1772475574.33 total 10.00 last 198.66) - open/close 3898 (time 1772475584.33 total 20.00 last 191.05) - open/close 6082 (time 1772475594.34 total 30.01 last 218.31) - open/close 8047 (time 1772475604.34 total 40.01 last 196.49) - open/close 10000 (time 1772475613.08 total 48.75 last 223.43) - open/close 12563 (time 1772475623.08 total 58.75 last 256.22) - open/close 14717 (time 1772475633.08 total 68.75 last 215.38) - open/close 16904 (time 1772475643.08 total 78.75 last 218.69) - open/close 19788 (time 1772475653.08 total 88.76 last 288.37) - open/close 22885 (time 1772475663.08 total 98.76 last 309.68) - open/close 25677 (time 1772475673.09 total 108.76 last 279.13) - open/close 28406 (time 1772475683.09 total 118.76 last 272.84) - open/close 30000 (time 1772475689.64 total 125.31 last 243.40) - open/close 32637 (time 1772475699.64 total 135.31 last 263.65) - open/close 34991 (time 1772475709.64 total 145.31 last 235.35) - open/close 37146 (time 1772475719.64 total 155.31 last 215.49) - open/close 39539 (time 1772475729.65 total 165.32 last 239.15) - open/close 40000 (time 1772475731.73 total 167.40 last 221.40) - open/close 42242 (time 1772475741.74 total 177.41 last 224.12) - open/close 44976 (time 1772475751.74 total 187.41 last 273.39) - open/close 48002 (time 1772475761.74 total 197.41 last 302.54) - open/close 50000 (time 1772475769.76 total 205.43 last 248.99) - open/close 52441 (time 1772475779.76 total 215.44 last 244.04) - open/close 55171 (time 1772475789.76 total 225.44 last 273.00) - open/close 57965 (time 1772475799.77 total 235.44 last 279.24) - open/close 60000 (time 1772475805.62 total 241.30 last 347.63) - open/close 63387 (time 1772475815.62 total 251.30 last 338.69) total: 64768 open/close in 257.09 seconds: 251.93 ops/second - /unlink 5001 (time 1772475833.07 total 10.00 last 500.04) - /unlink 9969 (time 1772475843.08 total 20.00 last 496.75) - /unlink 16707 (time 1772475853.08 total 30.00 last 673.73) - /unlink 20000 (time 1772475857.41 total 34.34 last 759.77) - /unlink 27066 (time 1772475867.41 total 44.34 last 706.54) - /unlink 30000 (time 1772475871.13 total 48.06 last 788.63) - /unlink 35877 (time 1772475881.13 total 58.06 last 587.62) - /unlink 40000 (time 1772475886.95 total 63.88 last 708.43) - /unlink 47145 (time 1772475896.95 total 73.88 last 714.49) - /unlink 50000 (time 1772475901.42 total 78.34 last 639.49) - /unlink 56326 (time 1772475911.42 total 88.34 last 632.59) - /unlink 60000 (time 1772475917.17 total 94.10 last 638.32) total: 64768 /unlink in 103.48 seconds: 625.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg446-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg446-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server stop mds service on oleg446-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server unloading modules via unload_modules_local on: 'oleg446-server' oleg446-server: oleg446-server.virtnet: executing unload_modules_local oleg446-server: modules unloaded. pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2 pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2