== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:31:42 (1767321102) Stopping clients: oleg409-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg409-client.virtnet /mnt/lustre2 (opts:-f) oleg409-server: oleg409-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing load_modules_local oleg409-server: Loading modules from /home/green/git/lustre-release/lustre oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg409-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy oleg409-server: detected 4 online CPUs by sysfs oleg409-server: MODOPTS_LIBCFS= oleg409-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg409-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg409-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 701 (time 1767321292.95 total 10.00 last 70.08) - open/close 1570 (time 1767321302.96 total 20.01 last 86.83) - open/close 2446 (time 1767321312.97 total 30.02 last 87.54) - open/close 3408 (time 1767321322.97 total 40.02 last 96.18) - open/close 4226 (time 1767321332.97 total 50.02 last 81.79) - open/close 5178 (time 1767321342.97 total 60.02 last 95.17) - open/close 6090 (time 1767321352.99 total 70.04 last 91.00) - open/close 6908 (time 1767321363.02 total 80.07 last 81.59) - open/close 7837 (time 1767321373.02 total 90.07 last 92.89) - open/close 8605 (time 1767321383.03 total 100.08 last 76.74) - open/close 9391 (time 1767321393.04 total 110.09 last 78.49) - open/close 10000 (time 1767321402.75 total 119.80 last 62.71) - open/close 10687 (time 1767321412.77 total 129.82 last 68.58) - open/close 11548 (time 1767321422.77 total 139.82 last 86.08) - open/close 12387 (time 1767321432.78 total 149.83 last 83.83) - open/close 13049 (time 1767321442.80 total 159.86 last 66.06) - open/close 13933 (time 1767321452.81 total 169.86 last 88.31) - open/close 14892 (time 1767321462.82 total 179.87 last 95.86) - open/close 15773 (time 1767321472.82 total 189.87 last 88.08) - open/close 16710 (time 1767321482.82 total 199.87 last 93.70) - open/close 17566 (time 1767321492.82 total 209.87 last 85.59) - open/close 18294 (time 1767321502.83 total 219.88 last 72.74) - open/close 19092 (time 1767321512.83 total 229.88 last 79.78) - open/close 20000 (time 1767321522.77 total 239.82 last 91.34) - open/close 20830 (time 1767321532.78 total 249.83 last 82.97) - open/close 21602 (time 1767321542.78 total 259.83 last 77.19) - open/close 22457 (time 1767321552.78 total 269.84 last 85.43) - open/close 23400 (time 1767321562.79 total 279.85 last 94.21) - open/close 24422 (time 1767321572.80 total 289.85 last 102.11) - open/close 25350 (time 1767321582.81 total 299.86 last 92.75) - open/close 26225 (time 1767321592.82 total 309.87 last 87.39) - open/close 27099 (time 1767321602.83 total 319.88 last 87.32) - open/close 27967 (time 1767321612.83 total 329.89 last 86.76) - open/close 28698 (time 1767321622.84 total 339.89 last 73.09) - open/close 29302 (time 1767321632.84 total 349.89 last 60.38) - open/close 29988 (time 1767321642.84 total 359.89 last 68.59) - open/close 30777 (time 1767321652.85 total 369.90 last 78.85) - open/close 31802 (time 1767321662.85 total 379.90 last 102.50) - open/close 32763 (time 1767321672.85 total 389.91 last 96.03) - open/close 33720 (time 1767321682.86 total 399.91 last 95.62) - open/close 34582 (time 1767321692.87 total 409.92 last 86.16) - open/close 35451 (time 1767321702.87 total 419.93 last 86.84) - open/close 36392 (time 1767321712.88 total 429.93 last 94.03) - open/close 37270 (time 1767321722.89 total 439.94 last 87.74) - open/close 38106 (time 1767321732.89 total 449.94 last 83.60) - open/close 38691 (time 1767321742.89 total 459.95 last 58.47) - open/close 39539 (time 1767321752.90 total 469.95 last 84.73) - open/close 40000 (time 1767321758.41 total 475.46 last 83.66) - open/close 40892 (time 1767321768.43 total 485.48 last 89.07) - open/close 41726 (time 1767321778.44 total 495.49 last 83.32) - open/close 42639 (time 1767321788.44 total 505.49 last 91.27) - open/close 43499 (time 1767321798.46 total 515.51 last 85.84) - open/close 44377 (time 1767321808.51 total 525.56 last 87.36) - open/close 45271 (time 1767321818.52 total 535.57 last 89.34) - open/close 46130 (time 1767321828.53 total 545.58 last 85.81) - open/close 47019 (time 1767321838.54 total 555.59 last 88.81) - open/close 47921 (time 1767321848.54 total 565.59 last 90.19) - open/close 48908 (time 1767321858.54 total 575.59 last 98.68) - open/close 49835 (time 1767321868.54 total 585.59 last 92.70) - open/close 50745 (time 1767321878.55 total 595.60 last 90.89) - open/close 51648 (time 1767321888.55 total 605.61 last 90.29) - open/close 52495 (time 1767321898.57 total 615.62 last 84.58) - open/close 53464 (time 1767321908.58 total 625.63 last 96.83) - open/close 54378 (time 1767321918.58 total 635.63 last 91.34) - open/close 55146 (time 1767321928.62 total 645.67 last 76.52) - open/close 55996 (time 1767321938.63 total 655.68 last 84.94) - open/close 56719 (time 1767321948.63 total 665.68 last 72.26) - open/close 57499 (time 1767321958.64 total 675.69 last 77.92) - open/close 58286 (time 1767321968.66 total 685.71 last 78.59) - open/close 58921 (time 1767321978.66 total 695.72 last 63.45) - open/close 59694 (time 1767321988.69 total 705.74 last 77.13) - open/close 60000 (time 1767321992.54 total 709.59 last 79.49) - open/close 60906 (time 1767322002.56 total 719.61 last 90.37) - open/close 61625 (time 1767322012.57 total 729.62 last 71.85) - open/close 62359 (time 1767322022.57 total 739.62 last 73.37) - open/close 63050 (time 1767322032.58 total 749.63 last 69.07) - open/close 63800 (time 1767322042.59 total 759.64 last 74.93) - open/close 64558 (time 1767322052.59 total 769.64 last 75.79) total: 64768 open/close in 772.15 seconds: 83.88 ops/second - /unlink 935 (time 1767322069.49 total 10.00 last 93.48) - /unlink 1721 (time 1767322079.50 total 20.01 last 78.57) - /unlink 2527 (time 1767322089.50 total 30.01 last 80.55) - /unlink 3423 (time 1767322099.51 total 40.02 last 89.56) - /unlink 4309 (time 1767322109.51 total 50.02 last 88.58) - /unlink 5147 (time 1767322119.52 total 60.03 last 83.74) - /unlink 5872 (time 1767322129.53 total 70.04 last 72.41) - /unlink 6613 (time 1767322139.55 total 80.06 last 73.96) - /unlink 7526 (time 1767322149.55 total 90.06 last 91.29) - /unlink 8401 (time 1767322159.55 total 100.06 last 87.48) - /unlink 9218 (time 1767322169.56 total 110.06 last 81.68) - /unlink 10000 (time 1767322178.65 total 119.16 last 85.95) - /unlink 10845 (time 1767322188.66 total 129.17 last 84.43) - /unlink 11737 (time 1767322198.67 total 139.18 last 89.10) - /unlink 12567 (time 1767322208.68 total 149.19 last 82.96) - /unlink 13305 (time 1767322218.69 total 159.20 last 73.68) - /unlink 13846 (time 1767322228.73 total 169.23 last 53.93) - /unlink 14365 (time 1767322238.74 total 179.24 last 51.84) - /unlink 14962 (time 1767322248.76 total 189.27 last 59.56) - /unlink 15750 (time 1767322258.77 total 199.28 last 78.74) - /unlink 16604 (time 1767322268.77 total 209.28 last 85.37) - /unlink 17380 (time 1767322278.78 total 219.28 last 77.56) - /unlink 18102 (time 1767322288.78 total 229.29 last 72.19) - /unlink 19022 (time 1767322298.78 total 239.29 last 91.96) - /unlink 19909 (time 1767322308.79 total 249.29 last 88.68) - /unlink 20789 (time 1767322318.79 total 259.30 last 87.93) - /unlink 21536 (time 1767322328.80 total 269.31 last 74.63) - /unlink 22296 (time 1767322338.81 total 279.32 last 75.91) - /unlink 23055 (time 1767322348.82 total 289.33 last 75.87) - /unlink 23831 (time 1767322358.82 total 299.33 last 77.56) - /unlink 24588 (time 1767322368.84 total 309.34 last 75.61) - /unlink 25406 (time 1767322378.84 total 319.35 last 81.73) - /unlink 26176 (time 1767322388.87 total 329.38 last 76.80) - /unlink 26989 (time 1767322398.89 total 339.40 last 81.10) - /unlink 27757 (time 1767322408.90 total 349.41 last 76.76) - /unlink 28650 (time 1767322418.90 total 359.41 last 89.29) - /unlink 29574 (time 1767322428.90 total 369.41 last 92.37) - /unlink 30000 (time 1767322433.92 total 374.43 last 84.90) - /unlink 30821 (time 1767322443.93 total 384.43 last 82.06) - /unlink 31660 (time 1767322453.93 total 394.44 last 83.85) - /unlink 32466 (time 1767322463.94 total 404.44 last 80.56) - /unlink 33293 (time 1767322473.95 total 414.45 last 82.63) - /unlink 34163 (time 1767322483.95 total 424.46 last 86.92) - /unlink 35084 (time 1767322493.95 total 434.46 last 92.09) - /unlink 35937 (time 1767322503.96 total 444.47 last 85.22) - /unlink 36888 (time 1767322513.98 total 454.48 last 94.99) - /unlink 37648 (time 1767322523.98 total 464.48 last 75.99) - /unlink 38271 (time 1767322533.99 total 474.49 last 62.24) - /unlink 38977 (time 1767322544.00 total 484.51 last 70.51) - /unlink 39752 (time 1767322554.01 total 494.51 last 77.45) - /unlink 40000 (time 1767322556.77 total 497.28 last 89.62) - /unlink 40907 (time 1767322566.78 total 507.29 last 90.66) - /unlink 41740 (time 1767322576.78 total 517.29 last 83.27) - /unlink 42586 (time 1767322586.79 total 527.30 last 84.50) - /unlink 43389 (time 1767322596.79 total 537.30 last 80.30) - /unlink 44337 (time 1767322606.81 total 547.31 last 94.67) - /unlink 45223 (time 1767322616.81 total 557.32 last 88.54) - /unlink 45940 (time 1767322626.81 total 567.32 last 71.70) - /unlink 46803 (time 1767322636.82 total 577.33 last 86.22) - /unlink 47641 (time 1767322646.83 total 587.33 last 83.79) - /unlink 48522 (time 1767322656.83 total 597.34 last 88.07) - /unlink 49484 (time 1767322666.84 total 607.34 last 96.13) - /unlink 50000 (time 1767322673.19 total 613.70 last 81.16) - /unlink 50866 (time 1767322683.20 total 623.71 last 86.52) - /unlink 51765 (time 1767322693.21 total 633.72 last 89.85) - /unlink 52706 (time 1767322703.23 total 643.74 last 93.91) - /unlink 53657 (time 1767322713.23 total 653.74 last 95.05) - /unlink 54528 (time 1767322723.24 total 663.75 last 87.06) - /unlink 55355 (time 1767322733.25 total 673.75 last 82.64) - /unlink 56313 (time 1767322743.26 total 683.76 last 95.71) - /unlink 57270 (time 1767322753.26 total 693.77 last 95.67) - /unlink 58153 (time 1767322763.26 total 703.77 last 88.26) - /unlink 59036 (time 1767322773.27 total 713.77 last 88.28) - /unlink 59895 (time 1767322783.27 total 723.78 last 85.87) - /unlink 60799 (time 1767322793.27 total 733.78 last 90.39) - /unlink 61807 (time 1767322803.28 total 743.78 last 100.75) - /unlink 62730 (time 1767322813.30 total 753.81 last 92.08) - /unlink 63532 (time 1767322823.30 total 763.81 last 80.19) - /unlink 64401 (time 1767322833.31 total 773.82 last 86.83) total: 64768 /unlink in 777.53 seconds: 83.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 901 (time 1767322873.15 total 10.02 last 89.91) - open/close 1816 (time 1767322883.15 total 20.03 last 91.46) - open/close 2753 (time 1767322893.16 total 30.03 last 93.67) - open/close 3749 (time 1767322903.16 total 40.04 last 99.53) - open/close 4818 (time 1767322913.17 total 50.04 last 106.88) - open/close 5947 (time 1767322923.17 total 60.04 last 112.90) - open/close 6922 (time 1767322933.17 total 70.04 last 97.44) - open/close 7895 (time 1767322943.18 total 80.05 last 97.26) - open/close 8946 (time 1767322953.18 total 90.05 last 105.06) - open/close 10000 (time 1767322963.00 total 99.87 last 107.32) - open/close 10974 (time 1767322973.00 total 109.87 last 97.40) - open/close 11961 (time 1767322983.01 total 119.88 last 98.64) - open/close 12856 (time 1767322993.02 total 129.89 last 89.41) - open/close 13772 (time 1767323003.03 total 139.90 last 91.53) - open/close 14723 (time 1767323013.03 total 149.90 last 95.09) - open/close 15689 (time 1767323023.03 total 159.90 last 96.57) - open/close 16526 (time 1767323033.03 total 169.90 last 83.69) - open/close 17464 (time 1767323043.04 total 179.91 last 93.69) - open/close 18371 (time 1767323053.04 total 189.91 last 90.69) - open/close 19292 (time 1767323063.05 total 199.93 last 92.00) - open/close 20000 (time 1767323070.04 total 206.91 last 101.29) - open/close 21043 (time 1767323080.05 total 216.92 last 104.27) - open/close 22015 (time 1767323090.05 total 226.93 last 97.11) - open/close 22860 (time 1767323100.06 total 236.93 last 84.44) - open/close 23771 (time 1767323110.07 total 246.94 last 91.07) - open/close 24630 (time 1767323120.08 total 256.95 last 85.82) - open/close 25341 (time 1767323130.10 total 266.97 last 70.93) - open/close 26165 (time 1767323140.10 total 276.98 last 82.36) - open/close 26966 (time 1767323150.11 total 286.98 last 80.08) - open/close 27899 (time 1767323160.14 total 297.01 last 93.01) - open/close 28660 (time 1767323170.14 total 307.02 last 76.06) - open/close 29637 (time 1767323180.15 total 317.02 last 97.65) - open/close 30000 (time 1767323183.72 total 320.59 last 101.70) - open/close 30964 (time 1767323193.73 total 330.60 last 96.27) - open/close 31956 (time 1767323203.73 total 340.61 last 99.17) - open/close 32738 (time 1767323213.74 total 350.61 last 78.17) - open/close 33556 (time 1767323223.74 total 360.61 last 81.77) - open/close 34477 (time 1767323233.75 total 370.62 last 92.06) - open/close 35355 (time 1767323243.75 total 380.63 last 87.73) - open/close 36198 (time 1767323253.76 total 390.63 last 84.29) - open/close 37129 (time 1767323263.76 total 400.63 last 93.08) - open/close 37997 (time 1767323273.76 total 410.63 last 86.76) - open/close 38968 (time 1767323283.76 total 420.63 last 97.10) - open/close 39909 (time 1767323293.77 total 430.64 last 94.05) - open/close 40952 (time 1767323303.77 total 440.65 last 104.24) - open/close 41962 (time 1767323313.77 total 450.65 last 100.98) - open/close 42845 (time 1767323323.78 total 460.65 last 88.25) - open/close 43807 (time 1767323333.78 total 470.65 last 96.19) - open/close 44924 (time 1767323343.78 total 480.65 last 111.68) - open/close 45938 (time 1767323353.79 total 490.66 last 101.37) - open/close 46938 (time 1767323363.79 total 500.66 last 99.96) - open/close 47877 (time 1767323373.80 total 510.67 last 93.82) - open/close 48828 (time 1767323383.81 total 520.68 last 95.03) - open/close 49765 (time 1767323393.81 total 530.68 last 93.69) - open/close 50000 (time 1767323396.45 total 533.32 last 88.94) - open/close 50954 (time 1767323406.45 total 543.32 last 95.36) - open/close 51903 (time 1767323416.46 total 553.33 last 94.81) - open/close 52719 (time 1767323426.47 total 563.34 last 81.55) - open/close 53468 (time 1767323436.47 total 573.35 last 74.87) - open/close 54341 (time 1767323446.48 total 583.35 last 87.23) - open/close 55352 (time 1767323456.48 total 593.35 last 101.08) - open/close 56335 (time 1767323466.49 total 603.36 last 98.25) - open/close 57355 (time 1767323476.49 total 613.36 last 101.98) - open/close 58328 (time 1767323486.49 total 623.37 last 97.25) - open/close 59194 (time 1767323496.50 total 633.37 last 86.56) - open/close 60000 (time 1767323504.72 total 641.59 last 98.02) - open/close 61139 (time 1767323514.74 total 651.61 last 113.76) - open/close 62052 (time 1767323524.74 total 661.61 last 91.28) - open/close 62958 (time 1767323534.74 total 671.61 last 90.59) - open/close 63897 (time 1767323544.74 total 681.61 last 93.89) total: 64768 open/close in 690.98 seconds: 93.73 ops/second - /unlink 870 (time 1767323567.11 total 10.00 last 86.96) - /unlink 1944 (time 1767323577.12 total 20.01 last 107.37) - /unlink 3052 (time 1767323587.12 total 30.01 last 110.74) - /unlink 3909 (time 1767323597.14 total 40.03 last 85.58) - /unlink 4699 (time 1767323607.14 total 50.03 last 78.99) - /unlink 5813 (time 1767323617.15 total 60.04 last 111.31) - /unlink 6832 (time 1767323627.15 total 70.04 last 101.84) - /unlink 7875 (time 1767323637.15 total 80.04 last 104.29) - /unlink 8761 (time 1767323647.16 total 90.05 last 88.54) - /unlink 9762 (time 1767323657.17 total 100.06 last 100.01) - /unlink 10000 (time 1767323659.51 total 102.40 last 101.76) - /unlink 10951 (time 1767323669.52 total 112.41 last 95.03) - /unlink 11894 (time 1767323679.52 total 122.41 last 94.27) - /unlink 12978 (time 1767323689.53 total 132.42 last 108.29) - /unlink 13992 (time 1767323699.54 total 142.43 last 101.30) - /unlink 15108 (time 1767323709.54 total 152.43 last 111.55) - /unlink 16239 (time 1767323719.55 total 162.44 last 113.03) - /unlink 17438 (time 1767323729.55 total 172.44 last 119.84) - /unlink 18655 (time 1767323739.56 total 182.45 last 121.58) - /unlink 19600 (time 1767323749.56 total 192.46 last 94.49) - /unlink 20000 (time 1767323753.54 total 196.43 last 100.67) - /unlink 21260 (time 1767323763.55 total 206.44 last 125.90) - /unlink 22590 (time 1767323773.56 total 216.45 last 132.89) - /unlink 23874 (time 1767323783.57 total 226.46 last 128.24) - /unlink 25257 (time 1767323793.57 total 236.46 last 138.26) - /unlink 26457 (time 1767323803.57 total 246.46 last 119.98) - /unlink 27708 (time 1767323813.58 total 256.47 last 125.06) - /unlink 28857 (time 1767323823.58 total 266.47 last 114.85) - /unlink 30000 (time 1767323833.16 total 276.05 last 119.29) - /unlink 31185 (time 1767323843.17 total 286.06 last 118.46) - /unlink 32341 (time 1767323853.17 total 296.06 last 115.51) - /unlink 33433 (time 1767323863.18 total 306.07 last 109.13) - /unlink 34509 (time 1767323873.19 total 316.08 last 107.45) - /unlink 35813 (time 1767323883.20 total 326.09 last 130.31) - /unlink 36971 (time 1767323893.20 total 336.09 last 115.76) - /unlink 38140 (time 1767323903.21 total 346.10 last 116.79) - /unlink 39206 (time 1767323913.22 total 356.11 last 106.56) - /unlink 40000 (time 1767323919.81 total 362.70 last 120.38) - /unlink 41260 (time 1767323929.81 total 372.70 last 126.00) - /unlink 42434 (time 1767323939.82 total 382.71 last 117.32) - /unlink 43606 (time 1767323949.82 total 392.71 last 117.15) - /unlink 44802 (time 1767323959.83 total 402.72 last 119.57) - /unlink 45896 (time 1767323969.83 total 412.72 last 109.40) - /unlink 47028 (time 1767323979.83 total 422.72 last 113.17) - /unlink 48074 (time 1767323989.84 total 432.73 last 104.51) - /unlink 49259 (time 1767323999.84 total 442.73 last 118.46) - /unlink 50000 (time 1767324006.69 total 449.58 last 108.16) - /unlink 51121 (time 1767324016.70 total 459.59 last 112.03) - /unlink 52040 (time 1767324026.71 total 469.60 last 91.80) - /unlink 53165 (time 1767324036.71 total 479.60 last 112.45) - /unlink 54460 (time 1767324046.71 total 489.61 last 129.48) - /unlink 56752 (time 1767324056.72 total 499.61 last 229.13) - /unlink 58749 (time 1767324066.72 total 509.61 last 199.65) - /unlink 60000 (time 1767324072.43 total 515.32 last 218.95) - /unlink 62099 (time 1767324082.44 total 525.33 last 209.86) - /unlink 64171 (time 1767324092.44 total 535.33 last 207.20) total: 64768 /unlink in 537.86 seconds: 120.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1274 (time 1767324131.31 total 10.01 last 127.28) - open/close 2579 (time 1767324141.31 total 20.01 last 130.43) - open/close 3911 (time 1767324151.32 total 30.02 last 133.16) - open/close 5148 (time 1767324161.32 total 40.02 last 123.69) - open/close 6294 (time 1767324171.32 total 50.02 last 114.57) - open/close 7526 (time 1767324181.32 total 60.03 last 123.14) - open/close 8653 (time 1767324191.33 total 70.03 last 112.63) - open/close 9869 (time 1767324201.34 total 80.04 last 121.49) - open/close 11128 (time 1767324211.34 total 90.04 last 125.89) - open/close 12375 (time 1767324221.35 total 100.05 last 124.57) - open/close 13554 (time 1767324231.35 total 110.06 last 117.84) - open/close 14759 (time 1767324241.36 total 120.07 last 120.39) - open/close 15913 (time 1767324251.37 total 130.07 last 115.39) - open/close 17066 (time 1767324261.37 total 140.07 last 115.23) - open/close 18306 (time 1767324271.38 total 150.08 last 123.90) - open/close 19527 (time 1767324281.38 total 160.09 last 122.04) - open/close 20000 (time 1767324285.15 total 163.86 last 125.49) - open/close 21138 (time 1767324295.16 total 173.86 last 113.71) - open/close 22367 (time 1767324305.17 total 183.87 last 122.80) - open/close 23639 (time 1767324315.17 total 193.87 last 127.18) - open/close 24947 (time 1767324325.17 total 203.87 last 130.78) - open/close 26233 (time 1767324335.18 total 213.88 last 128.54) - open/close 27423 (time 1767324345.18 total 223.88 last 118.95) - open/close 28577 (time 1767324355.19 total 233.89 last 115.33) - open/close 29895 (time 1767324365.19 total 243.90 last 131.72) - open/close 31283 (time 1767324375.20 total 253.91 last 138.65) - open/close 32647 (time 1767324385.21 total 263.91 last 136.33) - open/close 33939 (time 1767324395.21 total 273.92 last 129.14) - open/close 35273 (time 1767324405.22 total 283.92 last 133.33) - open/close 36584 (time 1767324415.22 total 293.92 last 131.07) - open/close 37999 (time 1767324425.22 total 303.92 last 141.49) - open/close 39597 (time 1767324435.23 total 313.93 last 159.72) - open/close 40000 (time 1767324437.69 total 316.39 last 163.63) - open/close 41709 (time 1767324447.69 total 326.39 last 170.85) - open/close 43108 (time 1767324457.70 total 336.40 last 139.83) - open/close 44958 (time 1767324467.70 total 346.40 last 184.96) - open/close 46849 (time 1767324477.70 total 356.40 last 189.04) - open/close 48404 (time 1767324487.71 total 366.41 last 155.45) - open/close 49841 (time 1767324497.71 total 376.41 last 143.62) - open/close 51209 (time 1767324507.72 total 386.42 last 136.69) - open/close 52907 (time 1767324517.72 total 396.42 last 169.76) - open/close 54503 (time 1767324527.72 total 406.43 last 159.56) - open/close 55818 (time 1767324537.73 total 416.43 last 131.40) - open/close 57460 (time 1767324547.73 total 426.44 last 164.16) - open/close 59261 (time 1767324557.74 total 436.44 last 180.07) - open/close 60000 (time 1767324561.00 total 439.70 last 226.46) - open/close 61753 (time 1767324571.00 total 449.70 last 175.28) - open/close 63942 (time 1767324581.01 total 459.71 last 218.75) total: 64768 open/close in 463.85 seconds: 139.63 ops/second - /unlink 3279 (time 1767324596.92 total 10.00 last 327.81) - /unlink 7432 (time 1767324606.92 total 20.00 last 415.25) - /unlink 10000 (time 1767324612.24 total 25.32 last 483.24) - /unlink 15264 (time 1767324622.24 total 35.32 last 526.29) - /unlink 18815 (time 1767324632.24 total 45.32 last 355.02) - /unlink 20000 (time 1767324635.00 total 48.08 last 430.20) - /unlink 23457 (time 1767324645.00 total 58.08 last 345.66) - /unlink 26889 (time 1767324655.00 total 68.08 last 343.19) - /unlink 30000 (time 1767324663.82 total 76.90 last 352.51) - /unlink 33710 (time 1767324673.83 total 86.91 last 370.84) - /unlink 38065 (time 1767324683.83 total 96.91 last 435.26) - /unlink 40000 (time 1767324688.96 total 102.04 last 377.63) - /unlink 43114 (time 1767324698.96 total 112.04 last 311.37) - /unlink 46799 (time 1767324708.96 total 122.04 last 368.41) - /unlink 50000 (time 1767324718.89 total 131.97 last 322.31) - /unlink 53601 (time 1767324728.89 total 141.97 last 360.05) - /unlink 56857 (time 1767324738.89 total 151.97 last 325.57) - /unlink 60000 (time 1767324745.93 total 159.02 last 446.40) - /unlink 64178 (time 1767324755.94 total 169.02 last 417.74) total: 64768 /unlink in 170.36 seconds: 380.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2025 (time 1767324790.07 total 10.00 last 202.46) - open/close 4357 (time 1767324800.07 total 20.00 last 233.17) - open/close 5943 (time 1767324810.07 total 30.01 last 158.55) - open/close 7480 (time 1767324820.08 total 40.01 last 153.59) - open/close 8830 (time 1767324830.08 total 50.02 last 134.95) - open/close 10000 (time 1767324837.91 total 57.85 last 149.41) - open/close 11812 (time 1767324847.92 total 67.85 last 181.09) - open/close 13651 (time 1767324857.92 total 77.86 last 183.86) - open/close 15771 (time 1767324867.92 total 87.86 last 212.00) - open/close 17651 (time 1767324877.93 total 97.86 last 187.94) - open/close 20000 (time 1767324886.87 total 106.80 last 262.66) - open/close 21789 (time 1767324896.87 total 116.81 last 178.83) - open/close 23431 (time 1767324906.88 total 126.81 last 164.12) - open/close 25296 (time 1767324916.88 total 136.81 last 186.45) - open/close 26768 (time 1767324926.89 total 146.82 last 147.10) - open/close 28234 (time 1767324936.89 total 156.82 last 146.60) - open/close 29677 (time 1767324946.89 total 166.83 last 144.19) - open/close 30000 (time 1767324949.22 total 169.15 last 138.96) - open/close 31342 (time 1767324959.22 total 179.16 last 134.17) - open/close 33111 (time 1767324969.23 total 189.16 last 176.77) - open/close 34574 (time 1767324979.23 total 199.16 last 146.27) - open/close 36109 (time 1767324989.23 total 209.17 last 153.50) - open/close 37635 (time 1767324999.23 total 219.17 last 152.55) - open/close 39282 (time 1767325009.24 total 229.17 last 164.63) - open/close 40000 (time 1767325013.73 total 233.66 last 159.88) - open/close 41432 (time 1767325023.73 total 243.67 last 143.10) - open/close 42925 (time 1767325033.74 total 253.67 last 149.27) - open/close 44437 (time 1767325043.74 total 263.67 last 151.19) - open/close 45918 (time 1767325053.74 total 273.68 last 148.04) - open/close 47988 (time 1767325063.75 total 283.68 last 206.89) - open/close 49634 (time 1767325073.75 total 293.68 last 164.58) - open/close 51847 (time 1767325083.75 total 303.68 last 221.26) - open/close 54533 (time 1767325093.75 total 313.69 last 268.55) - open/close 57065 (time 1767325103.76 total 323.69 last 253.10) - open/close 59371 (time 1767325113.76 total 333.69 last 230.50) - open/close 60000 (time 1767325116.19 total 336.13 last 258.51) - open/close 62906 (time 1767325126.21 total 346.15 last 290.10) total: 64768 open/close in 353.63 seconds: 183.15 ops/second - /unlink 5359 (time 1767325145.16 total 10.00 last 535.84) - /unlink 10000 (time 1767325152.30 total 17.14 last 649.67) - /unlink 15514 (time 1767325162.31 total 27.15 last 551.34) - /unlink 19355 (time 1767325172.31 total 37.15 last 383.83) - /unlink 20000 (time 1767325174.37 total 39.21 last 313.21) - /unlink 23038 (time 1767325184.37 total 49.21 last 303.77) - /unlink 27367 (time 1767325194.38 total 59.22 last 432.65) - /unlink 30000 (time 1767325200.50 total 65.34 last 430.14) - /unlink 34122 (time 1767325210.50 total 75.34 last 412.14) - /unlink 37692 (time 1767325220.50 total 85.34 last 356.95) - /unlink 40000 (time 1767325226.05 total 90.89 last 416.14) - /unlink 43698 (time 1767325236.05 total 100.89 last 369.74) - /unlink 47342 (time 1767325246.05 total 110.89 last 364.37) - /unlink 50000 (time 1767325253.01 total 117.85 last 381.95) - /unlink 53127 (time 1767325263.01 total 127.85 last 312.68) - /unlink 57592 (time 1767325273.01 total 137.85 last 446.38) - /unlink 60000 (time 1767325278.90 total 143.73 last 409.45) - /unlink 63218 (time 1767325288.90 total 153.74 last 321.78) total: 64768 /unlink in 158.23 seconds: 409.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3172 (time 1767325324.00 total 10.00 last 317.15) - open/close 6143 (time 1767325334.00 total 20.00 last 297.06) - open/close 8993 (time 1767325344.00 total 30.01 last 284.93) - open/close 10000 (time 1767325347.26 total 33.26 last 309.48) - open/close 13144 (time 1767325357.26 total 43.26 last 314.35) - open/close 15849 (time 1767325367.26 total 53.26 last 270.42) - open/close 18149 (time 1767325377.26 total 63.26 last 229.99) - open/close 20000 (time 1767325387.14 total 73.14 last 187.43) - open/close 22203 (time 1767325397.14 total 83.14 last 220.23) - open/close 24446 (time 1767325407.15 total 93.15 last 224.21) - open/close 27459 (time 1767325417.15 total 103.16 last 301.06) - open/close 29708 (time 1767325427.16 total 113.16 last 224.86) - open/close 32106 (time 1767325437.16 total 123.16 last 239.71) - open/close 33772 (time 1767325447.16 total 133.16 last 166.57) - open/close 35769 (time 1767325457.17 total 143.17 last 199.59) - open/close 37470 (time 1767325467.17 total 153.17 last 170.08) - open/close 39467 (time 1767325477.17 total 163.17 last 199.66) - open/close 40000 (time 1767325480.17 total 166.18 last 177.41) - open/close 41791 (time 1767325490.18 total 176.18 last 179.03) - open/close 43390 (time 1767325500.18 total 186.18 last 159.89) - open/close 45023 (time 1767325510.18 total 196.18 last 163.29) - open/close 47005 (time 1767325520.18 total 206.18 last 198.17) - open/close 49224 (time 1767325530.18 total 216.18 last 221.88) - open/close 50000 (time 1767325533.76 total 219.76 last 216.74) - open/close 52556 (time 1767325543.76 total 229.77 last 255.57) - open/close 54648 (time 1767325553.76 total 239.77 last 209.19) - open/close 57886 (time 1767325563.77 total 249.77 last 323.78) - open/close 60000 (time 1767325573.11 total 259.11 last 226.29) - open/close 61973 (time 1767325583.11 total 269.11 last 197.24) - open/close 64595 (time 1767325593.11 total 279.11 last 262.18) total: 64768 open/close in 279.70 seconds: 231.56 ops/second - /unlink 3755 (time 1767325605.33 total 10.00 last 375.50) - /unlink 7364 (time 1767325615.33 total 20.00 last 360.80) - /unlink 10000 (time 1767325621.70 total 26.37 last 413.70) - /unlink 14238 (time 1767325631.71 total 36.37 last 423.79) - /unlink 18445 (time 1767325641.71 total 46.38 last 420.66) - /unlink 20000 (time 1767325645.31 total 49.98 last 431.28) - /unlink 25692 (time 1767325655.31 total 59.98 last 569.18) - /unlink 29328 (time 1767325665.31 total 69.98 last 363.51) - /unlink 32939 (time 1767325675.32 total 79.99 last 361.03) - /unlink 36215 (time 1767325685.32 total 89.99 last 327.39) - /unlink 40000 (time 1767325691.73 total 96.40 last 591.03) - /unlink 44773 (time 1767325701.73 total 106.40 last 477.23) - /unlink 50000 (time 1767325711.56 total 116.23 last 531.74) - /unlink 53631 (time 1767325721.56 total 126.23 last 362.96) - /unlink 58017 (time 1767325731.56 total 136.23 last 438.55) - /unlink 60000 (time 1767325736.54 total 141.21 last 398.16) - /unlink 63180 (time 1767325746.55 total 151.22 last 317.85) total: 64768 /unlink in 155.06 seconds: 417.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg409-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2