== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:27:02 (1761316022) Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:-f) oleg136-server: oleg136-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: detected 4 online CPUs by sysfs oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg136-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg136-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock oleg136-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 738 (time 1761316108.57 total 10.00 last 73.78) - open/close 1470 (time 1761316118.57 total 20.00 last 73.19) - open/close 2182 (time 1761316128.57 total 30.01 last 71.16) - open/close 2882 (time 1761316138.58 total 40.01 last 69.98) - open/close 3625 (time 1761316148.58 total 50.02 last 74.26) - open/close 4343 (time 1761316158.59 total 60.03 last 71.70) - open/close 5117 (time 1761316168.60 total 70.03 last 77.40) - open/close 5880 (time 1761316178.60 total 80.04 last 76.23) - open/close 6609 (time 1761316188.61 total 90.04 last 72.89) - open/close 7334 (time 1761316198.62 total 100.05 last 72.43) - open/close 8075 (time 1761316208.64 total 110.08 last 73.90) - open/close 8829 (time 1761316218.65 total 120.08 last 75.37) - open/close 9705 (time 1761316228.65 total 130.08 last 87.60) - open/close 10000 (time 1761316231.83 total 133.26 last 92.78) - open/close 10839 (time 1761316241.83 total 143.27 last 83.84) - open/close 11765 (time 1761316251.84 total 153.27 last 92.57) - open/close 12629 (time 1761316261.84 total 163.28 last 86.35) - open/close 13366 (time 1761316271.85 total 173.29 last 73.65) - open/close 14142 (time 1761316281.86 total 183.30 last 77.53) - open/close 14946 (time 1761316291.87 total 193.30 last 80.33) - open/close 16141 (time 1761316301.87 total 203.31 last 119.47) - open/close 16969 (time 1761316311.89 total 213.32 last 82.65) - open/close 17839 (time 1761316321.89 total 223.33 last 86.94) - open/close 18837 (time 1761316331.90 total 233.34 last 99.73) - open/close 19767 (time 1761316341.90 total 243.34 last 92.99) - open/close 20000 (time 1761316344.46 total 245.89 last 91.23) - open/close 20925 (time 1761316354.46 total 255.90 last 92.42) - open/close 21798 (time 1761316364.48 total 265.91 last 87.19) - open/close 22626 (time 1761316374.49 total 275.92 last 82.73) - open/close 23451 (time 1761316384.49 total 285.93 last 82.47) - open/close 24295 (time 1761316394.51 total 295.94 last 84.26) - open/close 25143 (time 1761316404.52 total 305.96 last 84.68) - open/close 25947 (time 1761316414.52 total 315.96 last 80.38) - open/close 26717 (time 1761316424.54 total 325.97 last 76.90) - open/close 27479 (time 1761316434.54 total 335.98 last 76.18) - open/close 28196 (time 1761316444.55 total 345.98 last 71.66) - open/close 28974 (time 1761316454.55 total 355.99 last 77.76) - open/close 29654 (time 1761316464.56 total 365.99 last 67.98) - open/close 30000 (time 1761316469.34 total 370.77 last 72.39) - open/close 30750 (time 1761316479.34 total 380.78 last 74.96) - open/close 31368 (time 1761316489.35 total 390.79 last 61.74) - open/close 32032 (time 1761316499.36 total 400.79 last 66.36) - open/close 32736 (time 1761316509.37 total 410.80 last 70.33) - open/close 33434 (time 1761316519.38 total 420.82 last 69.72) - open/close 34132 (time 1761316529.38 total 430.82 last 69.79) - open/close 34877 (time 1761316539.38 total 440.82 last 74.49) - open/close 35655 (time 1761316549.39 total 450.83 last 77.74) - open/close 36394 (time 1761316559.40 total 460.84 last 73.83) - open/close 37157 (time 1761316569.41 total 470.84 last 76.24) - open/close 37915 (time 1761316579.41 total 480.84 last 75.79) - open/close 38723 (time 1761316589.41 total 490.85 last 80.79) - open/close 39441 (time 1761316599.41 total 500.85 last 71.80) - open/close 40000 (time 1761316607.37 total 508.80 last 70.25) - open/close 40825 (time 1761316617.37 total 518.81 last 82.43) - open/close 41621 (time 1761316627.38 total 528.82 last 79.54) - open/close 42359 (time 1761316637.40 total 538.83 last 73.69) - open/close 43046 (time 1761316647.40 total 548.84 last 68.68) - open/close 43792 (time 1761316657.41 total 558.85 last 74.49) - open/close 44520 (time 1761316667.42 total 568.86 last 72.74) - open/close 45390 (time 1761316677.42 total 578.86 last 86.99) - open/close 46214 (time 1761316687.43 total 588.86 last 82.37) - open/close 46997 (time 1761316697.43 total 598.87 last 78.26) - open/close 47770 (time 1761316707.43 total 608.87 last 77.29) - open/close 48702 (time 1761316717.44 total 618.87 last 93.18) - open/close 49743 (time 1761316727.44 total 628.88 last 104.07) - open/close 50000 (time 1761316730.66 total 632.10 last 79.73) - open/close 50955 (time 1761316740.70 total 642.14 last 95.11) - open/close 51684 (time 1761316750.71 total 652.15 last 72.82) - open/close 52347 (time 1761316760.72 total 662.16 last 66.26) - open/close 53028 (time 1761316770.73 total 672.17 last 68.04) - open/close 53778 (time 1761316780.73 total 682.17 last 75.00) - open/close 54548 (time 1761316790.73 total 692.17 last 76.97) - open/close 55427 (time 1761316800.74 total 702.18 last 87.80) - open/close 56241 (time 1761316810.75 total 712.19 last 81.33) - open/close 57097 (time 1761316820.76 total 722.19 last 85.56) - open/close 57894 (time 1761316830.76 total 732.20 last 79.66) - open/close 58672 (time 1761316840.76 total 742.20 last 77.79) - open/close 59427 (time 1761316850.77 total 752.21 last 75.45) - open/close 60000 (time 1761316859.27 total 760.71 last 67.41) - open/close 60719 (time 1761316869.28 total 770.71 last 71.87) - open/close 61390 (time 1761316879.28 total 780.72 last 67.08) - open/close 62018 (time 1761316889.29 total 790.73 last 62.72) - open/close 62685 (time 1761316899.30 total 800.73 last 66.66) - open/close 63370 (time 1761316909.30 total 810.74 last 68.46) - open/close 64109 (time 1761316919.31 total 820.75 last 73.82) total: 64768 open/close in 828.93 seconds: 78.13 ops/second - /unlink 1398 (time 1761316940.45 total 10.00 last 139.79) - /unlink 2635 (time 1761316950.45 total 20.00 last 123.66) - /unlink 3749 (time 1761316960.46 total 30.01 last 111.33) - /unlink 5016 (time 1761316970.46 total 40.02 last 126.62) - /unlink 6304 (time 1761316980.48 total 50.04 last 128.53) - /unlink 7583 (time 1761316990.48 total 60.04 last 127.88) - /unlink 8986 (time 1761317000.49 total 70.04 last 140.26) - /unlink 10000 (time 1761317007.71 total 77.26 last 140.46) - /unlink 11300 (time 1761317017.71 total 87.26 last 129.94) - /unlink 12614 (time 1761317027.71 total 97.27 last 131.35) - /unlink 13724 (time 1761317037.73 total 107.29 last 110.80) - /unlink 14947 (time 1761317047.74 total 117.29 last 122.27) - /unlink 16047 (time 1761317057.74 total 127.30 last 109.91) - /unlink 17129 (time 1761317067.75 total 137.30 last 108.16) - /unlink 18277 (time 1761317077.75 total 147.30 last 114.78) - /unlink 19388 (time 1761317087.75 total 157.31 last 111.07) - /unlink 20000 (time 1761317092.87 total 162.43 last 119.51) - /unlink 21131 (time 1761317102.88 total 172.44 last 112.99) - /unlink 22333 (time 1761317112.88 total 182.44 last 120.17) - /unlink 23473 (time 1761317122.89 total 192.44 last 113.98) - /unlink 24751 (time 1761317132.89 total 202.45 last 127.73) - /unlink 26132 (time 1761317142.89 total 212.45 last 138.09) - /unlink 27338 (time 1761317152.90 total 222.45 last 120.54) - /unlink 28381 (time 1761317162.90 total 232.46 last 104.22) - /unlink 29579 (time 1761317172.91 total 242.46 last 119.77) - /unlink 30000 (time 1761317176.23 total 245.78 last 126.88) - /unlink 31263 (time 1761317186.24 total 255.79 last 126.11) - /unlink 32410 (time 1761317196.25 total 265.81 last 114.54) - /unlink 33494 (time 1761317206.26 total 275.81 last 108.36) - /unlink 34580 (time 1761317216.26 total 285.82 last 108.55) - /unlink 35659 (time 1761317226.27 total 295.82 last 107.83) - /unlink 36760 (time 1761317236.27 total 305.83 last 110.07) - /unlink 37835 (time 1761317246.27 total 315.83 last 107.50) - /unlink 38957 (time 1761317256.28 total 325.83 last 112.15) - /unlink 40000 (time 1761317265.15 total 334.71 last 117.50) - /unlink 41153 (time 1761317275.16 total 344.71 last 115.24) - /unlink 42329 (time 1761317285.16 total 354.71 last 117.58) - /unlink 43625 (time 1761317295.16 total 364.72 last 129.56) - /unlink 44937 (time 1761317305.17 total 374.72 last 131.12) - /unlink 46187 (time 1761317315.17 total 384.72 last 124.99) - /unlink 47349 (time 1761317325.18 total 394.73 last 116.08) - /unlink 48511 (time 1761317335.19 total 404.74 last 116.13) - /unlink 49739 (time 1761317345.19 total 414.74 last 122.78) - /unlink 51078 (time 1761317355.19 total 424.75 last 133.81) - /unlink 52411 (time 1761317365.20 total 434.75 last 133.26) - /unlink 53666 (time 1761317375.20 total 444.75 last 125.45) - /unlink 54879 (time 1761317385.21 total 454.76 last 121.22) - /unlink 56131 (time 1761317395.22 total 464.77 last 125.03) - /unlink 57424 (time 1761317405.22 total 474.78 last 129.27) - /unlink 58673 (time 1761317415.23 total 484.78 last 124.82) - /unlink 59753 (time 1761317425.24 total 494.79 last 107.91) - /unlink 60000 (time 1761317427.54 total 497.10 last 107.16) - /unlink 61233 (time 1761317437.55 total 507.11 last 123.19) - /unlink 62514 (time 1761317447.55 total 517.11 last 128.10) - /unlink 63741 (time 1761317457.56 total 527.11 last 122.62) total: 64768 /unlink in 535.50 seconds: 120.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 668 (time 1761317498.35 total 10.01 last 66.75) - open/close 1343 (time 1761317508.35 total 20.01 last 67.47) - open/close 2056 (time 1761317518.35 total 30.01 last 71.29) - open/close 2760 (time 1761317528.38 total 40.04 last 70.21) - open/close 3430 (time 1761317538.38 total 50.04 last 66.99) - open/close 4089 (time 1761317548.39 total 60.05 last 65.85) - open/close 4810 (time 1761317558.39 total 70.05 last 72.08) - open/close 5594 (time 1761317568.39 total 80.05 last 78.39) - open/close 6311 (time 1761317578.40 total 90.06 last 71.66) - open/close 7057 (time 1761317588.40 total 100.06 last 74.58) - open/close 7839 (time 1761317598.41 total 110.07 last 78.10) - open/close 8644 (time 1761317608.41 total 120.07 last 80.49) - open/close 9422 (time 1761317618.42 total 130.08 last 77.78) - open/close 10000 (time 1761317625.71 total 137.37 last 79.23) - open/close 10820 (time 1761317635.71 total 147.37 last 81.99) - open/close 11812 (time 1761317645.71 total 157.38 last 99.17) - open/close 12628 (time 1761317655.73 total 167.39 last 81.50) - open/close 13339 (time 1761317665.73 total 177.39 last 71.08) - open/close 14117 (time 1761317675.74 total 187.40 last 77.74) - open/close 14923 (time 1761317685.74 total 197.40 last 80.60) - open/close 15617 (time 1761317695.74 total 207.41 last 69.36) - open/close 16345 (time 1761317705.75 total 217.41 last 72.78) - open/close 17050 (time 1761317715.75 total 227.41 last 70.47) - open/close 17728 (time 1761317725.76 total 237.43 last 67.71) - open/close 18433 (time 1761317735.77 total 247.43 last 70.47) - open/close 19261 (time 1761317745.78 total 257.44 last 82.71) - open/close 20000 (time 1761317754.82 total 266.48 last 81.73) - open/close 20860 (time 1761317764.82 total 276.48 last 85.99) - open/close 21615 (time 1761317774.82 total 286.49 last 75.48) - open/close 22317 (time 1761317784.83 total 296.49 last 70.15) - open/close 23034 (time 1761317794.84 total 306.50 last 71.64) - open/close 23699 (time 1761317804.84 total 316.51 last 66.47) - open/close 24379 (time 1761317814.85 total 326.51 last 67.99) - open/close 25209 (time 1761317824.85 total 336.51 last 82.95) - open/close 26044 (time 1761317834.85 total 346.52 last 83.47) - open/close 27027 (time 1761317844.86 total 356.53 last 98.21) - open/close 27866 (time 1761317854.87 total 366.53 last 83.88) - open/close 28656 (time 1761317864.88 total 376.54 last 78.89) - open/close 29493 (time 1761317874.88 total 386.54 last 83.70) - open/close 30000 (time 1761317880.25 total 391.91 last 94.49) - open/close 30743 (time 1761317890.25 total 401.91 last 74.29) - open/close 31433 (time 1761317900.27 total 411.93 last 68.88) - open/close 32160 (time 1761317910.27 total 421.93 last 72.70) - open/close 32913 (time 1761317920.27 total 431.93 last 75.25) - open/close 33588 (time 1761317930.28 total 441.94 last 67.47) - open/close 34408 (time 1761317940.29 total 451.95 last 81.93) - open/close 35204 (time 1761317950.29 total 461.95 last 79.60) - open/close 35964 (time 1761317960.29 total 471.95 last 76.00) - open/close 36707 (time 1761317970.29 total 481.95 last 74.30) - open/close 37465 (time 1761317980.29 total 491.96 last 75.74) - open/close 38203 (time 1761317990.30 total 501.96 last 73.79) - open/close 38875 (time 1761318000.31 total 511.97 last 67.10) - open/close 39582 (time 1761318010.32 total 521.98 last 70.64) - open/close 40000 (time 1761318015.89 total 527.56 last 74.97) - open/close 40751 (time 1761318025.91 total 537.57 last 74.99) - open/close 41512 (time 1761318035.92 total 547.58 last 76.02) - open/close 42296 (time 1761318045.92 total 557.59 last 78.36) - open/close 43133 (time 1761318055.93 total 567.59 last 83.66) - open/close 43884 (time 1761318065.93 total 577.60 last 75.06) - open/close 44655 (time 1761318075.94 total 587.60 last 77.07) - open/close 45361 (time 1761318085.94 total 597.60 last 70.59) - open/close 46079 (time 1761318095.95 total 607.61 last 71.74) - open/close 46767 (time 1761318105.96 total 617.62 last 68.74) - open/close 47404 (time 1761318115.96 total 627.62 last 63.66) - open/close 48141 (time 1761318125.97 total 637.63 last 73.64) - open/close 48919 (time 1761318135.98 total 647.64 last 77.74) - open/close 49746 (time 1761318145.98 total 657.65 last 82.66) - open/close 50000 (time 1761318148.99 total 660.65 last 84.52) - open/close 50929 (time 1761318158.99 total 670.65 last 92.89) - open/close 51765 (time 1761318169.00 total 680.67 last 83.48) - open/close 52710 (time 1761318179.00 total 690.67 last 94.49) - open/close 53613 (time 1761318189.01 total 700.67 last 90.24) - open/close 54370 (time 1761318199.01 total 710.68 last 75.68) - open/close 55117 (time 1761318209.02 total 720.68 last 74.68) - open/close 55856 (time 1761318219.02 total 730.68 last 73.88) - open/close 56634 (time 1761318229.03 total 740.69 last 77.74) - open/close 57299 (time 1761318239.03 total 750.70 last 66.45) - open/close 57999 (time 1761318249.08 total 760.74 last 69.70) - open/close 58716 (time 1761318259.09 total 770.75 last 71.62) - open/close 59445 (time 1761318269.11 total 780.77 last 72.77) - open/close 60000 (time 1761318276.77 total 788.44 last 72.38) - open/close 60755 (time 1761318286.78 total 798.44 last 75.48) - open/close 61506 (time 1761318296.79 total 808.45 last 75.01) - open/close 62291 (time 1761318306.80 total 818.46 last 78.42) - open/close 63091 (time 1761318316.80 total 828.46 last 79.99) - open/close 63971 (time 1761318326.81 total 838.47 last 87.92) total: 64768 open/close in 847.60 seconds: 76.41 ops/second - /unlink 1332 (time 1761318348.81 total 10.00 last 133.17) - /unlink 2659 (time 1761318358.82 total 20.01 last 132.64) - /unlink 3987 (time 1761318368.82 total 30.01 last 132.74) - /unlink 5381 (time 1761318378.82 total 40.01 last 139.38) - /unlink 6629 (time 1761318388.82 total 50.01 last 124.79) - /unlink 7929 (time 1761318398.83 total 60.02 last 129.96) - /unlink 9226 (time 1761318408.85 total 70.04 last 129.41) - /unlink 10000 (time 1761318415.63 total 76.83 last 114.07) - /unlink 11248 (time 1761318425.64 total 86.83 last 124.73) - /unlink 12803 (time 1761318435.65 total 96.85 last 155.28) - /unlink 14370 (time 1761318445.66 total 106.85 last 156.69) - /unlink 15915 (time 1761318455.66 total 116.85 last 154.48) - /unlink 17253 (time 1761318465.66 total 126.85 last 133.77) - /unlink 18792 (time 1761318475.66 total 136.85 last 153.86) - /unlink 20000 (time 1761318483.27 total 144.46 last 158.88) - /unlink 21617 (time 1761318493.27 total 154.46 last 161.68) - /unlink 23364 (time 1761318503.27 total 164.46 last 174.66) - /unlink 24944 (time 1761318513.27 total 174.46 last 157.95) - /unlink 26259 (time 1761318523.28 total 184.47 last 131.45) - /unlink 27602 (time 1761318533.28 total 194.47 last 134.25) - /unlink 28884 (time 1761318543.28 total 204.47 last 128.19) - /unlink 30000 (time 1761318552.33 total 213.52 last 123.34) - /unlink 31096 (time 1761318562.33 total 223.52 last 109.55) - /unlink 32375 (time 1761318572.34 total 233.53 last 127.84) - /unlink 33757 (time 1761318582.34 total 243.53 last 138.16) - /unlink 35066 (time 1761318592.34 total 253.53 last 130.85) - /unlink 36351 (time 1761318602.40 total 263.59 last 127.84) - /unlink 37573 (time 1761318612.40 total 273.59 last 122.16) - /unlink 38528 (time 1761318622.42 total 283.61 last 95.31) - /unlink 39529 (time 1761318632.43 total 293.62 last 99.95) - /unlink 40000 (time 1761318636.76 total 297.95 last 108.94) - /unlink 41130 (time 1761318646.76 total 307.95 last 113.00) - /unlink 42219 (time 1761318656.76 total 317.95 last 108.88) - /unlink 43327 (time 1761318666.77 total 327.96 last 110.72) - /unlink 44450 (time 1761318676.77 total 337.96 last 112.29) - /unlink 45561 (time 1761318686.77 total 347.97 last 111.02) - /unlink 46632 (time 1761318696.78 total 357.97 last 107.02) - /unlink 47748 (time 1761318706.79 total 367.98 last 111.50) - /unlink 48838 (time 1761318716.79 total 377.98 last 108.96) - /unlink 50000 (time 1761318726.67 total 387.86 last 117.68) - /unlink 51161 (time 1761318736.67 total 397.86 last 116.07) - /unlink 52377 (time 1761318746.68 total 407.87 last 121.44) - /unlink 53543 (time 1761318756.69 total 417.88 last 116.48) - /unlink 54680 (time 1761318766.71 total 427.90 last 113.56) - /unlink 55989 (time 1761318776.71 total 437.90 last 130.85) - /unlink 57278 (time 1761318786.71 total 447.91 last 128.85) - /unlink 58495 (time 1761318796.72 total 457.91 last 121.62) - /unlink 59655 (time 1761318806.72 total 467.91 last 115.97) - /unlink 60000 (time 1761318810.38 total 471.57 last 94.35) - /unlink 60979 (time 1761318820.38 total 481.57 last 97.88) - /unlink 62094 (time 1761318830.39 total 491.58 last 111.39) - /unlink 63211 (time 1761318840.40 total 501.59 last 111.64) - /unlink 64445 (time 1761318850.40 total 511.59 last 123.40) total: 64768 /unlink in 514.49 seconds: 125.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 726 (time 1761318885.77 total 10.01 last 72.49) - open/close 1407 (time 1761318895.78 total 20.02 last 68.08) - open/close 2227 (time 1761318905.78 total 30.03 last 81.94) - open/close 2997 (time 1761318915.79 total 40.03 last 76.97) - open/close 3727 (time 1761318925.80 total 50.04 last 72.93) - open/close 4444 (time 1761318935.81 total 60.05 last 71.60) - open/close 5244 (time 1761318945.81 total 70.06 last 79.96) - open/close 6092 (time 1761318955.82 total 80.06 last 84.76) - open/close 6863 (time 1761318965.84 total 90.08 last 76.95) - open/close 7576 (time 1761318975.84 total 100.09 last 71.28) - open/close 8320 (time 1761318985.85 total 110.09 last 74.38) - open/close 9037 (time 1761318995.85 total 120.09 last 71.65) - open/close 9819 (time 1761319005.86 total 130.10 last 78.17) - open/close 10000 (time 1761319008.41 total 132.65 last 70.96) - open/close 10781 (time 1761319018.41 total 142.65 last 78.07) - open/close 11674 (time 1761319028.42 total 152.66 last 89.24) - open/close 12553 (time 1761319038.43 total 162.67 last 87.80) - open/close 13468 (time 1761319048.43 total 172.67 last 91.49) - open/close 14331 (time 1761319058.44 total 182.68 last 86.24) - open/close 15113 (time 1761319068.44 total 192.68 last 78.15) - open/close 15902 (time 1761319078.45 total 202.69 last 78.83) - open/close 16728 (time 1761319088.45 total 212.70 last 82.56) - open/close 17658 (time 1761319098.46 total 222.70 last 92.97) - open/close 18613 (time 1761319108.46 total 232.70 last 95.50) - open/close 19539 (time 1761319118.46 total 242.70 last 92.58) - open/close 20000 (time 1761319122.86 total 247.10 last 104.83) - open/close 21028 (time 1761319132.86 total 257.11 last 102.74) - open/close 21919 (time 1761319142.87 total 267.11 last 89.07) - open/close 22704 (time 1761319152.87 total 277.12 last 78.46) - open/close 23520 (time 1761319162.88 total 287.12 last 81.57) - open/close 24437 (time 1761319172.88 total 297.13 last 91.62) - open/close 25411 (time 1761319182.89 total 307.13 last 97.33) - open/close 26361 (time 1761319192.89 total 317.14 last 94.98) - open/close 27533 (time 1761319202.90 total 327.14 last 117.16) - open/close 28494 (time 1761319212.91 total 337.15 last 96.02) - open/close 29281 (time 1761319222.91 total 347.15 last 78.69) - open/close 30000 (time 1761319231.49 total 355.73 last 83.79) - open/close 30898 (time 1761319241.50 total 365.74 last 89.72) - open/close 31797 (time 1761319251.50 total 375.74 last 89.89) - open/close 32702 (time 1761319261.50 total 385.74 last 90.49) - open/close 33631 (time 1761319271.52 total 395.76 last 92.75) - open/close 34539 (time 1761319281.52 total 405.77 last 90.71) - open/close 35457 (time 1761319291.54 total 415.78 last 91.66) - open/close 36701 (time 1761319301.55 total 425.79 last 124.27) - open/close 37829 (time 1761319311.55 total 435.80 last 112.78) - open/close 38799 (time 1761319321.56 total 445.81 last 96.90) - open/close 39699 (time 1761319331.56 total 455.81 last 89.99) - open/close 40000 (time 1761319335.06 total 459.30 last 86.15) - open/close 41393 (time 1761319345.06 total 469.30 last 139.27) - open/close 42659 (time 1761319355.06 total 479.31 last 126.55) - open/close 43840 (time 1761319365.07 total 489.31 last 118.03) - open/close 44742 (time 1761319375.08 total 499.33 last 90.07) - open/close 45640 (time 1761319385.10 total 509.34 last 89.71) - open/close 46646 (time 1761319395.10 total 519.35 last 100.50) - open/close 47892 (time 1761319405.11 total 529.35 last 124.57) - open/close 48860 (time 1761319415.11 total 539.35 last 96.79) - open/close 49851 (time 1761319425.15 total 549.39 last 98.70) - open/close 51013 (time 1761319435.15 total 559.39 last 116.18) - open/close 52143 (time 1761319445.16 total 569.40 last 112.94) - open/close 53257 (time 1761319455.16 total 579.40 last 111.38) - open/close 54690 (time 1761319465.16 total 589.41 last 143.22) - open/close 55884 (time 1761319475.17 total 599.41 last 119.34) - open/close 57200 (time 1761319485.17 total 609.42 last 131.54) - open/close 58441 (time 1761319495.18 total 619.42 last 124.06) - open/close 59493 (time 1761319505.18 total 629.42 last 105.17) - open/close 60000 (time 1761319510.26 total 634.51 last 99.72) - open/close 61101 (time 1761319520.27 total 644.51 last 110.04) - open/close 62165 (time 1761319530.27 total 654.51 last 106.36) - open/close 63290 (time 1761319540.27 total 664.52 last 112.47) - open/close 64572 (time 1761319550.28 total 674.52 last 128.11) total: 64768 open/close in 675.84 seconds: 95.83 ops/second - /unlink 2033 (time 1761319563.55 total 10.00 last 203.22) - /unlink 3779 (time 1761319573.55 total 20.01 last 174.56) - /unlink 5646 (time 1761319583.55 total 30.01 last 186.62) - /unlink 7400 (time 1761319593.55 total 40.01 last 175.36) - /unlink 8400 (time 1761319603.55 total 50.01 last 100.00) - /unlink 9338 (time 1761319613.55 total 60.01 last 93.80) - /unlink 10000 (time 1761319622.03 total 68.49 last 78.12) - /unlink 10959 (time 1761319632.04 total 78.50 last 95.74) - /unlink 11754 (time 1761319642.05 total 88.51 last 79.46) - /unlink 12869 (time 1761319652.05 total 98.51 last 111.45) - /unlink 14099 (time 1761319662.06 total 108.52 last 122.87) - /unlink 15403 (time 1761319672.07 total 118.53 last 130.33) - /unlink 16617 (time 1761319682.08 total 128.53 last 121.30) - /unlink 17893 (time 1761319692.08 total 138.54 last 127.57) - /unlink 19242 (time 1761319702.08 total 148.54 last 134.88) - /unlink 20000 (time 1761319707.29 total 153.75 last 145.38) - /unlink 21342 (time 1761319717.30 total 163.75 last 134.19) - /unlink 22651 (time 1761319727.30 total 173.76 last 130.79) - /unlink 23936 (time 1761319737.31 total 183.77 last 128.46) - /unlink 25239 (time 1761319747.31 total 193.77 last 130.30) - /unlink 26474 (time 1761319757.31 total 203.77 last 123.46) - /unlink 27866 (time 1761319767.32 total 213.78 last 139.10) - /unlink 29196 (time 1761319777.32 total 223.78 last 132.93) - /unlink 30000 (time 1761319783.96 total 230.42 last 121.11) - /unlink 31377 (time 1761319793.97 total 240.43 last 137.62) - /unlink 32668 (time 1761319803.97 total 250.43 last 129.01) - /unlink 34133 (time 1761319813.99 total 260.45 last 146.30) - /unlink 35346 (time 1761319823.99 total 270.45 last 121.29) - /unlink 36574 (time 1761319833.99 total 280.45 last 122.77) - /unlink 37844 (time 1761319843.99 total 290.45 last 126.97) - /unlink 39122 (time 1761319854.00 total 300.46 last 127.68) - /unlink 40000 (time 1761319860.80 total 307.26 last 129.17) - /unlink 41411 (time 1761319870.81 total 317.27 last 140.95) - /unlink 42695 (time 1761319880.82 total 327.27 last 128.35) - /unlink 44021 (time 1761319890.82 total 337.28 last 132.56) - /unlink 45253 (time 1761319900.83 total 347.29 last 123.09) - /unlink 46464 (time 1761319910.83 total 357.29 last 121.04) - /unlink 47657 (time 1761319920.84 total 367.30 last 119.22) - /unlink 48923 (time 1761319930.84 total 377.30 last 126.56) - /unlink 50000 (time 1761319939.41 total 385.86 last 125.75) - /unlink 51196 (time 1761319949.41 total 395.87 last 119.58) - /unlink 52481 (time 1761319959.41 total 405.87 last 128.46) - /unlink 53744 (time 1761319969.41 total 415.87 last 126.29) - /unlink 55099 (time 1761319979.41 total 425.87 last 135.45) - /unlink 56397 (time 1761319989.42 total 435.88 last 129.76) - /unlink 57760 (time 1761319999.42 total 445.88 last 136.28) - /unlink 59136 (time 1761320009.42 total 455.88 last 137.59) - /unlink 60000 (time 1761320016.00 total 462.46 last 131.33) - /unlink 61289 (time 1761320026.00 total 472.46 last 128.86) - /unlink 62673 (time 1761320036.01 total 482.47 last 138.30) - /unlink 64052 (time 1761320046.01 total 492.47 last 137.82) total: 64768 /unlink in 497.75 seconds: 130.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 812 (time 1761320083.46 total 10.00 last 81.16) - open/close 1648 (time 1761320093.46 total 20.01 last 83.57) - open/close 2530 (time 1761320103.47 total 30.01 last 88.19) - open/close 3347 (time 1761320113.48 total 40.02 last 81.58) - open/close 4208 (time 1761320123.49 total 50.04 last 85.98) - open/close 5038 (time 1761320133.51 total 60.05 last 82.91) - open/close 5817 (time 1761320143.51 total 70.05 last 77.89) - open/close 6610 (time 1761320153.51 total 80.05 last 79.29) - open/close 7395 (time 1761320163.52 total 90.06 last 78.43) - open/close 8185 (time 1761320173.53 total 100.07 last 78.90) - open/close 8971 (time 1761320183.53 total 110.08 last 78.58) - open/close 9765 (time 1761320193.54 total 120.08 last 79.33) - open/close 10000 (time 1761320196.39 total 122.93 last 82.50) - open/close 10835 (time 1761320206.39 total 132.93 last 83.49) - open/close 11651 (time 1761320216.39 total 142.94 last 81.59) - open/close 12444 (time 1761320226.41 total 152.95 last 79.17) - open/close 13226 (time 1761320236.41 total 162.95 last 78.18) - open/close 14034 (time 1761320246.41 total 172.96 last 80.77) - open/close 14870 (time 1761320256.42 total 182.97 last 83.53) - open/close 15673 (time 1761320266.43 total 192.97 last 80.27) - open/close 16544 (time 1761320276.43 total 202.97 last 87.08) - open/close 17440 (time 1761320286.43 total 212.98 last 89.56) - open/close 18333 (time 1761320296.45 total 222.99 last 89.16) - open/close 19236 (time 1761320306.46 total 233.00 last 90.20) - open/close 20000 (time 1761320314.88 total 241.42 last 90.73) - open/close 20927 (time 1761320324.89 total 251.43 last 92.60) - open/close 21768 (time 1761320334.89 total 261.44 last 84.10) - open/close 22636 (time 1761320344.90 total 271.45 last 86.69) - open/close 23454 (time 1761320354.91 total 281.45 last 81.75) - open/close 24265 (time 1761320364.91 total 291.46 last 81.08) - open/close 25232 (time 1761320374.92 total 301.46 last 96.65) - open/close 26092 (time 1761320384.93 total 311.47 last 85.93) - open/close 26931 (time 1761320394.93 total 321.47 last 83.87) - open/close 27793 (time 1761320404.94 total 331.48 last 86.13) - open/close 28718 (time 1761320414.94 total 341.48 last 92.48) - open/close 29577 (time 1761320424.95 total 351.49 last 85.81) - open/close 30000 (time 1761320429.82 total 356.36 last 86.85) - open/close 30793 (time 1761320439.83 total 366.37 last 79.22) - open/close 31620 (time 1761320449.84 total 376.38 last 82.63) - open/close 32471 (time 1761320459.85 total 386.39 last 85.05) - open/close 33380 (time 1761320469.85 total 396.40 last 90.84) - open/close 34211 (time 1761320479.87 total 406.41 last 82.95) - open/close 35080 (time 1761320489.87 total 416.42 last 86.88) - open/close 35998 (time 1761320499.88 total 426.42 last 91.74) - open/close 36943 (time 1761320509.89 total 436.43 last 94.43) - open/close 37874 (time 1761320519.89 total 446.44 last 93.02) - open/close 38772 (time 1761320529.90 total 456.45 last 89.72) - open/close 39610 (time 1761320539.90 total 466.45 last 83.79) - open/close 40000 (time 1761320544.92 total 471.46 last 77.75) - open/close 40865 (time 1761320554.93 total 481.48 last 86.41) - open/close 41805 (time 1761320564.95 total 491.49 last 93.86) - open/close 42759 (time 1761320574.95 total 501.50 last 95.33) - open/close 43777 (time 1761320584.96 total 511.50 last 101.74) - open/close 44767 (time 1761320594.96 total 521.51 last 98.98) - open/close 45766 (time 1761320604.96 total 531.51 last 99.90) - open/close 46717 (time 1761320614.97 total 541.51 last 95.04) - open/close 47818 (time 1761320624.97 total 551.52 last 110.03) - open/close 48730 (time 1761320634.98 total 561.53 last 91.13) - open/close 49656 (time 1761320644.99 total 571.53 last 92.55) - open/close 50000 (time 1761320648.45 total 575.00 last 99.25) - open/close 51067 (time 1761320658.46 total 585.00 last 106.66) - open/close 52226 (time 1761320668.47 total 595.01 last 115.80) - open/close 53315 (time 1761320678.47 total 605.02 last 108.82) - open/close 54496 (time 1761320688.47 total 615.02 last 118.10) - open/close 55524 (time 1761320698.48 total 625.02 last 102.76) - open/close 56636 (time 1761320708.48 total 635.02 last 111.17) - open/close 57580 (time 1761320718.48 total 645.02 last 94.39) - open/close 58633 (time 1761320728.49 total 655.04 last 105.18) - open/close 59683 (time 1761320738.50 total 665.04 last 104.92) - open/close 60000 (time 1761320741.23 total 667.78 last 116.03) - open/close 61175 (time 1761320751.23 total 677.78 last 117.47) - open/close 62207 (time 1761320761.24 total 687.79 last 103.12) - open/close 63489 (time 1761320771.25 total 697.79 last 128.12) - open/close 64742 (time 1761320781.25 total 707.80 last 125.24) total: 64768 open/close in 708.00 seconds: 91.48 ops/second - /unlink 2117 (time 1761320793.60 total 10.00 last 211.68) - /unlink 4121 (time 1761320803.60 total 20.01 last 200.32) - /unlink 6009 (time 1761320813.60 total 30.01 last 188.77) - /unlink 8281 (time 1761320823.60 total 40.01 last 227.14) - /unlink 10000 (time 1761320832.04 total 48.45 last 203.78) - /unlink 12078 (time 1761320842.05 total 58.45 last 207.69) - /unlink 14313 (time 1761320852.05 total 68.45 last 223.41) - /unlink 16342 (time 1761320862.06 total 78.46 last 202.76) - /unlink 18290 (time 1761320872.06 total 88.47 last 194.72) - /unlink 20000 (time 1761320879.86 total 96.27 last 219.19) - /unlink 22411 (time 1761320889.86 total 106.27 last 241.03) - /unlink 24981 (time 1761320899.87 total 116.27 last 256.96) - /unlink 27358 (time 1761320909.87 total 126.27 last 237.64) - /unlink 29785 (time 1761320919.87 total 136.28 last 242.66) - /unlink 32510 (time 1761320929.87 total 146.28 last 272.48) - /unlink 35486 (time 1761320939.87 total 156.28 last 297.55) - /unlink 38246 (time 1761320949.87 total 166.28 last 276.00) - /unlink 40000 (time 1761320955.86 total 172.26 last 293.01) - /unlink 42829 (time 1761320965.86 total 182.27 last 282.85) - /unlink 44831 (time 1761320975.86 total 192.27 last 200.16) - /unlink 47174 (time 1761320985.86 total 202.27 last 234.25) - /unlink 50000 (time 1761320995.82 total 212.23 last 283.77) - /unlink 52512 (time 1761321005.83 total 222.23 last 251.16) - /unlink 55734 (time 1761321015.83 total 232.24 last 322.03) - /unlink 58273 (time 1761321025.83 total 242.24 last 253.88) - /unlink 60000 (time 1761321031.95 total 248.35 last 282.42) - /unlink 62898 (time 1761321041.95 total 258.36 last 289.70) total: 64768 /unlink in 265.24 seconds: 244.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1591 (time 1761321078.79 total 10.01 last 158.99) - open/close 2871 (time 1761321088.80 total 20.01 last 127.94) - open/close 4634 (time 1761321098.80 total 30.02 last 176.22) - open/close 6112 (time 1761321108.81 total 40.02 last 147.75) - open/close 7770 (time 1761321118.81 total 50.03 last 165.68) - open/close 9240 (time 1761321128.82 total 60.03 last 146.93) - open/close 10000 (time 1761321133.89 total 65.11 last 149.78) - open/close 11582 (time 1761321143.90 total 75.11 last 158.11) - open/close 13432 (time 1761321153.90 total 85.12 last 184.89) - open/close 15274 (time 1761321163.90 total 95.12 last 184.18) - open/close 17107 (time 1761321173.91 total 105.13 last 183.15) - open/close 18550 (time 1761321183.92 total 115.13 last 144.24) - open/close 19805 (time 1761321193.92 total 125.13 last 125.46) - open/close 21484 (time 1761321203.92 total 135.14 last 167.84) - open/close 22809 (time 1761321213.92 total 145.14 last 132.49) - open/close 24481 (time 1761321223.93 total 155.14 last 167.18) - open/close 26037 (time 1761321233.93 total 165.15 last 155.52) - open/close 27508 (time 1761321243.93 total 175.15 last 147.04) - open/close 29367 (time 1761321253.94 total 185.15 last 185.84) - open/close 30000 (time 1761321257.20 total 188.41 last 194.20) - open/close 31819 (time 1761321267.21 total 198.43 last 181.63) - open/close 33471 (time 1761321277.22 total 208.43 last 165.15) - open/close 35590 (time 1761321287.22 total 218.43 last 211.86) - open/close 37430 (time 1761321297.22 total 228.43 last 183.98) - open/close 39801 (time 1761321307.22 total 238.44 last 237.03) - open/close 42350 (time 1761321317.22 total 248.44 last 254.85) - open/close 44654 (time 1761321327.23 total 258.44 last 230.35) - open/close 46587 (time 1761321337.23 total 268.44 last 193.30) - open/close 48282 (time 1761321347.23 total 278.44 last 169.49) - open/close 50000 (time 1761321354.94 total 286.16 last 222.62) - open/close 52094 (time 1761321364.95 total 296.16 last 209.37) - open/close 54142 (time 1761321374.95 total 306.17 last 204.65) - open/close 56319 (time 1761321384.96 total 316.17 last 217.63) - open/close 58758 (time 1761321394.96 total 326.17 last 243.83) - open/close 60000 (time 1761321400.72 total 331.93 last 215.71) - open/close 62255 (time 1761321410.72 total 341.93 last 225.45) - open/close 64531 (time 1761321420.72 total 351.93 last 227.59) total: 64768 open/close in 353.08 seconds: 183.44 ops/second - /unlink 4361 (time 1761321433.47 total 10.00 last 436.04) - /unlink 8573 (time 1761321443.47 total 20.00 last 421.18) - /unlink 10000 (time 1761321447.06 total 23.60 last 396.69) - /unlink 13076 (time 1761321457.07 total 33.60 last 307.55) - /unlink 16519 (time 1761321467.07 total 43.60 last 344.28) - /unlink 20000 (time 1761321476.63 total 53.16 last 364.14) - /unlink 23581 (time 1761321486.63 total 63.16 last 358.04) - /unlink 26854 (time 1761321496.63 total 73.17 last 327.19) - /unlink 29790 (time 1761321506.63 total 83.17 last 293.59) - /unlink 32999 (time 1761321516.64 total 93.17 last 320.75) - /unlink 36089 (time 1761321526.64 total 103.17 last 308.96) - /unlink 39374 (time 1761321536.64 total 113.18 last 328.39) - /unlink 40000 (time 1761321538.85 total 115.38 last 283.71) - /unlink 43604 (time 1761321548.85 total 125.38 last 360.39) - /unlink 47234 (time 1761321558.85 total 135.38 last 362.90) - /unlink 50000 (time 1761321564.94 total 141.47 last 454.44) - /unlink 54839 (time 1761321574.94 total 151.47 last 483.78) - /unlink 59095 (time 1761321584.94 total 161.47 last 425.57) - /unlink 63790 (time 1761321594.94 total 171.48 last 469.37) total: 64768 /unlink in 174.00 seconds: 372.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg136-server: LNET ready to unload oleg136-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2