== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:55:01 (1761317701) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock oleg408-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 681 (time 1761317805.16 total 10.01 last 68.05) - open/close 1357 (time 1761317815.17 total 20.02 last 67.53) - open/close 1975 (time 1761317825.19 total 30.03 last 61.73) - open/close 2659 (time 1761317835.19 total 40.03 last 68.40) - open/close 3363 (time 1761317845.19 total 50.03 last 70.39) - open/close 4057 (time 1761317855.19 total 60.03 last 69.37) - open/close 4747 (time 1761317865.19 total 70.03 last 69.00) - open/close 5459 (time 1761317875.19 total 80.03 last 71.19) - open/close 6135 (time 1761317885.20 total 90.04 last 67.57) - open/close 6816 (time 1761317895.21 total 100.05 last 68.02) - open/close 7495 (time 1761317905.21 total 110.05 last 67.87) - open/close 8164 (time 1761317915.22 total 120.06 last 66.87) - open/close 8816 (time 1761317925.23 total 130.07 last 65.11) - open/close 9533 (time 1761317935.24 total 140.08 last 71.63) - open/close 10000 (time 1761317941.40 total 146.24 last 75.82) - open/close 10766 (time 1761317951.40 total 156.25 last 76.56) - open/close 11501 (time 1761317961.41 total 166.25 last 73.46) - open/close 12164 (time 1761317971.41 total 176.25 last 66.29) - open/close 12822 (time 1761317981.41 total 186.26 last 65.78) - open/close 13486 (time 1761317991.42 total 196.26 last 66.37) - open/close 14230 (time 1761318001.43 total 206.27 last 74.35) - open/close 14880 (time 1761318011.43 total 216.27 last 64.97) - open/close 15518 (time 1761318021.44 total 226.28 last 63.77) - open/close 16129 (time 1761318031.44 total 236.28 last 61.08) - open/close 16786 (time 1761318041.44 total 246.29 last 65.67) - open/close 17432 (time 1761318051.46 total 256.30 last 64.49) - open/close 18119 (time 1761318061.49 total 266.33 last 68.53) - open/close 18711 (time 1761318071.49 total 276.33 last 59.18) - open/close 19321 (time 1761318081.49 total 286.34 last 60.97) - open/close 19987 (time 1761318091.50 total 296.34 last 66.58) - open/close 20625 (time 1761318101.50 total 306.34 last 63.78) - open/close 21272 (time 1761318111.51 total 316.35 last 64.65) - open/close 21937 (time 1761318121.51 total 326.36 last 66.46) - open/close 22613 (time 1761318131.52 total 336.36 last 67.57) - open/close 23270 (time 1761318141.54 total 346.38 last 65.59) - open/close 23895 (time 1761318151.54 total 356.38 last 62.49) - open/close 24570 (time 1761318161.55 total 366.39 last 67.43) - open/close 25296 (time 1761318171.55 total 376.39 last 72.59) - open/close 25953 (time 1761318181.56 total 386.40 last 65.66) - open/close 26593 (time 1761318191.56 total 396.41 last 63.95) - open/close 27252 (time 1761318201.59 total 406.43 last 65.74) - open/close 27877 (time 1761318211.61 total 416.45 last 62.39) - open/close 28541 (time 1761318221.62 total 426.46 last 66.33) - open/close 29287 (time 1761318231.62 total 436.46 last 74.59) - open/close 29973 (time 1761318241.63 total 446.47 last 68.52) - open/close 30598 (time 1761318251.65 total 456.49 last 62.38) - open/close 31225 (time 1761318261.68 total 466.53 last 62.47) - open/close 31839 (time 1761318271.69 total 476.53 last 61.37) - open/close 32467 (time 1761318281.70 total 486.55 last 62.72) - open/close 33055 (time 1761318291.71 total 496.55 last 58.78) - open/close 33680 (time 1761318301.73 total 506.58 last 62.33) - open/close 34255 (time 1761318311.75 total 516.59 last 57.43) - open/close 34865 (time 1761318321.76 total 526.60 last 60.91) - open/close 35487 (time 1761318331.76 total 536.61 last 62.18) - open/close 36157 (time 1761318341.77 total 546.61 last 66.97) - open/close 36846 (time 1761318351.77 total 556.61 last 68.88) - open/close 37577 (time 1761318361.77 total 566.62 last 73.08) - open/close 38244 (time 1761318371.78 total 576.62 last 66.66) - open/close 38902 (time 1761318381.78 total 586.63 last 65.77) - open/close 39552 (time 1761318391.78 total 596.63 last 65.00) - open/close 40000 (time 1761318398.63 total 603.48 last 65.42) - open/close 40603 (time 1761318408.64 total 613.49 last 60.24) - open/close 41237 (time 1761318418.64 total 623.49 last 63.40) - open/close 41830 (time 1761318428.65 total 633.49 last 59.28) - open/close 42455 (time 1761318438.66 total 643.50 last 62.44) - open/close 43158 (time 1761318448.66 total 653.50 last 70.27) - open/close 43843 (time 1761318458.66 total 663.50 last 68.49) - open/close 44541 (time 1761318468.67 total 673.52 last 69.72) - open/close 45256 (time 1761318478.69 total 683.54 last 71.36) - open/close 45967 (time 1761318488.70 total 693.54 last 71.05) - open/close 46683 (time 1761318498.72 total 703.56 last 71.48) - open/close 47337 (time 1761318508.73 total 713.57 last 65.34) - open/close 48039 (time 1761318518.74 total 723.58 last 70.13) - open/close 48729 (time 1761318528.76 total 733.60 last 68.85) - open/close 49352 (time 1761318538.77 total 743.61 last 62.21) - open/close 49905 (time 1761318548.78 total 753.62 last 55.24) - open/close 50511 (time 1761318558.78 total 763.63 last 60.60) - open/close 51170 (time 1761318568.79 total 773.63 last 65.84) - open/close 51842 (time 1761318578.80 total 783.64 last 67.15) - open/close 52526 (time 1761318588.80 total 793.65 last 68.36) - open/close 53139 (time 1761318598.82 total 803.66 last 61.23) - open/close 53780 (time 1761318608.83 total 813.67 last 64.02) - open/close 54350 (time 1761318618.84 total 823.68 last 56.92) - open/close 55007 (time 1761318628.86 total 833.70 last 65.59) - open/close 55662 (time 1761318638.86 total 843.70 last 65.48) - open/close 56316 (time 1761318648.87 total 853.71 last 65.35) - open/close 56954 (time 1761318658.88 total 863.72 last 63.74) - open/close 57589 (time 1761318668.88 total 873.72 last 63.49) - open/close 58270 (time 1761318678.88 total 883.72 last 68.09) - open/close 58961 (time 1761318688.89 total 893.74 last 69.02) - open/close 59618 (time 1761318698.90 total 903.74 last 65.69) - open/close 60000 (time 1761318704.65 total 909.49 last 66.36) - open/close 60515 (time 1761318714.65 total 919.50 last 51.50) - open/close 61132 (time 1761318724.67 total 929.51 last 61.59) - open/close 61762 (time 1761318734.68 total 939.52 last 62.97) - open/close 62359 (time 1761318744.68 total 949.52 last 59.69) - open/close 63028 (time 1761318754.69 total 959.53 last 66.84) - open/close 63672 (time 1761318764.69 total 969.53 last 64.40) - open/close 64319 (time 1761318774.69 total 979.53 last 64.67) total: 64768 open/close in 986.16 seconds: 65.68 ops/second - /unlink 1162 (time 1761318795.00 total 10.01 last 116.14) - /unlink 2313 (time 1761318805.00 total 20.01 last 115.04) - /unlink 3466 (time 1761318815.00 total 30.01 last 115.29) - /unlink 4557 (time 1761318825.01 total 40.02 last 109.01) - /unlink 5681 (time 1761318835.01 total 50.02 last 112.34) - /unlink 6784 (time 1761318845.02 total 60.03 last 110.20) - /unlink 7980 (time 1761318855.03 total 70.04 last 119.55) - /unlink 9148 (time 1761318865.03 total 80.04 last 116.77) - /unlink 10000 (time 1761318871.98 total 86.99 last 122.62) - /unlink 11233 (time 1761318881.98 total 96.99 last 123.27) - /unlink 12478 (time 1761318891.99 total 107.00 last 124.43) - /unlink 13592 (time 1761318901.99 total 117.00 last 111.32) - /unlink 14829 (time 1761318912.01 total 127.02 last 123.56) - /unlink 15982 (time 1761318922.01 total 137.02 last 115.29) - /unlink 17231 (time 1761318932.01 total 147.02 last 124.85) - /unlink 18412 (time 1761318942.02 total 157.03 last 118.00) - /unlink 19567 (time 1761318952.02 total 167.03 last 115.49) - /unlink 20000 (time 1761318955.58 total 170.59 last 121.57) - /unlink 21162 (time 1761318965.59 total 180.60 last 116.15) - /unlink 22272 (time 1761318975.59 total 190.60 last 110.98) - /unlink 23390 (time 1761318985.59 total 200.60 last 111.72) - /unlink 24577 (time 1761318995.60 total 210.61 last 118.65) - /unlink 25791 (time 1761319005.61 total 220.62 last 121.24) - /unlink 26970 (time 1761319015.62 total 230.63 last 117.83) - /unlink 28170 (time 1761319025.62 total 240.63 last 119.94) - /unlink 29375 (time 1761319035.63 total 250.64 last 120.45) - /unlink 30000 (time 1761319041.54 total 256.55 last 105.63) - /unlink 31203 (time 1761319051.55 total 266.56 last 120.23) - /unlink 32441 (time 1761319061.56 total 276.57 last 123.70) - /unlink 33706 (time 1761319071.56 total 286.57 last 126.41) - /unlink 34853 (time 1761319081.57 total 296.58 last 114.69) - /unlink 36029 (time 1761319091.57 total 306.58 last 117.55) - /unlink 37083 (time 1761319101.57 total 316.58 last 105.36) - /unlink 38175 (time 1761319111.58 total 326.59 last 109.11) - /unlink 39282 (time 1761319121.60 total 336.61 last 110.56) - /unlink 40000 (time 1761319127.35 total 342.36 last 124.82) - /unlink 41215 (time 1761319137.35 total 352.36 last 121.47) - /unlink 42361 (time 1761319147.35 total 362.36 last 114.54) - /unlink 43493 (time 1761319157.36 total 372.37 last 113.18) - /unlink 44688 (time 1761319167.37 total 382.38 last 119.37) - /unlink 45855 (time 1761319177.37 total 392.38 last 116.67) - /unlink 47009 (time 1761319187.37 total 402.38 last 115.39) - /unlink 48328 (time 1761319197.38 total 412.39 last 131.78) - /unlink 49621 (time 1761319207.38 total 422.39 last 129.26) - /unlink 50000 (time 1761319210.17 total 425.18 last 135.94) - /unlink 51258 (time 1761319220.17 total 435.18 last 125.78) - /unlink 52693 (time 1761319230.18 total 445.19 last 143.46) - /unlink 53918 (time 1761319240.18 total 455.19 last 122.50) - /unlink 55269 (time 1761319250.18 total 465.19 last 135.04) - /unlink 56819 (time 1761319260.18 total 475.19 last 154.93) - /unlink 58385 (time 1761319270.19 total 485.20 last 156.50) - /unlink 60000 (time 1761319279.25 total 494.26 last 178.29) - /unlink 61521 (time 1761319289.26 total 504.27 last 151.99) - /unlink 62972 (time 1761319299.26 total 514.27 last 145.04) - /unlink 64490 (time 1761319309.26 total 524.27 last 151.77) total: 64768 /unlink in 525.96 seconds: 123.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 872 (time 1761319341.61 total 10.01 last 87.14) - open/close 1713 (time 1761319351.62 total 20.01 last 84.06) - open/close 2578 (time 1761319361.62 total 30.02 last 86.46) - open/close 3441 (time 1761319371.62 total 40.02 last 86.28) - open/close 4250 (time 1761319381.63 total 50.03 last 80.84) - open/close 5157 (time 1761319391.64 total 60.03 last 90.65) - open/close 6091 (time 1761319401.64 total 70.04 last 93.34) - open/close 7052 (time 1761319411.65 total 80.04 last 96.06) - open/close 8138 (time 1761319421.65 total 90.04 last 108.58) - open/close 9146 (time 1761319431.65 total 100.05 last 100.76) - open/close 10000 (time 1761319440.32 total 108.71 last 98.56) - open/close 10947 (time 1761319450.32 total 118.72 last 94.65) - open/close 11890 (time 1761319460.33 total 128.72 last 94.27) - open/close 13000 (time 1761319470.33 total 138.73 last 110.94) - open/close 14283 (time 1761319480.33 total 148.73 last 128.30) - open/close 15579 (time 1761319490.34 total 158.73 last 129.53) - open/close 16817 (time 1761319500.35 total 168.75 last 123.61) - open/close 17978 (time 1761319510.35 total 178.75 last 116.10) - open/close 19108 (time 1761319520.35 total 188.75 last 112.99) - open/close 20000 (time 1761319527.36 total 195.76 last 127.25) - open/close 21276 (time 1761319537.37 total 205.76 last 127.59) - open/close 22599 (time 1761319547.37 total 215.76 last 132.29) - open/close 23880 (time 1761319557.37 total 225.77 last 128.04) - open/close 25000 (time 1761319567.37 total 235.77 last 111.99) - open/close 26144 (time 1761319577.38 total 245.77 last 114.30) - open/close 27305 (time 1761319587.38 total 255.78 last 116.07) - open/close 28055 (time 1761319597.39 total 265.79 last 74.93) - open/close 28734 (time 1761319607.40 total 275.79 last 67.87) - open/close 29416 (time 1761319617.41 total 285.81 last 68.08) - open/close 29958 (time 1761319627.42 total 295.82 last 54.16) - open/close 30539 (time 1761319637.42 total 305.82 last 58.10) - open/close 31092 (time 1761319647.43 total 315.82 last 55.29) - open/close 31793 (time 1761319657.43 total 325.82 last 70.08) - open/close 32523 (time 1761319667.43 total 335.83 last 72.98) - open/close 33288 (time 1761319677.44 total 345.83 last 76.43) - open/close 34054 (time 1761319687.45 total 355.84 last 76.52) - open/close 34660 (time 1761319697.46 total 365.86 last 60.54) - open/close 35287 (time 1761319707.47 total 375.87 last 62.64) - open/close 35924 (time 1761319717.49 total 385.88 last 63.60) - open/close 36651 (time 1761319727.50 total 395.89 last 72.61) - open/close 37329 (time 1761319737.50 total 405.90 last 67.76) - open/close 38026 (time 1761319747.51 total 415.90 last 69.68) - open/close 38764 (time 1761319757.52 total 425.92 last 73.69) - open/close 39417 (time 1761319767.56 total 435.95 last 65.07) - open/close 40000 (time 1761319776.03 total 444.43 last 68.79) - open/close 40736 (time 1761319786.04 total 454.44 last 73.54) - open/close 41454 (time 1761319796.04 total 464.44 last 71.79) - open/close 42204 (time 1761319806.05 total 474.45 last 74.93) - open/close 42860 (time 1761319816.07 total 484.47 last 65.47) - open/close 43558 (time 1761319826.08 total 494.47 last 69.74) - open/close 44305 (time 1761319836.08 total 504.48 last 74.68) - open/close 44997 (time 1761319846.09 total 514.48 last 69.17) - open/close 45626 (time 1761319856.10 total 524.49 last 62.82) - open/close 46318 (time 1761319866.11 total 534.50 last 69.15) - open/close 47061 (time 1761319876.12 total 544.51 last 74.20) - open/close 47835 (time 1761319886.12 total 554.52 last 77.37) - open/close 48632 (time 1761319896.13 total 564.52 last 79.67) - open/close 49330 (time 1761319906.13 total 574.52 last 69.78) - open/close 50000 (time 1761319915.31 total 583.71 last 72.97) - open/close 50796 (time 1761319925.32 total 593.71 last 79.57) - open/close 51444 (time 1761319935.33 total 603.72 last 64.73) - open/close 52140 (time 1761319945.34 total 613.73 last 69.54) - open/close 52874 (time 1761319955.34 total 623.73 last 73.40) - open/close 53646 (time 1761319965.34 total 633.74 last 77.14) - open/close 54427 (time 1761319975.34 total 643.74 last 78.10) - open/close 55219 (time 1761319985.34 total 653.74 last 79.20) - open/close 56072 (time 1761319995.34 total 663.74 last 85.30) - open/close 56859 (time 1761320005.35 total 673.75 last 78.62) - open/close 57564 (time 1761320015.36 total 683.75 last 70.48) - open/close 58274 (time 1761320025.38 total 693.77 last 70.86) - open/close 59018 (time 1761320035.40 total 703.79 last 74.25) - open/close 59857 (time 1761320045.40 total 713.80 last 83.86) - open/close 60770 (time 1761320055.40 total 723.80 last 91.29) - open/close 61589 (time 1761320065.40 total 733.80 last 81.90) - open/close 62408 (time 1761320075.41 total 743.80 last 81.89) - open/close 63180 (time 1761320085.41 total 753.80 last 77.18) - open/close 63929 (time 1761320095.42 total 763.81 last 74.84) - open/close 64707 (time 1761320105.42 total 773.82 last 77.77) total: 64768 open/close in 774.54 seconds: 83.62 ops/second - /unlink 1349 (time 1761320118.74 total 10.01 last 134.78) - /unlink 2675 (time 1761320128.74 total 20.01 last 132.56) - /unlink 4076 (time 1761320138.75 total 30.02 last 140.01) - /unlink 5434 (time 1761320148.75 total 40.02 last 135.75) - /unlink 6944 (time 1761320158.76 total 50.03 last 150.92) - /unlink 8348 (time 1761320168.76 total 60.03 last 140.38) - /unlink 9914 (time 1761320178.76 total 70.03 last 156.54) - /unlink 11422 (time 1761320188.77 total 80.03 last 150.78) - /unlink 13001 (time 1761320198.77 total 90.04 last 157.88) - /unlink 14991 (time 1761320208.77 total 100.04 last 198.92) - /unlink 17190 (time 1761320218.77 total 110.04 last 219.88) - /unlink 19091 (time 1761320228.77 total 120.04 last 190.06) - /unlink 20000 (time 1761320233.30 total 124.56 last 201.05) - /unlink 21768 (time 1761320243.30 total 134.57 last 176.76) - /unlink 23518 (time 1761320253.30 total 144.57 last 174.97) - /unlink 24971 (time 1761320263.30 total 154.57 last 145.28) - /unlink 26475 (time 1761320273.30 total 164.57 last 150.35) - /unlink 27699 (time 1761320283.31 total 174.57 last 122.39) - /unlink 29140 (time 1761320293.31 total 184.58 last 144.05) - /unlink 30000 (time 1761320299.19 total 190.46 last 146.24) - /unlink 31787 (time 1761320309.19 total 200.46 last 178.67) - /unlink 33719 (time 1761320319.19 total 210.46 last 193.14) - /unlink 35140 (time 1761320329.20 total 220.47 last 142.04) - /unlink 36709 (time 1761320339.20 total 230.47 last 156.84) - /unlink 38394 (time 1761320349.20 total 240.47 last 168.46) - /unlink 40000 (time 1761320359.08 total 250.35 last 162.61) - /unlink 41880 (time 1761320369.08 total 260.35 last 188.00) - /unlink 43993 (time 1761320379.09 total 270.35 last 211.17) - /unlink 45954 (time 1761320389.09 total 280.36 last 195.96) - /unlink 48281 (time 1761320399.09 total 290.36 last 232.68) - /unlink 50000 (time 1761320406.59 total 297.86 last 229.32) - /unlink 52082 (time 1761320416.59 total 307.86 last 208.16) - /unlink 54252 (time 1761320426.60 total 317.86 last 216.93) - /unlink 56616 (time 1761320436.61 total 327.87 last 236.15) - /unlink 58819 (time 1761320446.61 total 337.88 last 220.27) - /unlink 60000 (time 1761320452.38 total 343.65 last 204.44) - /unlink 62059 (time 1761320462.39 total 353.65 last 205.86) - /unlink 64347 (time 1761320472.41 total 363.68 last 228.32) total: 64768 /unlink in 365.52 seconds: 177.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1487 (time 1761320502.14 total 10.01 last 148.60) - open/close 2814 (time 1761320512.15 total 20.02 last 132.56) - open/close 4070 (time 1761320522.16 total 30.02 last 125.56) - open/close 5141 (time 1761320532.16 total 40.03 last 107.04) - open/close 6106 (time 1761320542.17 total 50.03 last 96.43) - open/close 7379 (time 1761320552.18 total 60.04 last 127.18) - open/close 8692 (time 1761320562.18 total 70.05 last 131.25) - open/close 9993 (time 1761320572.18 total 80.05 last 130.07) - open/close 11277 (time 1761320582.19 total 90.05 last 128.34) - open/close 12807 (time 1761320592.20 total 100.06 last 152.91) - open/close 14238 (time 1761320602.20 total 110.06 last 143.09) - open/close 15675 (time 1761320612.20 total 120.07 last 143.58) - open/close 16874 (time 1761320622.21 total 130.08 last 119.79) - open/close 18139 (time 1761320632.22 total 140.09 last 126.36) - open/close 19284 (time 1761320642.23 total 150.10 last 114.42) - open/close 20000 (time 1761320646.95 total 154.81 last 151.84) - open/close 21251 (time 1761320656.95 total 164.82 last 125.06) - open/close 22450 (time 1761320666.96 total 174.82 last 119.83) - open/close 23620 (time 1761320676.96 total 184.83 last 116.93) - open/close 24657 (time 1761320686.97 total 194.83 last 103.64) - open/close 25921 (time 1761320696.97 total 204.84 last 126.36) - open/close 27152 (time 1761320706.98 total 214.84 last 123.03) - open/close 28372 (time 1761320716.98 total 224.84 last 121.99) - open/close 29546 (time 1761320726.98 total 234.85 last 117.33) - open/close 30000 (time 1761320731.05 total 238.92 last 111.57) - open/close 31276 (time 1761320741.06 total 248.92 last 127.53) - open/close 32479 (time 1761320751.06 total 258.93 last 120.24) - open/close 34052 (time 1761320761.06 total 268.93 last 157.28) - open/close 35648 (time 1761320771.07 total 278.93 last 159.55) - open/close 37290 (time 1761320781.07 total 288.93 last 164.17) - open/close 38858 (time 1761320791.07 total 298.94 last 156.72) - open/close 40000 (time 1761320797.07 total 304.94 last 190.47) - open/close 41643 (time 1761320807.08 total 314.94 last 164.22) - open/close 43147 (time 1761320817.09 total 324.95 last 150.26) - open/close 44666 (time 1761320827.09 total 334.95 last 151.85) - open/close 46121 (time 1761320837.09 total 344.96 last 145.43) - open/close 47439 (time 1761320847.10 total 354.96 last 131.75) - open/close 48830 (time 1761320857.10 total 364.97 last 139.04) - open/close 50000 (time 1761320865.41 total 373.28 last 140.76) - open/close 51494 (time 1761320875.42 total 383.28 last 149.33) - open/close 52782 (time 1761320885.42 total 393.28 last 128.78) - open/close 54162 (time 1761320895.43 total 403.29 last 137.92) - open/close 55693 (time 1761320905.43 total 413.30 last 153.01) - open/close 57485 (time 1761320915.43 total 423.30 last 179.20) - open/close 58997 (time 1761320925.43 total 433.30 last 151.17) - open/close 60000 (time 1761320931.90 total 439.77 last 155.08) - open/close 61478 (time 1761320941.91 total 449.77 last 147.66) - open/close 62865 (time 1761320951.91 total 459.78 last 138.64) - open/close 64435 (time 1761320961.92 total 469.79 last 156.88) total: 64768 open/close in 471.82 seconds: 137.27 ops/second - /unlink 2956 (time 1761320975.76 total 10.00 last 295.58) - /unlink 5770 (time 1761320985.77 total 20.00 last 281.29) - /unlink 8615 (time 1761320995.77 total 30.01 last 284.46) - /unlink 10000 (time 1761321000.10 total 34.34 last 319.81) - /unlink 13490 (time 1761321010.10 total 44.34 last 349.00) - /unlink 16532 (time 1761321020.10 total 54.34 last 304.15) - /unlink 19780 (time 1761321030.10 total 64.34 last 324.72) - /unlink 22744 (time 1761321040.10 total 74.34 last 296.39) - /unlink 26329 (time 1761321050.10 total 84.34 last 358.45) - /unlink 29227 (time 1761321060.11 total 94.35 last 289.70) - /unlink 30000 (time 1761321062.61 total 96.85 last 308.98) - /unlink 33317 (time 1761321072.61 total 106.85 last 331.70) - /unlink 36845 (time 1761321082.61 total 116.85 last 352.69) - /unlink 40000 (time 1761321091.77 total 126.01 last 344.58) - /unlink 43281 (time 1761321101.77 total 136.01 last 328.06) - /unlink 46395 (time 1761321111.77 total 146.01 last 311.34) - /unlink 49533 (time 1761321121.78 total 156.01 last 313.68) - /unlink 52610 (time 1761321131.78 total 166.01 last 307.69) - /unlink 56213 (time 1761321141.78 total 176.02 last 360.24) - /unlink 59709 (time 1761321151.78 total 186.02 last 349.57) - /unlink 63429 (time 1761321161.78 total 196.02 last 371.96) total: 64768 /unlink in 199.41 seconds: 324.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2759 (time 1761321196.48 total 10.00 last 275.89) - open/close 5003 (time 1761321206.48 total 20.00 last 224.38) - open/close 6387 (time 1761321216.48 total 30.00 last 138.36) - open/close 8042 (time 1761321226.49 total 40.01 last 165.45) - open/close 9945 (time 1761321236.49 total 50.01 last 190.27) - open/close 12033 (time 1761321246.49 total 60.01 last 208.68) - open/close 13824 (time 1761321256.49 total 70.02 last 179.07) - open/close 15567 (time 1761321266.50 total 80.02 last 174.26) - open/close 17778 (time 1761321276.50 total 90.02 last 221.03) - open/close 19384 (time 1761321286.50 total 100.02 last 160.57) - open/close 20000 (time 1761321291.46 total 104.99 last 124.11) - open/close 22139 (time 1761321301.47 total 114.99 last 213.78) - open/close 24089 (time 1761321311.47 total 124.99 last 194.97) - open/close 25963 (time 1761321321.48 total 135.00 last 187.29) - open/close 27764 (time 1761321331.48 total 145.00 last 180.06) - open/close 30000 (time 1761321341.18 total 154.70 last 230.51) - open/close 32081 (time 1761321351.18 total 164.70 last 208.10) - open/close 33841 (time 1761321361.19 total 174.71 last 175.90) - open/close 35582 (time 1761321371.19 total 184.71 last 174.09) - open/close 38330 (time 1761321381.19 total 194.71 last 274.79) - open/close 40000 (time 1761321387.52 total 201.04 last 263.79) - open/close 42537 (time 1761321397.54 total 211.06 last 253.22) - open/close 45183 (time 1761321407.54 total 221.06 last 264.51) - open/close 47665 (time 1761321417.54 total 231.06 last 248.14) - open/close 49438 (time 1761321427.54 total 241.07 last 177.27) - open/close 50000 (time 1761321430.18 total 243.70 last 213.20) - open/close 51754 (time 1761321440.19 total 253.71 last 175.18) - open/close 53383 (time 1761321450.19 total 263.71 last 162.89) - open/close 55240 (time 1761321460.20 total 273.72 last 185.56) - open/close 57044 (time 1761321470.20 total 283.72 last 180.40) - open/close 59011 (time 1761321480.21 total 293.73 last 196.58) - open/close 60000 (time 1761321485.72 total 299.24 last 179.44) - open/close 61831 (time 1761321495.72 total 309.25 last 183.00) - open/close 63568 (time 1761321505.73 total 319.25 last 173.68) total: 64768 open/close in 324.62 seconds: 199.52 ops/second - /unlink 4154 (time 1761321522.65 total 10.00 last 415.31) - /unlink 8033 (time 1761321532.65 total 20.00 last 387.84) - /unlink 10000 (time 1761321538.44 total 25.79 last 339.84) - /unlink 13119 (time 1761321548.45 total 35.80 last 311.79) - /unlink 16198 (time 1761321558.45 total 45.80 last 307.83) - /unlink 18981 (time 1761321568.45 total 55.80 last 278.30) - /unlink 20000 (time 1761321571.31 total 58.65 last 356.63) - /unlink 23028 (time 1761321581.31 total 68.66 last 302.75) - /unlink 25519 (time 1761321591.31 total 78.66 last 249.07) - /unlink 28680 (time 1761321601.31 total 88.66 last 316.08) - /unlink 30000 (time 1761321604.50 total 91.84 last 414.19) - /unlink 33442 (time 1761321614.50 total 101.85 last 344.14) - /unlink 36358 (time 1761321624.50 total 111.85 last 291.54) - /unlink 39884 (time 1761321634.50 total 121.85 last 352.42) - /unlink 44663 (time 1761321644.51 total 131.86 last 477.79) - /unlink 49053 (time 1761321654.51 total 141.86 last 438.94) - /unlink 50000 (time 1761321657.08 total 144.42 last 368.94) - /unlink 54228 (time 1761321667.08 total 154.43 last 422.76) - /unlink 57799 (time 1761321677.08 total 164.43 last 357.07) - /unlink 60000 (time 1761321683.72 total 171.07 last 331.47) - /unlink 64241 (time 1761321693.72 total 181.07 last 424.06) total: 64768 /unlink in 182.58 seconds: 354.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3173 (time 1761321725.64 total 10.00 last 317.30) - open/close 5632 (time 1761321735.64 total 20.00 last 245.79) - open/close 8091 (time 1761321745.64 total 30.01 last 245.87) - open/close 10000 (time 1761321755.62 total 39.98 last 191.32) - open/close 11439 (time 1761321765.63 total 49.99 last 143.76) - open/close 12244 (time 1761321775.63 total 59.99 last 80.50) - open/close 13063 (time 1761321785.64 total 70.00 last 81.86) - open/close 13834 (time 1761321795.65 total 80.01 last 77.01) - open/close 14596 (time 1761321805.68 total 90.04 last 75.98) - open/close 15437 (time 1761321815.68 total 100.04 last 84.09) - open/close 16196 (time 1761321825.69 total 110.05 last 75.82) - open/close 17100 (time 1761321835.69 total 120.05 last 90.38) - open/close 18141 (time 1761321845.70 total 130.06 last 103.98) - open/close 19147 (time 1761321855.71 total 140.07 last 100.57) - open/close 20000 (time 1761321863.79 total 148.15 last 105.56) - open/close 20975 (time 1761321873.80 total 158.16 last 97.41) - open/close 21838 (time 1761321883.80 total 168.16 last 86.26) - open/close 22774 (time 1761321893.80 total 178.16 last 93.58) - open/close 23834 (time 1761321903.81 total 188.17 last 105.90) - open/close 24932 (time 1761321913.81 total 198.17 last 109.79) - open/close 26282 (time 1761321923.82 total 208.18 last 134.89) - open/close 27955 (time 1761321933.83 total 218.19 last 167.23) - open/close 28890 (time 1761321943.83 total 228.19 last 93.47) - open/close 29949 (time 1761321953.84 total 238.20 last 105.83) - open/close 30890 (time 1761321963.84 total 248.20 last 94.07) - open/close 31807 (time 1761321973.84 total 258.20 last 91.67) - open/close 32931 (time 1761321983.85 total 268.21 last 112.31) - open/close 33907 (time 1761321993.85 total 278.21 last 97.57) - open/close 35149 (time 1761322003.86 total 288.22 last 124.18) - open/close 36572 (time 1761322013.86 total 298.22 last 142.20) - open/close 37863 (time 1761322023.86 total 308.23 last 129.07) - open/close 39501 (time 1761322033.86 total 318.23 last 163.80) - open/close 40000 (time 1761322037.07 total 321.43 last 155.88) - open/close 41449 (time 1761322047.07 total 331.43 last 144.81) - open/close 43386 (time 1761322057.08 total 341.44 last 193.54) - open/close 45509 (time 1761322067.08 total 351.44 last 212.24) - open/close 47999 (time 1761322077.08 total 361.44 last 248.97) - open/close 50000 (time 1761322083.89 total 368.25 last 293.90) - open/close 52630 (time 1761322093.89 total 378.25 last 262.99) - open/close 54897 (time 1761322103.89 total 388.26 last 226.67) - open/close 56825 (time 1761322113.90 total 398.26 last 192.76) - open/close 58663 (time 1761322123.90 total 408.26 last 183.67) - open/close 60000 (time 1761322130.88 total 415.24 last 191.69) - open/close 61568 (time 1761322140.88 total 425.25 last 156.69) - open/close 63045 (time 1761322150.89 total 435.25 last 147.66) total: 64768 open/close in 444.41 seconds: 145.74 ops/second - /unlink 4259 (time 1761322171.67 total 10.00 last 425.82) - /unlink 7977 (time 1761322181.67 total 20.00 last 371.73) - /unlink 10000 (time 1761322186.28 total 24.62 last 438.51) - /unlink 13601 (time 1761322196.29 total 34.62 last 359.98) - /unlink 16737 (time 1761322206.29 total 44.62 last 313.57) - /unlink 20000 (time 1761322215.01 total 53.35 last 374.03) - /unlink 23447 (time 1761322225.02 total 63.35 last 344.61) - /unlink 26573 (time 1761322235.02 total 73.35 last 312.55) - /unlink 30000 (time 1761322244.97 total 83.30 last 344.26) - /unlink 33945 (time 1761322254.97 total 93.31 last 394.40) - /unlink 38519 (time 1761322265.03 total 103.36 last 454.84) - /unlink 40000 (time 1761322268.66 total 106.99 last 408.56) - /unlink 44333 (time 1761322278.66 total 116.99 last 433.26) - /unlink 48489 (time 1761322288.66 total 126.99 last 415.53) - /unlink 50000 (time 1761322292.48 total 130.82 last 394.96) - /unlink 54378 (time 1761322302.49 total 140.82 last 437.72) - /unlink 58235 (time 1761322312.49 total 150.82 last 385.66) - /unlink 60000 (time 1761322316.75 total 155.09 last 413.62) - /unlink 63657 (time 1761322326.76 total 165.09 last 365.62) total: 64768 /unlink in 168.33 seconds: 384.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg408-server: LNET ready to unload oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2