-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 10:52:43 EDT 2025 oleg408-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:55:01 (1761317701) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock oleg408-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 681 (time 1761317805.16 total 10.01 last 68.05) - open/close 1357 (time 1761317815.17 total 20.02 last 67.53) - open/close 1975 (time 1761317825.19 total 30.03 last 61.73) - open/close 2659 (time 1761317835.19 total 40.03 last 68.40) - open/close 3363 (time 1761317845.19 total 50.03 last 70.39) - open/close 4057 (time 1761317855.19 total 60.03 last 69.37) - open/close 4747 (time 1761317865.19 total 70.03 last 69.00) - open/close 5459 (time 1761317875.19 total 80.03 last 71.19) - open/close 6135 (time 1761317885.20 total 90.04 last 67.57) - open/close 6816 (time 1761317895.21 total 100.05 last 68.02) - open/close 7495 (time 1761317905.21 total 110.05 last 67.87) - open/close 8164 (time 1761317915.22 total 120.06 last 66.87) - open/close 8816 (time 1761317925.23 total 130.07 last 65.11) - open/close 9533 (time 1761317935.24 total 140.08 last 71.63) - open/close 10000 (time 1761317941.40 total 146.24 last 75.82) - open/close 10766 (time 1761317951.40 total 156.25 last 76.56) - open/close 11501 (time 1761317961.41 total 166.25 last 73.46) - open/close 12164 (time 1761317971.41 total 176.25 last 66.29) - open/close 12822 (time 1761317981.41 total 186.26 last 65.78) - open/close 13486 (time 1761317991.42 total 196.26 last 66.37) - open/close 14230 (time 1761318001.43 total 206.27 last 74.35) - open/close 14880 (time 1761318011.43 total 216.27 last 64.97) - open/close 15518 (time 1761318021.44 total 226.28 last 63.77) - open/close 16129 (time 1761318031.44 total 236.28 last 61.08) - open/close 16786 (time 1761318041.44 total 246.29 last 65.67) - open/close 17432 (time 1761318051.46 total 256.30 last 64.49) - open/close 18119 (time 1761318061.49 total 266.33 last 68.53) - open/close 18711 (time 1761318071.49 total 276.33 last 59.18) - open/close 19321 (time 1761318081.49 total 286.34 last 60.97) - open/close 19987 (time 1761318091.50 total 296.34 last 66.58) - open/close 20625 (time 1761318101.50 total 306.34 last 63.78) - open/close 21272 (time 1761318111.51 total 316.35 last 64.65) - open/close 21937 (time 1761318121.51 total 326.36 last 66.46) - open/close 22613 (time 1761318131.52 total 336.36 last 67.57) - open/close 23270 (time 1761318141.54 total 346.38 last 65.59) - open/close 23895 (time 1761318151.54 total 356.38 last 62.49) - open/close 24570 (time 1761318161.55 total 366.39 last 67.43) - open/close 25296 (time 1761318171.55 total 376.39 last 72.59) - open/close 25953 (time 1761318181.56 total 386.40 last 65.66) - open/close 26593 (time 1761318191.56 total 396.41 last 63.95) - open/close 27252 (time 1761318201.59 total 406.43 last 65.74) - open/close 27877 (time 1761318211.61 total 416.45 last 62.39) - open/close 28541 (time 1761318221.62 total 426.46 last 66.33) - open/close 29287 (time 1761318231.62 total 436.46 last 74.59) - open/close 29973 (time 1761318241.63 total 446.47 last 68.52) - open/close 30598 (time 1761318251.65 total 456.49 last 62.38) - open/close 31225 (time 1761318261.68 total 466.53 last 62.47) - open/close 31839 (time 1761318271.69 total 476.53 last 61.37) - open/close 32467 (time 1761318281.70 total 486.55 last 62.72) - open/close 33055 (time 1761318291.71 total 496.55 last 58.78) - open/close 33680 (time 1761318301.73 total 506.58 last 62.33) - open/close 34255 (time 1761318311.75 total 516.59 last 57.43) - open/close 34865 (time 1761318321.76 total 526.60 last 60.91) - open/close 35487 (time 1761318331.76 total 536.61 last 62.18) - open/close 36157 (time 1761318341.77 total 546.61 last 66.97) - open/close 36846 (time 1761318351.77 total 556.61 last 68.88) - open/close 37577 (time 1761318361.77 total 566.62 last 73.08) - open/close 38244 (time 1761318371.78 total 576.62 last 66.66) - open/close 38902 (time 1761318381.78 total 586.63 last 65.77) - open/close 39552 (time 1761318391.78 total 596.63 last 65.00) - open/close 40000 (time 1761318398.63 total 603.48 last 65.42) - open/close 40603 (time 1761318408.64 total 613.49 last 60.24) - open/close 41237 (time 1761318418.64 total 623.49 last 63.40) - open/close 41830 (time 1761318428.65 total 633.49 last 59.28) - open/close 42455 (time 1761318438.66 total 643.50 last 62.44) - open/close 43158 (time 1761318448.66 total 653.50 last 70.27) - open/close 43843 (time 1761318458.66 total 663.50 last 68.49) - open/close 44541 (time 1761318468.67 total 673.52 last 69.72) - open/close 45256 (time 1761318478.69 total 683.54 last 71.36) - open/close 45967 (time 1761318488.70 total 693.54 last 71.05) - open/close 46683 (time 1761318498.72 total 703.56 last 71.48) - open/close 47337 (time 1761318508.73 total 713.57 last 65.34) - open/close 48039 (time 1761318518.74 total 723.58 last 70.13) - open/close 48729 (time 1761318528.76 total 733.60 last 68.85) - open/close 49352 (time 1761318538.77 total 743.61 last 62.21) - open/close 49905 (time 1761318548.78 total 753.62 last 55.24) - open/close 50511 (time 1761318558.78 total 763.63 last 60.60) - open/close 51170 (time 1761318568.79 total 773.63 last 65.84) - open/close 51842 (time 1761318578.80 total 783.64 last 67.15) - open/close 52526 (time 1761318588.80 total 793.65 last 68.36) - open/close 53139 (time 1761318598.82 total 803.66 last 61.23) - open/close 53780 (time 1761318608.83 total 813.67 last 64.02) - open/close 54350 (time 1761318618.84 total 823.68 last 56.92) - open/close 55007 (time 1761318628.86 total 833.70 last 65.59) - open/close 55662 (time 1761318638.86 total 843.70 last 65.48) - open/close 56316 (time 1761318648.87 total 853.71 last 65.35) - open/close 56954 (time 1761318658.88 total 863.72 last 63.74) - open/close 57589 (time 1761318668.88 total 873.72 last 63.49) - open/close 58270 (time 1761318678.88 total 883.72 last 68.09) - open/close 58961 (time 1761318688.89 total 893.74 last 69.02) - open/close 59618 (time 1761318698.90 total 903.74 last 65.69) - open/close 60000 (time 1761318704.65 total 909.49 last 66.36) - open/close 60515 (time 1761318714.65 total 919.50 last 51.50) - open/close 61132 (time 1761318724.67 total 929.51 last 61.59) - open/close 61762 (time 1761318734.68 total 939.52 last 62.97) - open/close 62359 (time 1761318744.68 total 949.52 last 59.69) - open/close 63028 (time 1761318754.69 total 959.53 last 66.84) - open/close 63672 (time 1761318764.69 total 969.53 last 64.40) - open/close 64319 (time 1761318774.69 total 979.53 last 64.67) total: 64768 open/close in 986.16 seconds: 65.68 ops/second - /unlink 1162 (time 1761318795.00 total 10.01 last 116.14) - /unlink 2313 (time 1761318805.00 total 20.01 last 115.04) - /unlink 3466 (time 1761318815.00 total 30.01 last 115.29) - /unlink 4557 (time 1761318825.01 total 40.02 last 109.01) - /unlink 5681 (time 1761318835.01 total 50.02 last 112.34) - /unlink 6784 (time 1761318845.02 total 60.03 last 110.20) - /unlink 7980 (time 1761318855.03 total 70.04 last 119.55) - /unlink 9148 (time 1761318865.03 total 80.04 last 116.77) - /unlink 10000 (time 1761318871.98 total 86.99 last 122.62) - /unlink 11233 (time 1761318881.98 total 96.99 last 123.27) - /unlink 12478 (time 1761318891.99 total 107.00 last 124.43) - /unlink 13592 (time 1761318901.99 total 117.00 last 111.32) - /unlink 14829 (time 1761318912.01 total 127.02 last 123.56) - /unlink 15982 (time 1761318922.01 total 137.02 last 115.29) - /unlink 17231 (time 1761318932.01 total 147.02 last 124.85) - /unlink 18412 (time 1761318942.02 total 157.03 last 118.00) - /unlink 19567 (time 1761318952.02 total 167.03 last 115.49) - /unlink 20000 (time 1761318955.58 total 170.59 last 121.57) - /unlink 21162 (time 1761318965.59 total 180.60 last 116.15) - /unlink 22272 (time 1761318975.59 total 190.60 last 110.98) - /unlink 23390 (time 1761318985.59 total 200.60 last 111.72) - /unlink 24577 (time 1761318995.60 total 210.61 last 118.65) - /unlink 25791 (time 1761319005.61 total 220.62 last 121.24) - /unlink 26970 (time 1761319015.62 total 230.63 last 117.83) - /unlink 28170 (time 1761319025.62 total 240.63 last 119.94) - /unlink 29375 (time 1761319035.63 total 250.64 last 120.45) - /unlink 30000 (time 1761319041.54 total 256.55 last 105.63) - /unlink 31203 (time 1761319051.55 total 266.56 last 120.23) - /unlink 32441 (time 1761319061.56 total 276.57 last 123.70) - /unlink 33706 (time 1761319071.56 total 286.57 last 126.41) - /unlink 34853 (time 1761319081.57 total 296.58 last 114.69) - /unlink 36029 (time 1761319091.57 total 306.58 last 117.55) - /unlink 37083 (time 1761319101.57 total 316.58 last 105.36) - /unlink 38175 (time 1761319111.58 total 326.59 last 109.11) - /unlink 39282 (time 1761319121.60 total 336.61 last 110.56) - /unlink 40000 (time 1761319127.35 total 342.36 last 124.82) - /unlink 41215 (time 1761319137.35 total 352.36 last 121.47) - /unlink 42361 (time 1761319147.35 total 362.36 last 114.54) - /unlink 43493 (time 1761319157.36 total 372.37 last 113.18) - /unlink 44688 (time 1761319167.37 total 382.38 last 119.37) - /unlink 45855 (time 1761319177.37 total 392.38 last 116.67) - /unlink 47009 (time 1761319187.37 total 402.38 last 115.39) - /unlink 48328 (time 1761319197.38 total 412.39 last 131.78) - /unlink 49621 (time 1761319207.38 total 422.39 last 129.26) - /unlink 50000 (time 1761319210.17 total 425.18 last 135.94) - /unlink 51258 (time 1761319220.17 total 435.18 last 125.78) - /unlink 52693 (time 1761319230.18 total 445.19 last 143.46) - /unlink 53918 (time 1761319240.18 total 455.19 last 122.50) - /unlink 55269 (time 1761319250.18 total 465.19 last 135.04) - /unlink 56819 (time 1761319260.18 total 475.19 last 154.93) - /unlink 58385 (time 1761319270.19 total 485.20 last 156.50) - /unlink 60000 (time 1761319279.25 total 494.26 last 178.29) - /unlink 61521 (time 1761319289.26 total 504.27 last 151.99) - /unlink 62972 (time 1761319299.26 total 514.27 last 145.04) - /unlink 64490 (time 1761319309.26 total 524.27 last 151.77) total: 64768 /unlink in 525.96 seconds: 123.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 872 (time 1761319341.61 total 10.01 last 87.14) - open/close 1713 (time 1761319351.62 total 20.01 last 84.06) - open/close 2578 (time 1761319361.62 total 30.02 last 86.46) - open/close 3441 (time 1761319371.62 total 40.02 last 86.28) - open/close 4250 (time 1761319381.63 total 50.03 last 80.84) - open/close 5157 (time 1761319391.64 total 60.03 last 90.65) - open/close 6091 (time 1761319401.64 total 70.04 last 93.34) - open/close 7052 (time 1761319411.65 total 80.04 last 96.06) - open/close 8138 (time 1761319421.65 total 90.04 last 108.58) - open/close 9146 (time 1761319431.65 total 100.05 last 100.76) - open/close 10000 (time 1761319440.32 total 108.71 last 98.56) - open/close 10947 (time 1761319450.32 total 118.72 last 94.65) - open/close 11890 (time 1761319460.33 total 128.72 last 94.27) - open/close 13000 (time 1761319470.33 total 138.73 last 110.94) - open/close 14283 (time 1761319480.33 total 148.73 last 128.30) - open/close 15579 (time 1761319490.34 total 158.73 last 129.53) - open/close 16817 (time 1761319500.35 total 168.75 last 123.61) - open/close 17978 (time 1761319510.35 total 178.75 last 116.10) - open/close 19108 (time 1761319520.35 total 188.75 last 112.99) - open/close 20000 (time 1761319527.36 total 195.76 last 127.25) - open/close 21276 (time 1761319537.37 total 205.76 last 127.59) - open/close 22599 (time 1761319547.37 total 215.76 last 132.29) - open/close 23880 (time 1761319557.37 total 225.77 last 128.04) - open/close 25000 (time 1761319567.37 total 235.77 last 111.99) - open/close 26144 (time 1761319577.38 total 245.77 last 114.30) - open/close 27305 (time 1761319587.38 total 255.78 last 116.07) - open/close 28055 (time 1761319597.39 total 265.79 last 74.93) - open/close 28734 (time 1761319607.40 total 275.79 last 67.87) - open/close 29416 (time 1761319617.41 total 285.81 last 68.08) - open/close 29958 (time 1761319627.42 total 295.82 last 54.16) - open/close 30539 (time 1761319637.42 total 305.82 last 58.10) - open/close 31092 (time 1761319647.43 total 315.82 last 55.29) - open/close 31793 (time 1761319657.43 total 325.82 last 70.08) - open/close 32523 (time 1761319667.43 total 335.83 last 72.98) - open/close 33288 (time 1761319677.44 total 345.83 last 76.43) - open/close 34054 (time 1761319687.45 total 355.84 last 76.52) - open/close 34660 (time 1761319697.46 total 365.86 last 60.54) - open/close 35287 (time 1761319707.47 total 375.87 last 62.64) - open/close 35924 (time 1761319717.49 total 385.88 last 63.60) - open/close 36651 (time 1761319727.50 total 395.89 last 72.61) - open/close 37329 (time 1761319737.50 total 405.90 last 67.76) - open/close 38026 (time 1761319747.51 total 415.90 last 69.68) - open/close 38764 (time 1761319757.52 total 425.92 last 73.69) - open/close 39417 (time 1761319767.56 total 435.95 last 65.07) - open/close 40000 (time 1761319776.03 total 444.43 last 68.79) - open/close 40736 (time 1761319786.04 total 454.44 last 73.54) - open/close 41454 (time 1761319796.04 total 464.44 last 71.79) - open/close 42204 (time 1761319806.05 total 474.45 last 74.93) - open/close 42860 (time 1761319816.07 total 484.47 last 65.47) - open/close 43558 (time 1761319826.08 total 494.47 last 69.74) - open/close 44305 (time 1761319836.08 total 504.48 last 74.68) - open/close 44997 (time 1761319846.09 total 514.48 last 69.17) - open/close 45626 (time 1761319856.10 total 524.49 last 62.82) - open/close 46318 (time 1761319866.11 total 534.50 last 69.15) - open/close 47061 (time 1761319876.12 total 544.51 last 74.20) - open/close 47835 (time 1761319886.12 total 554.52 last 77.37) - open/close 48632 (time 1761319896.13 total 564.52 last 79.67) - open/close 49330 (time 1761319906.13 total 574.52 last 69.78) - open/close 50000 (time 1761319915.31 total 583.71 last 72.97) - open/close 50796 (time 1761319925.32 total 593.71 last 79.57) - open/close 51444 (time 1761319935.33 total 603.72 last 64.73) - open/close 52140 (time 1761319945.34 total 613.73 last 69.54) - open/close 52874 (time 1761319955.34 total 623.73 last 73.40) - open/close 53646 (time 1761319965.34 total 633.74 last 77.14) - open/close 54427 (time 1761319975.34 total 643.74 last 78.10) - open/close 55219 (time 1761319985.34 total 653.74 last 79.20) - open/close 56072 (time 1761319995.34 total 663.74 last 85.30) - open/close 56859 (time 1761320005.35 total 673.75 last 78.62) - open/close 57564 (time 1761320015.36 total 683.75 last 70.48) - open/close 58274 (time 1761320025.38 total 693.77 last 70.86) - open/close 59018 (time 1761320035.40 total 703.79 last 74.25) - open/close 59857 (time 1761320045.40 total 713.80 last 83.86) - open/close 60770 (time 1761320055.40 total 723.80 last 91.29) - open/close 61589 (time 1761320065.40 total 733.80 last 81.90) - open/close 62408 (time 1761320075.41 total 743.80 last 81.89) - open/close 63180 (time 1761320085.41 total 753.80 last 77.18) - open/close 63929 (time 1761320095.42 total 763.81 last 74.84) - open/close 64707 (time 1761320105.42 total 773.82 last 77.77) total: 64768 open/close in 774.54 seconds: 83.62 ops/second - /unlink 1349 (time 1761320118.74 total 10.01 last 134.78) - /unlink 2675 (time 1761320128.74 total 20.01 last 132.56) - /unlink 4076 (time 1761320138.75 total 30.02 last 140.01) - /unlink 5434 (time 1761320148.75 total 40.02 last 135.75) - /unlink 6944 (time 1761320158.76 total 50.03 last 150.92) - /unlink 8348 (time 1761320168.76 total 60.03 last 140.38) - /unlink 9914 (time 1761320178.76 total 70.03 last 156.54) - /unlink 11422 (time 1761320188.77 total 80.03 last 150.78) - /unlink 13001 (time 1761320198.77 total 90.04 last 157.88) - /unlink 14991 (time 1761320208.77 total 100.04 last 198.92) - /unlink 17190 (time 1761320218.77 total 110.04 last 219.88) - /unlink 19091 (time 1761320228.77 total 120.04 last 190.06) - /unlink 20000 (time 1761320233.30 total 124.56 last 201.05) - /unlink 21768 (time 1761320243.30 total 134.57 last 176.76) - /unlink 23518 (time 1761320253.30 total 144.57 last 174.97) - /unlink 24971 (time 1761320263.30 total 154.57 last 145.28) - /unlink 26475 (time 1761320273.30 total 164.57 last 150.35) - /unlink 27699 (time 1761320283.31 total 174.57 last 122.39) - /unlink 29140 (time 1761320293.31 total 184.58 last 144.05) - /unlink 30000 (time 1761320299.19 total 190.46 last 146.24) - /unlink 31787 (time 1761320309.19 total 200.46 last 178.67) - /unlink 33719 (time 1761320319.19 total 210.46 last 193.14) - /unlink 35140 (time 1761320329.20 total 220.47 last 142.04) - /unlink 36709 (time 1761320339.20 total 230.47 last 156.84) - /unlink 38394 (time 1761320349.20 total 240.47 last 168.46) - /unlink 40000 (time 1761320359.08 total 250.35 last 162.61) - /unlink 41880 (time 1761320369.08 total 260.35 last 188.00) - /unlink 43993 (time 1761320379.09 total 270.35 last 211.17) - /unlink 45954 (time 1761320389.09 total 280.36 last 195.96) - /unlink 48281 (time 1761320399.09 total 290.36 last 232.68) - /unlink 50000 (time 1761320406.59 total 297.86 last 229.32) - /unlink 52082 (time 1761320416.59 total 307.86 last 208.16) - /unlink 54252 (time 1761320426.60 total 317.86 last 216.93) - /unlink 56616 (time 1761320436.61 total 327.87 last 236.15) - /unlink 58819 (time 1761320446.61 total 337.88 last 220.27) - /unlink 60000 (time 1761320452.38 total 343.65 last 204.44) - /unlink 62059 (time 1761320462.39 total 353.65 last 205.86) - /unlink 64347 (time 1761320472.41 total 363.68 last 228.32) total: 64768 /unlink in 365.52 seconds: 177.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1487 (time 1761320502.14 total 10.01 last 148.60) - open/close 2814 (time 1761320512.15 total 20.02 last 132.56) - open/close 4070 (time 1761320522.16 total 30.02 last 125.56) - open/close 5141 (time 1761320532.16 total 40.03 last 107.04) - open/close 6106 (time 1761320542.17 total 50.03 last 96.43) - open/close 7379 (time 1761320552.18 total 60.04 last 127.18) - open/close 8692 (time 1761320562.18 total 70.05 last 131.25) - open/close 9993 (time 1761320572.18 total 80.05 last 130.07) - open/close 11277 (time 1761320582.19 total 90.05 last 128.34) - open/close 12807 (time 1761320592.20 total 100.06 last 152.91) - open/close 14238 (time 1761320602.20 total 110.06 last 143.09) - open/close 15675 (time 1761320612.20 total 120.07 last 143.58) - open/close 16874 (time 1761320622.21 total 130.08 last 119.79) - open/close 18139 (time 1761320632.22 total 140.09 last 126.36) - open/close 19284 (time 1761320642.23 total 150.10 last 114.42) - open/close 20000 (time 1761320646.95 total 154.81 last 151.84) - open/close 21251 (time 1761320656.95 total 164.82 last 125.06) - open/close 22450 (time 1761320666.96 total 174.82 last 119.83) - open/close 23620 (time 1761320676.96 total 184.83 last 116.93) - open/close 24657 (time 1761320686.97 total 194.83 last 103.64) - open/close 25921 (time 1761320696.97 total 204.84 last 126.36) - open/close 27152 (time 1761320706.98 total 214.84 last 123.03) - open/close 28372 (time 1761320716.98 total 224.84 last 121.99) - open/close 29546 (time 1761320726.98 total 234.85 last 117.33) - open/close 30000 (time 1761320731.05 total 238.92 last 111.57) - open/close 31276 (time 1761320741.06 total 248.92 last 127.53) - open/close 32479 (time 1761320751.06 total 258.93 last 120.24) - open/close 34052 (time 1761320761.06 total 268.93 last 157.28) - open/close 35648 (time 1761320771.07 total 278.93 last 159.55) - open/close 37290 (time 1761320781.07 total 288.93 last 164.17) - open/close 38858 (time 1761320791.07 total 298.94 last 156.72) - open/close 40000 (time 1761320797.07 total 304.94 last 190.47) - open/close 41643 (time 1761320807.08 total 314.94 last 164.22) - open/close 43147 (time 1761320817.09 total 324.95 last 150.26) - open/close 44666 (time 1761320827.09 total 334.95 last 151.85) - open/close 46121 (time 1761320837.09 total 344.96 last 145.43) - open/close 47439 (time 1761320847.10 total 354.96 last 131.75) - open/close 48830 (time 1761320857.10 total 364.97 last 139.04) - open/close 50000 (time 1761320865.41 total 373.28 last 140.76) - open/close 51494 (time 1761320875.42 total 383.28 last 149.33) - open/close 52782 (time 1761320885.42 total 393.28 last 128.78) - open/close 54162 (time 1761320895.43 total 403.29 last 137.92) - open/close 55693 (time 1761320905.43 total 413.30 last 153.01) - open/close 57485 (time 1761320915.43 total 423.30 last 179.20) - open/close 58997 (time 1761320925.43 total 433.30 last 151.17) - open/close 60000 (time 1761320931.90 total 439.77 last 155.08) - open/close 61478 (time 1761320941.91 total 449.77 last 147.66) - open/close 62865 (time 1761320951.91 total 459.78 last 138.64) - open/close 64435 (time 1761320961.92 total 469.79 last 156.88) total: 64768 open/close in 471.82 seconds: 137.27 ops/second - /unlink 2956 (time 1761320975.76 total 10.00 last 295.58) - /unlink 5770 (time 1761320985.77 total 20.00 last 281.29) - /unlink 8615 (time 1761320995.77 total 30.01 last 284.46) - /unlink 10000 (time 1761321000.10 total 34.34 last 319.81) - /unlink 13490 (time 1761321010.10 total 44.34 last 349.00) - /unlink 16532 (time 1761321020.10 total 54.34 last 304.15) - /unlink 19780 (time 1761321030.10 total 64.34 last 324.72) - /unlink 22744 (time 1761321040.10 total 74.34 last 296.39) - /unlink 26329 (time 1761321050.10 total 84.34 last 358.45) - /unlink 29227 (time 1761321060.11 total 94.35 last 289.70) - /unlink 30000 (time 1761321062.61 total 96.85 last 308.98) - /unlink 33317 (time 1761321072.61 total 106.85 last 331.70) - /unlink 36845 (time 1761321082.61 total 116.85 last 352.69) - /unlink 40000 (time 1761321091.77 total 126.01 last 344.58) - /unlink 43281 (time 1761321101.77 total 136.01 last 328.06) - /unlink 46395 (time 1761321111.77 total 146.01 last 311.34) - /unlink 49533 (time 1761321121.78 total 156.01 last 313.68) - /unlink 52610 (time 1761321131.78 total 166.01 last 307.69) - /unlink 56213 (time 1761321141.78 total 176.02 last 360.24) - /unlink 59709 (time 1761321151.78 total 186.02 last 349.57) - /unlink 63429 (time 1761321161.78 total 196.02 last 371.96) total: 64768 /unlink in 199.41 seconds: 324.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2759 (time 1761321196.48 total 10.00 last 275.89) - open/close 5003 (time 1761321206.48 total 20.00 last 224.38) - open/close 6387 (time 1761321216.48 total 30.00 last 138.36) - open/close 8042 (time 1761321226.49 total 40.01 last 165.45) - open/close 9945 (time 1761321236.49 total 50.01 last 190.27) - open/close 12033 (time 1761321246.49 total 60.01 last 208.68) - open/close 13824 (time 1761321256.49 total 70.02 last 179.07) - open/close 15567 (time 1761321266.50 total 80.02 last 174.26) - open/close 17778 (time 1761321276.50 total 90.02 last 221.03) - open/close 19384 (time 1761321286.50 total 100.02 last 160.57) - open/close 20000 (time 1761321291.46 total 104.99 last 124.11) - open/close 22139 (time 1761321301.47 total 114.99 last 213.78) - open/close 24089 (time 1761321311.47 total 124.99 last 194.97) - open/close 25963 (time 1761321321.48 total 135.00 last 187.29) - open/close 27764 (time 1761321331.48 total 145.00 last 180.06) - open/close 30000 (time 1761321341.18 total 154.70 last 230.51) - open/close 32081 (time 1761321351.18 total 164.70 last 208.10) - open/close 33841 (time 1761321361.19 total 174.71 last 175.90) - open/close 35582 (time 1761321371.19 total 184.71 last 174.09) - open/close 38330 (time 1761321381.19 total 194.71 last 274.79) - open/close 40000 (time 1761321387.52 total 201.04 last 263.79) - open/close 42537 (time 1761321397.54 total 211.06 last 253.22) - open/close 45183 (time 1761321407.54 total 221.06 last 264.51) - open/close 47665 (time 1761321417.54 total 231.06 last 248.14) - open/close 49438 (time 1761321427.54 total 241.07 last 177.27) - open/close 50000 (time 1761321430.18 total 243.70 last 213.20) - open/close 51754 (time 1761321440.19 total 253.71 last 175.18) - open/close 53383 (time 1761321450.19 total 263.71 last 162.89) - open/close 55240 (time 1761321460.20 total 273.72 last 185.56) - open/close 57044 (time 1761321470.20 total 283.72 last 180.40) - open/close 59011 (time 1761321480.21 total 293.73 last 196.58) - open/close 60000 (time 1761321485.72 total 299.24 last 179.44) - open/close 61831 (time 1761321495.72 total 309.25 last 183.00) - open/close 63568 (time 1761321505.73 total 319.25 last 173.68) total: 64768 open/close in 324.62 seconds: 199.52 ops/second - /unlink 4154 (time 1761321522.65 total 10.00 last 415.31) - /unlink 8033 (time 1761321532.65 total 20.00 last 387.84) - /unlink 10000 (time 1761321538.44 total 25.79 last 339.84) - /unlink 13119 (time 1761321548.45 total 35.80 last 311.79) - /unlink 16198 (time 1761321558.45 total 45.80 last 307.83) - /unlink 18981 (time 1761321568.45 total 55.80 last 278.30) - /unlink 20000 (time 1761321571.31 total 58.65 last 356.63) - /unlink 23028 (time 1761321581.31 total 68.66 last 302.75) - /unlink 25519 (time 1761321591.31 total 78.66 last 249.07) - /unlink 28680 (time 1761321601.31 total 88.66 last 316.08) - /unlink 30000 (time 1761321604.50 total 91.84 last 414.19) - /unlink 33442 (time 1761321614.50 total 101.85 last 344.14) - /unlink 36358 (time 1761321624.50 total 111.85 last 291.54) - /unlink 39884 (time 1761321634.50 total 121.85 last 352.42) - /unlink 44663 (time 1761321644.51 total 131.86 last 477.79) - /unlink 49053 (time 1761321654.51 total 141.86 last 438.94) - /unlink 50000 (time 1761321657.08 total 144.42 last 368.94) - /unlink 54228 (time 1761321667.08 total 154.43 last 422.76) - /unlink 57799 (time 1761321677.08 total 164.43 last 357.07) - /unlink 60000 (time 1761321683.72 total 171.07 last 331.47) - /unlink 64241 (time 1761321693.72 total 181.07 last 424.06) total: 64768 /unlink in 182.58 seconds: 354.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3173 (time 1761321725.64 total 10.00 last 317.30) - open/close 5632 (time 1761321735.64 total 20.00 last 245.79) - open/close 8091 (time 1761321745.64 total 30.01 last 245.87) - open/close 10000 (time 1761321755.62 total 39.98 last 191.32) - open/close 11439 (time 1761321765.63 total 49.99 last 143.76) - open/close 12244 (time 1761321775.63 total 59.99 last 80.50) - open/close 13063 (time 1761321785.64 total 70.00 last 81.86) - open/close 13834 (time 1761321795.65 total 80.01 last 77.01) - open/close 14596 (time 1761321805.68 total 90.04 last 75.98) - open/close 15437 (time 1761321815.68 total 100.04 last 84.09) - open/close 16196 (time 1761321825.69 total 110.05 last 75.82) - open/close 17100 (time 1761321835.69 total 120.05 last 90.38) - open/close 18141 (time 1761321845.70 total 130.06 last 103.98) - open/close 19147 (time 1761321855.71 total 140.07 last 100.57) - open/close 20000 (time 1761321863.79 total 148.15 last 105.56) - open/close 20975 (time 1761321873.80 total 158.16 last 97.41) - open/close 21838 (time 1761321883.80 total 168.16 last 86.26) - open/close 22774 (time 1761321893.80 total 178.16 last 93.58) - open/close 23834 (time 1761321903.81 total 188.17 last 105.90) - open/close 24932 (time 1761321913.81 total 198.17 last 109.79) - open/close 26282 (time 1761321923.82 total 208.18 last 134.89) - open/close 27955 (time 1761321933.83 total 218.19 last 167.23) - open/close 28890 (time 1761321943.83 total 228.19 last 93.47) - open/close 29949 (time 1761321953.84 total 238.20 last 105.83) - open/close 30890 (time 1761321963.84 total 248.20 last 94.07) - open/close 31807 (time 1761321973.84 total 258.20 last 91.67) - open/close 32931 (time 1761321983.85 total 268.21 last 112.31) - open/close 33907 (time 1761321993.85 total 278.21 last 97.57) - open/close 35149 (time 1761322003.86 total 288.22 last 124.18) - open/close 36572 (time 1761322013.86 total 298.22 last 142.20) - open/close 37863 (time 1761322023.86 total 308.23 last 129.07) - open/close 39501 (time 1761322033.86 total 318.23 last 163.80) - open/close 40000 (time 1761322037.07 total 321.43 last 155.88) - open/close 41449 (time 1761322047.07 total 331.43 last 144.81) - open/close 43386 (time 1761322057.08 total 341.44 last 193.54) - open/close 45509 (time 1761322067.08 total 351.44 last 212.24) - open/close 47999 (time 1761322077.08 total 361.44 last 248.97) - open/close 50000 (time 1761322083.89 total 368.25 last 293.90) - open/close 52630 (time 1761322093.89 total 378.25 last 262.99) - open/close 54897 (time 1761322103.89 total 388.26 last 226.67) - open/close 56825 (time 1761322113.90 total 398.26 last 192.76) - open/close 58663 (time 1761322123.90 total 408.26 last 183.67) - open/close 60000 (time 1761322130.88 total 415.24 last 191.69) - open/close 61568 (time 1761322140.88 total 425.25 last 156.69) - open/close 63045 (time 1761322150.89 total 435.25 last 147.66) total: 64768 open/close in 444.41 seconds: 145.74 ops/second - /unlink 4259 (time 1761322171.67 total 10.00 last 425.82) - /unlink 7977 (time 1761322181.67 total 20.00 last 371.73) - /unlink 10000 (time 1761322186.28 total 24.62 last 438.51) - /unlink 13601 (time 1761322196.29 total 34.62 last 359.98) - /unlink 16737 (time 1761322206.29 total 44.62 last 313.57) - /unlink 20000 (time 1761322215.01 total 53.35 last 374.03) - /unlink 23447 (time 1761322225.02 total 63.35 last 344.61) - /unlink 26573 (time 1761322235.02 total 73.35 last 312.55) - /unlink 30000 (time 1761322244.97 total 83.30 last 344.26) - /unlink 33945 (time 1761322254.97 total 93.31 last 394.40) - /unlink 38519 (time 1761322265.03 total 103.36 last 454.84) - /unlink 40000 (time 1761322268.66 total 106.99 last 408.56) - /unlink 44333 (time 1761322278.66 total 116.99 last 433.26) - /unlink 48489 (time 1761322288.66 total 126.99 last 415.53) - /unlink 50000 (time 1761322292.48 total 130.82 last 394.96) - /unlink 54378 (time 1761322302.49 total 140.82 last 437.72) - /unlink 58235 (time 1761322312.49 total 150.82 last 385.66) - /unlink 60000 (time 1761322316.75 total 155.09 last 413.62) - /unlink 63657 (time 1761322326.76 total 165.09 last 365.62) total: 64768 /unlink in 168.33 seconds: 384.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg408-server: LNET ready to unload oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 106 (4669s) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4834 sec ========== 12:13:19 (1761322399)