-----============= acceptance-small: conf-sanity ============----- Thu Jul 24 09:42:49 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg221-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg221-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg221-server oleg221-server: oleg221-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg221-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg221-server: oleg221-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg221-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:44:10 (1753364650) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) oleg221-server: oleg221-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg221-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg221-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1432 (time 1753364700.97 total 10.01 last 143.10) - open/close 2813 (time 1753364710.97 total 20.01 last 138.06) - open/close 4070 (time 1753364720.98 total 30.01 last 125.64) - open/close 5447 (time 1753364730.99 total 40.02 last 137.61) - open/close 6765 (time 1753364740.99 total 50.02 last 131.77) - open/close 8192 (time 1753364751.00 total 60.03 last 142.58) - open/close 9254 (time 1753364761.00 total 70.04 last 106.17) - open/close 10000 (time 1753364767.19 total 76.23 last 120.43) - open/close 11448 (time 1753364777.19 total 86.23 last 144.78) - open/close 13208 (time 1753364787.20 total 96.23 last 175.94) - open/close 14996 (time 1753364797.20 total 106.24 last 178.75) - open/close 16845 (time 1753364807.21 total 116.24 last 184.77) - open/close 18282 (time 1753364817.21 total 126.25 last 143.64) - open/close 19681 (time 1753364827.22 total 136.25 last 139.83) - open/close 20000 (time 1753364829.66 total 138.70 last 130.61) - open/close 21659 (time 1753364839.66 total 148.70 last 165.89) - open/close 23472 (time 1753364849.66 total 158.70 last 181.23) - open/close 25185 (time 1753364859.66 total 168.70 last 171.28) - open/close 26988 (time 1753364869.67 total 178.70 last 180.28) - open/close 29164 (time 1753364879.67 total 188.70 last 217.57) - open/close 30000 (time 1753364885.90 total 194.94 last 134.11) - open/close 31459 (time 1753364895.90 total 204.94 last 145.85) - open/close 33081 (time 1753364905.91 total 214.94 last 162.17) - open/close 35085 (time 1753364915.91 total 224.95 last 200.34) - open/close 37227 (time 1753364925.91 total 234.95 last 214.11) - open/close 39364 (time 1753364935.91 total 244.95 last 213.68) - open/close 40000 (time 1753364940.05 total 249.08 last 153.86) - open/close 41194 (time 1753364950.09 total 259.12 last 118.94) - open/close 41922 (time 1753364960.09 total 269.12 last 72.80) - open/close 42588 (time 1753364970.11 total 279.15 last 66.45) - open/close 43282 (time 1753364980.11 total 289.15 last 69.37) - open/close 44237 (time 1753364990.12 total 299.16 last 95.43) - open/close 45474 (time 1753365000.12 total 309.16 last 123.67) - open/close 46731 (time 1753365010.13 total 319.17 last 125.57) - open/close 48223 (time 1753365020.14 total 329.17 last 149.15) - open/close 49479 (time 1753365030.14 total 339.17 last 125.58) - open/close 50000 (time 1753365034.43 total 343.47 last 121.27) - open/close 51141 (time 1753365044.44 total 353.48 last 114.04) - open/close 52418 (time 1753365054.44 total 363.48 last 127.63) - open/close 53744 (time 1753365064.45 total 373.49 last 132.54) - open/close 54933 (time 1753365074.45 total 383.49 last 118.88) - open/close 56121 (time 1753365084.45 total 393.49 last 118.77) - open/close 57313 (time 1753365094.46 total 403.49 last 119.14) - open/close 58415 (time 1753365104.46 total 413.50 last 110.18) - open/close 59582 (time 1753365114.47 total 423.50 last 116.62) - open/close 60000 (time 1753365117.17 total 426.21 last 154.47) - open/close 61518 (time 1753365127.17 total 436.21 last 151.77) - open/close 63210 (time 1753365137.18 total 446.21 last 169.13) total: 64768 open/close in 454.96 seconds: 142.36 ops/second - /unlink 3024 (time 1753365157.60 total 10.00 last 302.38) - /unlink 5543 (time 1753365167.61 total 20.01 last 251.79) - /unlink 8270 (time 1753365177.61 total 30.01 last 272.69) - /unlink 10000 (time 1753365183.55 total 35.95 last 290.92) - /unlink 12883 (time 1753365193.55 total 45.95 last 288.27) - /unlink 16300 (time 1753365203.55 total 55.95 last 341.65) - /unlink 19846 (time 1753365213.56 total 65.96 last 354.55) - /unlink 23258 (time 1753365223.56 total 75.96 last 341.15) - /unlink 26585 (time 1753365233.56 total 85.96 last 332.60) - /unlink 29881 (time 1753365243.56 total 95.96 last 329.58) - /unlink 33201 (time 1753365253.56 total 105.96 last 331.92) - /unlink 36688 (time 1753365263.57 total 115.97 last 348.59) - /unlink 40000 (time 1753365273.47 total 125.87 last 334.29) - /unlink 43950 (time 1753365283.47 total 135.87 last 394.99) - /unlink 47533 (time 1753365293.48 total 145.88 last 358.25) - /unlink 50000 (time 1753365300.53 total 152.93 last 349.95) - /unlink 53473 (time 1753365310.53 total 162.93 last 347.29) - /unlink 57737 (time 1753365320.53 total 172.93 last 426.37) - /unlink 60000 (time 1753365326.96 total 179.36 last 351.49) - /unlink 63770 (time 1753365336.97 total 189.37 last 376.94) total: 64768 /unlink in 192.15 seconds: 337.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2598 (time 1753365369.60 total 10.00 last 259.73) - open/close 5007 (time 1753365379.60 total 20.00 last 240.87) - open/close 7693 (time 1753365389.61 total 30.00 last 268.59) - open/close 10000 (time 1753365399.04 total 39.44 last 244.62) - open/close 12400 (time 1753365409.04 total 49.44 last 239.90) - open/close 14824 (time 1753365419.04 total 59.44 last 242.38) - open/close 17907 (time 1753365429.04 total 69.44 last 308.23) - open/close 20000 (time 1753365435.84 total 76.24 last 307.91) - open/close 22930 (time 1753365445.84 total 86.24 last 292.95) - open/close 25145 (time 1753365455.84 total 96.24 last 221.48) - open/close 27228 (time 1753365465.85 total 106.25 last 208.23) - open/close 29677 (time 1753365475.85 total 116.25 last 244.82) - open/close 32091 (time 1753365485.85 total 126.25 last 241.36) - open/close 34160 (time 1753365495.86 total 136.26 last 206.80) - open/close 36235 (time 1753365505.86 total 146.26 last 207.47) - open/close 38932 (time 1753365515.86 total 156.26 last 269.65) - open/close 40000 (time 1753365520.20 total 160.60 last 246.18) - open/close 42508 (time 1753365530.20 total 170.60 last 250.75) - open/close 44858 (time 1753365540.20 total 180.60 last 234.94) - open/close 47241 (time 1753365550.20 total 190.60 last 238.28) - open/close 49661 (time 1753365560.21 total 200.61 last 241.95) - open/close 52623 (time 1753365570.21 total 210.61 last 296.13) - open/close 55085 (time 1753365580.21 total 220.61 last 246.13) - open/close 57660 (time 1753365590.21 total 230.61 last 257.50) - open/close 60000 (time 1753365596.91 total 237.30 last 349.56) - open/close 63454 (time 1753365606.91 total 247.31 last 345.38) total: 64768 open/close in 251.59 seconds: 257.44 ops/second - /unlink 5330 (time 1753365622.64 total 10.00 last 532.92) - /unlink 10000 (time 1753365631.10 total 18.46 last 551.81) - /unlink 15111 (time 1753365641.10 total 28.47 last 511.00) - /unlink 20000 (time 1753365650.12 total 37.48 last 542.30) - /unlink 26436 (time 1753365660.12 total 47.48 last 643.57) - /unlink 30000 (time 1753365665.89 total 53.25 last 617.88) - /unlink 36653 (time 1753365675.89 total 63.25 last 665.25) - /unlink 40000 (time 1753365680.98 total 68.35 last 656.71) - /unlink 45677 (time 1753365690.99 total 78.35 last 567.62) - /unlink 50000 (time 1753365699.15 total 86.52 last 529.38) - /unlink 55759 (time 1753365709.15 total 96.52 last 575.89) - /unlink 60000 (time 1753365716.01 total 103.38 last 618.09) - /unlink 62016 (time 1753365726.02 total 113.39 last 201.41) - /unlink 63145 (time 1753365736.03 total 123.39 last 112.86) - /unlink 64554 (time 1753365746.04 total 133.40 last 140.78) total: 64768 /unlink in 135.50 seconds: 478.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 399 (time 1753365787.10 total 10.01 last 39.85) - open/close 768 (time 1753365797.11 total 20.02 last 36.88) - open/close 1171 (time 1753365807.12 total 30.03 last 40.25) - open/close 1713 (time 1753365817.13 total 40.04 last 54.13) - open/close 2346 (time 1753365827.13 total 50.04 last 63.29) - open/close 2989 (time 1753365837.14 total 60.05 last 64.25) - open/close 3735 (time 1753365847.14 total 70.05 last 74.59) - open/close 4527 (time 1753365857.15 total 80.06 last 79.13) - open/close 5318 (time 1753365867.16 total 90.07 last 79.04) - open/close 6070 (time 1753365877.17 total 100.08 last 75.10) - open/close 6702 (time 1753365887.18 total 110.09 last 63.15) - open/close 7417 (time 1753365897.19 total 120.10 last 71.41) - open/close 8007 (time 1753365907.19 total 130.11 last 58.97) - open/close 8718 (time 1753365917.21 total 140.12 last 71.02) - open/close 9433 (time 1753365927.22 total 150.13 last 71.43) - open/close 10000 (time 1753365934.71 total 157.63 last 75.62) - open/close 10722 (time 1753365944.72 total 167.63 last 72.19) - open/close 11505 (time 1753365954.73 total 177.64 last 78.21) - open/close 12255 (time 1753365964.74 total 187.65 last 74.92) - open/close 13155 (time 1753365974.74 total 197.66 last 89.94) - open/close 13972 (time 1753365984.75 total 207.66 last 81.63) - open/close 14740 (time 1753365994.76 total 217.67 last 76.76) - open/close 15565 (time 1753366004.79 total 227.70 last 82.23) - open/close 16350 (time 1753366014.80 total 237.71 last 78.41) - open/close 17015 (time 1753366024.81 total 247.72 last 66.46) - open/close 17785 (time 1753366034.82 total 257.73 last 76.90) - open/close 18645 (time 1753366044.82 total 267.74 last 85.98) - open/close 19632 (time 1753366054.84 total 277.75 last 98.59) - open/close 20000 (time 1753366059.19 total 282.10 last 84.47) - open/close 20927 (time 1753366069.20 total 292.11 last 92.66) - open/close 21991 (time 1753366079.20 total 302.11 last 106.36) - open/close 23055 (time 1753366089.21 total 312.12 last 106.34) - open/close 24228 (time 1753366099.21 total 322.12 last 117.25) - open/close 25546 (time 1753366109.22 total 332.13 last 131.74) - open/close 26941 (time 1753366119.22 total 342.13 last 139.47) - open/close 28840 (time 1753366129.22 total 352.13 last 189.85) - open/close 30000 (time 1753366134.24 total 357.15 last 231.14) - open/close 32053 (time 1753366144.24 total 367.16 last 205.18) - open/close 33777 (time 1753366154.24 total 377.16 last 172.40) - open/close 35940 (time 1753366164.25 total 387.16 last 216.28) - open/close 38159 (time 1753366174.25 total 397.16 last 221.89) - open/close 40000 (time 1753366179.96 total 402.87 last 322.00) - open/close 42281 (time 1753366189.97 total 412.88 last 228.06) - open/close 45459 (time 1753366199.97 total 422.88 last 317.72) - open/close 48520 (time 1753366209.97 total 432.88 last 306.02) - open/close 50000 (time 1753366214.36 total 437.27 last 337.11) - open/close 54193 (time 1753366224.36 total 447.27 last 419.27) - open/close 57439 (time 1753366234.36 total 457.27 last 324.56) - open/close 60000 (time 1753366244.25 total 467.16 last 259.12) - open/close 63263 (time 1753366254.25 total 477.16 last 326.23) total: 64768 open/close in 481.04 seconds: 134.64 ops/second - /unlink 5428 (time 1753366269.65 total 10.00 last 542.78) - /unlink 10000 (time 1753366277.88 total 18.23 last 555.58) - /unlink 15302 (time 1753366287.88 total 28.23 last 530.18) - /unlink 20000 (time 1753366297.41 total 37.76 last 493.16) - /unlink 26108 (time 1753366307.41 total 47.76 last 610.76) - /unlink 30000 (time 1753366315.88 total 56.22 last 459.71) - /unlink 35414 (time 1753366325.88 total 66.22 last 541.35) - /unlink 40000 (time 1753366334.69 total 75.04 last 520.29) - /unlink 45370 (time 1753366344.69 total 85.04 last 536.97) - /unlink 49536 (time 1753366354.69 total 95.04 last 416.55) - /unlink 55700 (time 1753366364.69 total 105.04 last 616.39) - /unlink 60000 (time 1753366371.12 total 111.47 last 668.65) total: 64768 /unlink in 119.05 seconds: 544.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3951 (time 1753366408.90 total 10.00 last 395.06) - open/close 8188 (time 1753366418.90 total 20.00 last 423.66) - open/close 10000 (time 1753366422.84 total 23.94 last 460.25) - open/close 14785 (time 1753366432.84 total 33.94 last 478.49) - open/close 19728 (time 1753366442.84 total 43.94 last 494.25) - open/close 24351 (time 1753366452.84 total 53.94 last 462.22) - open/close 28901 (time 1753366462.84 total 63.94 last 454.93) - open/close 30000 (time 1753366464.86 total 65.96 last 545.81) - open/close 34990 (time 1753366474.86 total 75.96 last 498.92) - open/close 39695 (time 1753366484.86 total 85.96 last 470.48) - open/close 44427 (time 1753366494.86 total 95.96 last 473.14) - open/close 49121 (time 1753366504.86 total 105.96 last 469.39) - open/close 50000 (time 1753366507.14 total 108.24 last 385.49) - open/close 54755 (time 1753366517.14 total 118.24 last 475.41) - open/close 59527 (time 1753366527.14 total 128.24 last 477.18) - open/close 64429 (time 1753366537.14 total 138.24 last 490.15) total: 64768 open/close in 139.12 seconds: 465.56 ops/second - /unlink 7325 (time 1753366549.30 total 10.00 last 732.46) - /unlink 10000 (time 1753366552.87 total 13.57 last 748.99) - /unlink 17262 (time 1753366562.87 total 23.57 last 726.12) - /unlink 20000 (time 1753366567.15 total 27.85 last 640.16) - /unlink 26252 (time 1753366577.15 total 37.85 last 625.18) - /unlink 30000 (time 1753366582.21 total 42.91 last 741.12) - /unlink 36267 (time 1753366592.21 total 52.91 last 626.66) - /unlink 40000 (time 1753366598.30 total 59.00 last 613.08) - /unlink 46157 (time 1753366608.30 total 69.00 last 615.66) - /unlink 50000 (time 1753366614.13 total 74.83 last 659.10) - /unlink 56411 (time 1753366624.13 total 84.83 last 641.06) - /unlink 60000 (time 1753366630.92 total 91.62 last 528.33) - /unlink 61652 (time 1753366640.92 total 101.62 last 165.17) - /unlink 63343 (time 1753366650.94 total 111.64 last 168.83) total: 64768 /unlink in 120.46 seconds: 537.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 848 (time 1753366693.84 total 10.00 last 84.80) - open/close 1767 (time 1753366703.85 total 20.01 last 91.85) - open/close 2695 (time 1753366713.86 total 30.01 last 92.73) - open/close 3753 (time 1753366723.87 total 40.02 last 105.68) - open/close 5085 (time 1753366733.87 total 50.02 last 133.20) - open/close 6181 (time 1753366743.87 total 60.03 last 109.56) - open/close 7228 (time 1753366753.88 total 70.03 last 104.63) - open/close 8380 (time 1753366763.88 total 80.04 last 115.14) - open/close 9582 (time 1753366773.90 total 90.05 last 120.04) - open/close 10000 (time 1753366776.91 total 93.07 last 138.70) - open/close 11415 (time 1753366786.91 total 103.07 last 141.45) - open/close 12631 (time 1753366796.92 total 113.07 last 121.56) - open/close 13681 (time 1753366806.92 total 123.08 last 104.96) - open/close 14690 (time 1753366816.92 total 133.08 last 100.86) - open/close 15900 (time 1753366826.93 total 143.09 last 120.93) - open/close 16860 (time 1753366836.93 total 153.09 last 95.97) - open/close 17912 (time 1753366846.93 total 163.09 last 105.18) - open/close 19018 (time 1753366856.94 total 173.09 last 110.56) - open/close 20000 (time 1753366866.46 total 182.62 last 103.11) - open/close 21061 (time 1753366876.46 total 192.62 last 106.09) - open/close 22112 (time 1753366886.47 total 202.62 last 105.05) - open/close 23031 (time 1753366896.47 total 212.63 last 91.87) - open/close 23880 (time 1753366906.47 total 222.63 last 84.88) - open/close 24745 (time 1753366916.48 total 232.64 last 86.42) - open/close 25677 (time 1753366926.49 total 242.65 last 93.14) - open/close 26744 (time 1753366936.50 total 252.65 last 106.63) - open/close 27775 (time 1753366946.50 total 262.65 last 103.09) - open/close 28968 (time 1753366956.50 total 272.66 last 119.28) - open/close 30000 (time 1753366965.19 total 281.35 last 118.73) - open/close 30982 (time 1753366975.20 total 291.35 last 98.15) - open/close 31889 (time 1753366985.21 total 301.36 last 90.62) - open/close 32818 (time 1753366995.21 total 311.37 last 92.86) - open/close 33759 (time 1753367005.22 total 321.37 last 94.03) - open/close 34774 (time 1753367015.22 total 331.38 last 101.46) - open/close 36020 (time 1753367025.23 total 341.39 last 124.50) - open/close 37091 (time 1753367035.24 total 351.39 last 107.04) - open/close 38144 (time 1753367045.24 total 361.39 last 105.30) - open/close 39552 (time 1753367055.24 total 371.39 last 140.77) - open/close 40000 (time 1753367059.41 total 375.56 last 107.48) - open/close 41296 (time 1753367069.41 total 385.57 last 129.53) - open/close 43100 (time 1753367079.41 total 395.57 last 180.34) - open/close 44728 (time 1753367089.42 total 405.58 last 162.70) - open/close 45947 (time 1753367099.42 total 415.58 last 121.85) - open/close 47288 (time 1753367109.43 total 425.59 last 133.98) - open/close 48459 (time 1753367119.45 total 435.61 last 116.86) - open/close 49666 (time 1753367129.46 total 445.61 last 120.68) - open/close 50000 (time 1753367132.63 total 448.79 last 105.10) - open/close 51248 (time 1753367142.64 total 458.79 last 124.77) - open/close 52977 (time 1753367152.64 total 468.79 last 172.86) - open/close 54881 (time 1753367162.64 total 478.79 last 190.39) - open/close 56639 (time 1753367172.64 total 488.80 last 175.71) - open/close 58179 (time 1753367182.65 total 498.81 last 153.86) - open/close 59912 (time 1753367192.65 total 508.81 last 173.28) - open/close 61563 (time 1753367202.66 total 518.81 last 165.07) - open/close 63211 (time 1753367212.66 total 528.82 last 164.69) - open/close 64709 (time 1753367222.67 total 538.83 last 149.70) total: 64768 open/close in 539.36 seconds: 120.08 ops/second - /unlink 1962 (time 1753367235.21 total 10.00 last 196.16) - /unlink 3817 (time 1753367245.21 total 20.00 last 185.47) - /unlink 5919 (time 1753367255.21 total 30.00 last 210.18) - /unlink 7773 (time 1753367265.21 total 40.00 last 185.39) - /unlink 9565 (time 1753367275.21 total 50.01 last 179.18) - /unlink 10000 (time 1753367277.54 total 52.33 last 187.21) - /unlink 12148 (time 1753367287.54 total 62.33 last 214.79) - /unlink 14616 (time 1753367297.54 total 72.33 last 246.72) - /unlink 16541 (time 1753367307.54 total 82.33 last 192.47) - /unlink 18637 (time 1753367317.54 total 92.34 last 209.59) - /unlink 20000 (time 1753367324.04 total 98.84 last 209.68) - /unlink 22174 (time 1753367334.05 total 108.84 last 217.30) - /unlink 24395 (time 1753367344.05 total 118.84 last 222.00) - /unlink 26576 (time 1753367354.05 total 128.85 last 218.07) - /unlink 28367 (time 1753367364.06 total 138.85 last 178.99) - /unlink 30000 (time 1753367373.36 total 148.16 last 175.53) - /unlink 31940 (time 1753367383.37 total 158.16 last 193.90) - /unlink 33838 (time 1753367393.37 total 168.17 last 189.70) - /unlink 35774 (time 1753367403.38 total 178.17 last 193.49) - /unlink 38707 (time 1753367413.38 total 188.17 last 293.24) - /unlink 40000 (time 1753367418.06 total 192.86 last 276.16) - /unlink 42750 (time 1753367428.06 total 202.86 last 274.97) - /unlink 45015 (time 1753367438.06 total 212.86 last 226.46) - /unlink 47141 (time 1753367448.07 total 222.86 last 212.58) - /unlink 49030 (time 1753367458.07 total 232.86 last 188.87) - /unlink 50000 (time 1753367464.21 total 239.00 last 157.91) - /unlink 51848 (time 1753367474.23 total 249.02 last 184.46) - /unlink 53674 (time 1753367484.23 total 259.02 last 182.55) - /unlink 55490 (time 1753367494.23 total 269.03 last 181.58) - /unlink 57421 (time 1753367504.23 total 279.03 last 193.09) - /unlink 59823 (time 1753367514.24 total 289.03 last 240.13) - /unlink 62191 (time 1753367524.24 total 299.03 last 236.76) total: 64768 /unlink in 308.73 seconds: 209.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local oleg221-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg221-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 PASS 106 (2919s) Stopping clients: oleg221-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg221-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 oleg221-server: oleg221-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg221-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3028 sec ========== 10:33:18 (1753367598) === conf-sanity: start cleanup 10:33:19 (1753367599) === === conf-sanity: finish cleanup 10:33:20 (1753367600) ===