-----============= acceptance-small: conf-sanity ============----- Fri Jul 25 06:44:45 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg137-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg137-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg137-server oleg137-server: oleg137-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg137-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:46:28 (1753440388) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) oleg137-server: oleg137-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg137-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 827 (time 1753440466.33 total 10.01 last 82.63) - open/close 1618 (time 1753440476.33 total 20.01 last 79.09) - open/close 2416 (time 1753440486.34 total 30.02 last 79.72) - open/close 3243 (time 1753440496.34 total 40.03 last 82.64) - open/close 4060 (time 1753440506.36 total 50.04 last 81.57) - open/close 5079 (time 1753440516.37 total 60.06 last 101.76) - open/close 6100 (time 1753440526.39 total 70.07 last 101.96) - open/close 7025 (time 1753440536.39 total 80.08 last 92.44) - open/close 7936 (time 1753440546.40 total 90.08 last 91.07) - open/close 8894 (time 1753440556.40 total 100.09 last 95.76) - open/close 10000 (time 1753440565.74 total 109.42 last 118.45) - open/close 11402 (time 1753440575.74 total 119.42 last 140.18) - open/close 12726 (time 1753440585.75 total 129.43 last 132.34) - open/close 13953 (time 1753440595.75 total 139.43 last 122.63) - open/close 14934 (time 1753440605.76 total 149.44 last 98.05) - open/close 15836 (time 1753440615.76 total 159.44 last 90.20) - open/close 16756 (time 1753440625.76 total 169.44 last 91.99) - open/close 17918 (time 1753440635.76 total 179.45 last 116.12) - open/close 19221 (time 1753440645.77 total 189.45 last 130.29) - open/close 20000 (time 1753440652.74 total 196.43 last 111.61) - open/close 21188 (time 1753440662.75 total 206.43 last 118.77) - open/close 22195 (time 1753440672.75 total 216.43 last 100.67) - open/close 23721 (time 1753440682.76 total 226.44 last 152.49) - open/close 25126 (time 1753440692.76 total 236.44 last 140.50) - open/close 26349 (time 1753440702.76 total 246.44 last 122.29) - open/close 27595 (time 1753440712.76 total 256.45 last 124.54) - open/close 28848 (time 1753440722.77 total 266.45 last 125.20) - open/close 30000 (time 1753440730.25 total 273.93 last 154.09) - open/close 31313 (time 1753440740.25 total 283.93 last 131.26) - open/close 32911 (time 1753440750.25 total 293.94 last 159.74) - open/close 34580 (time 1753440760.26 total 303.94 last 166.83) - open/close 36714 (time 1753440770.26 total 313.94 last 213.36) - open/close 38295 (time 1753440780.26 total 323.95 last 158.08) - open/close 39864 (time 1753440790.26 total 333.95 last 156.89) - open/close 41572 (time 1753440800.27 total 343.95 last 170.74) - open/close 43520 (time 1753440810.27 total 353.96 last 194.66) - open/close 45438 (time 1753440820.27 total 363.96 last 191.79) - open/close 47214 (time 1753440830.28 total 373.96 last 177.48) - open/close 48741 (time 1753440840.28 total 383.96 last 152.69) - open/close 50000 (time 1753440848.61 total 392.29 last 151.24) - open/close 51644 (time 1753440858.61 total 402.29 last 164.37) - open/close 53736 (time 1753440868.61 total 412.30 last 209.12) - open/close 55533 (time 1753440878.62 total 422.30 last 179.63) - open/close 57025 (time 1753440888.63 total 432.31 last 148.98) - open/close 58462 (time 1753440898.63 total 442.32 last 143.66) - open/close 59961 (time 1753440908.64 total 452.32 last 149.86) - open/close 61408 (time 1753440918.64 total 462.32 last 144.68) - open/close 62993 (time 1753440928.64 total 472.32 last 158.49) - open/close 64236 (time 1753440938.64 total 482.32 last 124.26) total: 64768 open/close in 487.05 seconds: 132.98 ops/second - /unlink 2558 (time 1753440955.48 total 10.00 last 255.77) - /unlink 5373 (time 1753440965.52 total 20.04 last 280.39) - /unlink 7969 (time 1753440975.52 total 30.04 last 259.54) - /unlink 9818 (time 1753440985.52 total 40.04 last 184.86) - /unlink 11992 (time 1753440995.53 total 50.05 last 217.28) - /unlink 13783 (time 1753441005.53 total 60.05 last 179.02) - /unlink 14977 (time 1753441015.54 total 70.06 last 119.34) - /unlink 15961 (time 1753441025.54 total 80.06 last 98.37) - /unlink 16532 (time 1753441035.56 total 90.08 last 56.98) - /unlink 17330 (time 1753441045.56 total 100.08 last 79.79) - /unlink 18434 (time 1753441055.57 total 110.09 last 110.35) - /unlink 19590 (time 1753441065.57 total 120.09 last 115.56) - /unlink 20000 (time 1753441068.86 total 123.38 last 124.70) - /unlink 21156 (time 1753441078.86 total 133.38 last 115.57) - /unlink 22551 (time 1753441088.87 total 143.39 last 139.39) - /unlink 23909 (time 1753441098.87 total 153.39 last 135.75) - /unlink 25264 (time 1753441108.88 total 163.41 last 135.32) - /unlink 26584 (time 1753441118.89 total 173.41 last 131.97) - /unlink 27953 (time 1753441128.89 total 183.41 last 136.88) - /unlink 29350 (time 1753441138.89 total 193.42 last 139.61) - /unlink 30000 (time 1753441143.77 total 198.29 last 133.33) - /unlink 31350 (time 1753441153.77 total 208.29 last 134.99) - /unlink 32634 (time 1753441163.78 total 218.30 last 128.29) - /unlink 33857 (time 1753441173.78 total 228.30 last 122.30) - /unlink 35117 (time 1753441183.78 total 238.30 last 125.96) - /unlink 36388 (time 1753441193.79 total 248.31 last 127.05) - /unlink 37695 (time 1753441203.79 total 258.31 last 130.69) - /unlink 39073 (time 1753441213.79 total 268.31 last 137.77) - /unlink 40000 (time 1753441220.27 total 274.79 last 143.04) - /unlink 41362 (time 1753441230.27 total 284.80 last 136.14) - /unlink 42659 (time 1753441240.27 total 294.80 last 129.69) - /unlink 43940 (time 1753441250.28 total 304.80 last 128.09) - /unlink 45230 (time 1753441260.28 total 314.80 last 128.95) - /unlink 46447 (time 1753441270.28 total 324.81 last 121.66) - /unlink 47531 (time 1753441280.29 total 334.82 last 108.28) - /unlink 48712 (time 1753441290.29 total 344.82 last 118.09) - /unlink 49994 (time 1753441300.30 total 354.82 last 128.15) - /unlink 51267 (time 1753441310.30 total 364.82 last 127.29) - /unlink 52576 (time 1753441320.30 total 374.83 last 130.83) - /unlink 53800 (time 1753441330.31 total 384.83 last 122.32) - /unlink 54925 (time 1753441340.32 total 394.84 last 112.44) - /unlink 56298 (time 1753441350.32 total 404.84 last 137.24) - /unlink 57690 (time 1753441360.32 total 414.85 last 139.16) - /unlink 59021 (time 1753441370.33 total 424.86 last 132.96) - /unlink 60000 (time 1753441378.35 total 432.87 last 122.14) - /unlink 61245 (time 1753441388.36 total 442.88 last 124.36) - /unlink 62502 (time 1753441398.37 total 452.89 last 125.59) - /unlink 63832 (time 1753441408.37 total 462.89 last 133.00) total: 64768 /unlink in 470.08 seconds: 137.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 815 (time 1753441450.51 total 10.00 last 81.47) - open/close 1623 (time 1753441460.52 total 20.01 last 80.77) - open/close 2515 (time 1753441470.53 total 30.02 last 89.06) - open/close 3465 (time 1753441480.54 total 40.03 last 94.97) - open/close 4348 (time 1753441490.54 total 50.03 last 88.27) - open/close 5276 (time 1753441500.54 total 60.03 last 92.76) - open/close 6460 (time 1753441510.55 total 70.04 last 118.39) - open/close 7599 (time 1753441520.56 total 80.05 last 113.77) - open/close 8565 (time 1753441530.57 total 90.06 last 96.51) - open/close 9461 (time 1753441540.57 total 100.06 last 89.53) - open/close 10000 (time 1753441545.53 total 105.02 last 108.67) - open/close 11117 (time 1753441555.55 total 115.04 last 111.56) - open/close 12035 (time 1753441565.56 total 125.05 last 91.71) - open/close 13117 (time 1753441575.56 total 135.05 last 108.17) - open/close 14108 (time 1753441585.57 total 145.06 last 99.03) - open/close 15045 (time 1753441595.57 total 155.06 last 93.70) - open/close 16365 (time 1753441605.57 total 165.06 last 131.93) - open/close 17680 (time 1753441615.57 total 175.06 last 131.47) - open/close 18981 (time 1753441625.58 total 185.06 last 130.09) - open/close 20000 (time 1753441633.54 total 193.03 last 127.89) - open/close 21384 (time 1753441643.55 total 203.04 last 138.36) - open/close 22761 (time 1753441653.55 total 213.04 last 137.69) - open/close 24073 (time 1753441663.55 total 223.04 last 131.19) - open/close 25279 (time 1753441673.55 total 233.04 last 120.56) - open/close 26513 (time 1753441683.56 total 243.04 last 123.33) - open/close 27867 (time 1753441693.56 total 253.05 last 135.34) - open/close 29414 (time 1753441703.56 total 263.05 last 154.67) - open/close 30000 (time 1753441707.60 total 267.09 last 145.07) - open/close 31406 (time 1753441717.61 total 277.09 last 140.54) - open/close 32707 (time 1753441727.61 total 287.10 last 130.06) - open/close 33932 (time 1753441737.61 total 297.10 last 122.48) - open/close 35257 (time 1753441747.61 total 307.10 last 132.45) - open/close 36530 (time 1753441757.62 total 317.11 last 127.21) - open/close 38013 (time 1753441767.62 total 327.11 last 148.25) - open/close 39412 (time 1753441777.63 total 337.11 last 139.87) - open/close 40000 (time 1753441781.24 total 340.73 last 162.53) - open/close 41422 (time 1753441791.25 total 350.74 last 142.13) - open/close 42526 (time 1753441801.25 total 360.74 last 110.37) - open/close 43707 (time 1753441811.25 total 370.74 last 118.06) - open/close 45089 (time 1753441821.26 total 380.74 last 138.17) - open/close 46549 (time 1753441831.26 total 390.75 last 145.95) - open/close 47966 (time 1753441841.26 total 400.75 last 141.65) - open/close 49126 (time 1753441851.27 total 410.76 last 115.92) - open/close 50000 (time 1753441858.89 total 418.37 last 114.76) - open/close 51549 (time 1753441868.89 total 428.38 last 154.86) - open/close 53064 (time 1753441878.89 total 438.38 last 151.49) - open/close 54816 (time 1753441888.89 total 448.38 last 175.20) - open/close 56482 (time 1753441898.89 total 458.38 last 166.59) - open/close 58257 (time 1753441908.89 total 468.38 last 177.49) - open/close 60000 (time 1753441918.51 total 478.00 last 181.13) - open/close 62053 (time 1753441928.51 total 488.00 last 205.29) - open/close 64174 (time 1753441938.52 total 498.01 last 212.00) total: 64768 open/close in 501.47 seconds: 129.16 ops/second - /unlink 2743 (time 1753441953.78 total 10.00 last 274.30) - /unlink 5385 (time 1753441963.78 total 20.00 last 264.11) - /unlink 8256 (time 1753441973.78 total 30.01 last 287.04) - /unlink 10000 (time 1753441980.39 total 36.61 last 264.03) - /unlink 13115 (time 1753441990.39 total 46.61 last 311.44) - /unlink 16311 (time 1753442000.39 total 56.61 last 319.55) - /unlink 19030 (time 1753442010.39 total 66.62 last 271.81) - /unlink 20000 (time 1753442014.29 total 70.51 last 249.22) - /unlink 22391 (time 1753442024.29 total 80.51 last 239.10) - /unlink 25171 (time 1753442034.29 total 90.51 last 277.96) - /unlink 27173 (time 1753442044.29 total 100.52 last 200.09) - /unlink 29241 (time 1753442054.29 total 110.52 last 206.77) - /unlink 30000 (time 1753442057.72 total 113.94 last 221.71) - /unlink 32431 (time 1753442067.72 total 123.94 last 243.04) - /unlink 34874 (time 1753442077.72 total 133.94 last 244.29) - /unlink 37387 (time 1753442087.72 total 143.95 last 251.20) - /unlink 39721 (time 1753442097.73 total 153.95 last 233.34) - /unlink 43038 (time 1753442107.73 total 163.95 last 331.62) - /unlink 45985 (time 1753442117.73 total 173.95 last 294.69) - /unlink 49263 (time 1753442127.73 total 183.95 last 327.77) - /unlink 50000 (time 1753442130.34 total 186.56 last 282.62) - /unlink 52038 (time 1753442140.34 total 196.56 last 203.77) - /unlink 54208 (time 1753442150.34 total 206.56 last 217.00) - /unlink 56999 (time 1753442160.34 total 216.57 last 279.02) - /unlink 59423 (time 1753442170.35 total 226.57 last 242.32) - /unlink 60000 (time 1753442172.60 total 228.83 last 255.70) - /unlink 62616 (time 1753442182.61 total 238.83 last 261.50) total: 64768 /unlink in 245.95 seconds: 263.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2053 (time 1753442220.93 total 10.00 last 205.23) - open/close 3806 (time 1753442230.93 total 20.01 last 175.26) - open/close 5600 (time 1753442240.93 total 30.01 last 179.37) - open/close 7413 (time 1753442250.94 total 40.01 last 181.23) - open/close 9261 (time 1753442260.94 total 50.01 last 184.73) - open/close 10000 (time 1753442265.00 total 54.08 last 181.95) - open/close 12028 (time 1753442275.01 total 64.08 last 202.73) - open/close 14523 (time 1753442285.01 total 74.08 last 249.39) - open/close 16694 (time 1753442295.01 total 84.09 last 217.00) - open/close 18983 (time 1753442305.02 total 94.09 last 228.85) - open/close 20000 (time 1753442308.59 total 97.66 last 284.96) - open/close 22394 (time 1753442318.59 total 107.66 last 239.30) - open/close 24851 (time 1753442328.59 total 117.67 last 245.59) - open/close 26494 (time 1753442338.60 total 127.68 last 164.18) - open/close 28795 (time 1753442348.60 total 137.68 last 230.03) - open/close 30000 (time 1753442352.81 total 141.89 last 286.44) - open/close 32611 (time 1753442362.81 total 151.89 last 261.03) - open/close 35232 (time 1753442372.82 total 161.89 last 262.05) - open/close 37880 (time 1753442382.82 total 171.89 last 264.77) - open/close 40000 (time 1753442391.82 total 180.89 last 235.59) - open/close 42856 (time 1753442401.82 total 190.89 last 285.54) - open/close 45123 (time 1753442411.82 total 200.89 last 226.69) - open/close 47379 (time 1753442421.82 total 210.89 last 225.60) - open/close 49447 (time 1753442431.82 total 220.89 last 206.79) - open/close 50000 (time 1753442434.35 total 223.43 last 218.36) - open/close 52714 (time 1753442444.36 total 233.43 last 271.27) - open/close 55399 (time 1753442454.36 total 243.43 last 268.41) - open/close 57733 (time 1753442464.36 total 253.44 last 233.37) - open/close 60000 (time 1753442472.85 total 261.93 last 267.00) - open/close 63723 (time 1753442482.86 total 271.93 last 372.17) total: 64768 open/close in 275.78 seconds: 234.86 ops/second - /unlink 4698 (time 1753442498.29 total 10.00 last 469.71) - /unlink 8683 (time 1753442508.29 total 20.00 last 398.48) - /unlink 10000 (time 1753442511.09 total 22.81 last 468.93) - /unlink 14486 (time 1753442521.10 total 32.81 last 448.55) - /unlink 18871 (time 1753442531.10 total 42.81 last 438.42) - /unlink 20000 (time 1753442533.16 total 44.88 last 546.44) - /unlink 25582 (time 1753442543.16 total 54.88 last 558.13) - /unlink 30000 (time 1753442552.30 total 64.02 last 483.63) - /unlink 35365 (time 1753442562.30 total 74.02 last 536.41) - /unlink 40000 (time 1753442569.24 total 80.96 last 668.16) - /unlink 45562 (time 1753442579.24 total 90.96 last 556.18) - /unlink 49993 (time 1753442589.24 total 100.96 last 443.06) - /unlink 54654 (time 1753442599.24 total 110.96 last 466.08) - /unlink 60000 (time 1753442607.15 total 118.87 last 675.66) total: 64768 /unlink in 125.85 seconds: 514.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3392 (time 1753442642.91 total 10.01 last 339.00) - open/close 6607 (time 1753442652.91 total 20.01 last 321.49) - open/close 9543 (time 1753442662.91 total 30.01 last 293.60) - open/close 13956 (time 1753442672.91 total 40.01 last 441.25) - open/close 17535 (time 1753442682.91 total 50.01 last 357.81) - open/close 20000 (time 1753442692.26 total 59.36 last 263.74) - open/close 23460 (time 1753442702.26 total 69.36 last 345.94) - open/close 26418 (time 1753442712.26 total 79.36 last 295.77) - open/close 30000 (time 1753442721.68 total 88.78 last 380.22) - open/close 33013 (time 1753442731.68 total 98.78 last 301.30) - open/close 36220 (time 1753442741.68 total 108.78 last 320.66) - open/close 39984 (time 1753442751.68 total 118.78 last 376.34) - open/close 44226 (time 1753442761.74 total 128.84 last 421.65) - open/close 48520 (time 1753442771.74 total 138.84 last 429.39) - open/close 50000 (time 1753442774.98 total 142.08 last 458.01) - open/close 54362 (time 1753442784.98 total 152.08 last 436.12) - open/close 58280 (time 1753442794.98 total 162.08 last 391.77) - open/close 60000 (time 1753442799.19 total 166.29 last 408.80) - open/close 62522 (time 1753442809.19 total 176.29 last 252.12) total: 64768 open/close in 183.80 seconds: 352.38 ops/second - /unlink 8075 (time 1753442827.99 total 10.00 last 807.44) - /unlink 10000 (time 1753442830.76 total 12.77 last 696.16) - /unlink 17164 (time 1753442840.76 total 22.77 last 716.35) - /unlink 20000 (time 1753442844.81 total 26.82 last 699.16) - /unlink 26445 (time 1753442854.81 total 36.82 last 644.45) - /unlink 30000 (time 1753442860.59 total 42.60 last 615.56) - /unlink 36283 (time 1753442870.59 total 52.60 last 628.20) - /unlink 40000 (time 1753442876.21 total 58.22 last 661.55) - /unlink 46093 (time 1753442886.21 total 68.22 last 609.23) - /unlink 50000 (time 1753442891.69 total 73.70 last 713.27) - /unlink 56459 (time 1753442901.69 total 83.70 last 645.87) - /unlink 60000 (time 1753442907.91 total 89.92 last 569.11) total: 64768 /unlink in 97.65 seconds: 663.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3021 (time 1753442947.39 total 10.00 last 302.07) - open/close 5967 (time 1753442957.39 total 20.00 last 294.56) - open/close 8402 (time 1753442967.39 total 30.00 last 243.45) - open/close 10000 (time 1753442972.86 total 35.47 last 292.37) - open/close 12903 (time 1753442982.86 total 45.47 last 290.25) - open/close 15335 (time 1753442992.86 total 55.47 last 243.18) - open/close 18193 (time 1753443002.87 total 65.48 last 285.70) - open/close 20000 (time 1753443009.29 total 71.90 last 281.49) - open/close 22924 (time 1753443019.29 total 81.90 last 292.37) - open/close 26757 (time 1753443029.29 total 91.90 last 383.27) - open/close 29415 (time 1753443039.29 total 101.90 last 265.68) - open/close 30000 (time 1753443045.25 total 107.86 last 98.17) - open/close 31005 (time 1753443055.25 total 117.86 last 100.46) - open/close 32424 (time 1753443065.26 total 127.87 last 141.80) - open/close 33319 (time 1753443075.27 total 137.88 last 89.39) - open/close 34028 (time 1753443085.29 total 147.90 last 70.81) - open/close 34556 (time 1753443095.30 total 157.91 last 52.74) - open/close 35104 (time 1753443105.31 total 167.92 last 54.75) - open/close 35738 (time 1753443115.32 total 177.93 last 63.35) - open/close 36442 (time 1753443125.32 total 187.93 last 70.36) - open/close 37251 (time 1753443135.32 total 197.93 last 80.89) - open/close 38034 (time 1753443145.32 total 207.93 last 78.30) - open/close 38886 (time 1753443155.33 total 217.94 last 85.18) - open/close 39737 (time 1753443165.33 total 227.94 last 85.08) - open/close 40000 (time 1753443168.60 total 231.21 last 80.37) - open/close 40883 (time 1753443178.61 total 241.22 last 88.25) - open/close 41730 (time 1753443188.61 total 251.22 last 84.69) - open/close 42581 (time 1753443198.61 total 261.22 last 85.05) - open/close 43425 (time 1753443208.62 total 271.23 last 84.35) - open/close 44268 (time 1753443218.63 total 281.24 last 84.17) - open/close 45075 (time 1753443228.64 total 291.25 last 80.70) - open/close 45878 (time 1753443238.65 total 301.26 last 80.18) - open/close 46740 (time 1753443248.66 total 311.27 last 86.14) - open/close 47676 (time 1753443258.66 total 321.27 last 93.57) - open/close 48484 (time 1753443268.67 total 331.28 last 80.73) - open/close 49332 (time 1753443278.68 total 341.29 last 84.74) - open/close 50000 (time 1753443286.75 total 349.36 last 82.71) - open/close 50799 (time 1753443296.76 total 359.37 last 79.86) - open/close 51612 (time 1753443306.79 total 369.40 last 81.08) - open/close 52364 (time 1753443316.79 total 379.40 last 75.20) - open/close 53065 (time 1753443326.79 total 389.40 last 70.05) - open/close 53784 (time 1753443336.80 total 399.41 last 71.88) - open/close 54615 (time 1753443346.80 total 409.41 last 83.03) - open/close 55371 (time 1753443356.81 total 419.42 last 75.57) - open/close 56122 (time 1753443366.81 total 429.42 last 75.08) - open/close 56909 (time 1753443376.82 total 439.43 last 78.65) - open/close 57712 (time 1753443386.83 total 449.44 last 80.18) - open/close 58505 (time 1753443396.84 total 459.45 last 79.26) - open/close 59108 (time 1753443406.85 total 469.46 last 60.23) - open/close 59696 (time 1753443416.85 total 479.46 last 58.76) - open/close 60000 (time 1753443422.80 total 485.41 last 51.10) - open/close 60559 (time 1753443432.81 total 495.42 last 55.87) - open/close 61230 (time 1753443442.82 total 505.43 last 67.05) - open/close 61941 (time 1753443452.82 total 515.43 last 71.10) - open/close 62701 (time 1753443462.83 total 525.44 last 75.93) - open/close 63423 (time 1753443472.83 total 535.44 last 72.17) - open/close 64150 (time 1753443482.84 total 545.45 last 72.67) total: 64768 open/close in 553.70 seconds: 116.97 ops/second - /unlink 1317 (time 1753443504.16 total 10.00 last 131.64) - /unlink 2601 (time 1753443514.17 total 20.02 last 128.24) - /unlink 3868 (time 1753443524.18 total 30.03 last 126.53) - /unlink 5135 (time 1753443534.18 total 40.03 last 126.65) - /unlink 6392 (time 1753443544.19 total 50.04 last 125.62) - /unlink 7615 (time 1753443554.20 total 60.05 last 122.16) - /unlink 8862 (time 1753443564.22 total 70.07 last 124.54) - /unlink 9965 (time 1753443574.22 total 80.07 last 110.26) - /unlink 10681 (time 1753443584.23 total 90.08 last 71.56) - /unlink 11400 (time 1753443594.23 total 100.08 last 71.87) - /unlink 12125 (time 1753443604.23 total 110.08 last 72.49) - /unlink 12795 (time 1753443614.23 total 120.08 last 66.99) - /unlink 13507 (time 1753443624.25 total 130.10 last 71.09) - /unlink 14201 (time 1753443634.27 total 140.12 last 69.28) - /unlink 14970 (time 1753443644.27 total 150.12 last 76.86) - /unlink 15681 (time 1753443654.28 total 160.12 last 71.07) - /unlink 16371 (time 1753443664.28 total 170.13 last 68.99) - /unlink 17087 (time 1753443674.28 total 180.13 last 71.59) - /unlink 17824 (time 1753443684.28 total 190.13 last 73.67) - /unlink 18561 (time 1753443694.29 total 200.14 last 73.65) - /unlink 19328 (time 1753443704.30 total 210.14 last 76.65) - /unlink 20000 (time 1753443714.04 total 219.89 last 68.97) - /unlink 20721 (time 1753443724.04 total 229.89 last 72.05) - /unlink 21438 (time 1753443734.05 total 239.90 last 71.68) - /unlink 22139 (time 1753443744.05 total 249.90 last 70.08) - /unlink 22910 (time 1753443754.06 total 259.90 last 77.06) - /unlink 23650 (time 1753443764.07 total 269.91 last 73.93) - /unlink 24370 (time 1753443774.08 total 279.92 last 71.92) - /unlink 25122 (time 1753443784.09 total 289.94 last 75.11) - /unlink 25879 (time 1753443794.09 total 299.94 last 75.69) - /unlink 26612 (time 1753443804.10 total 309.94 last 73.26) - /unlink 27362 (time 1753443814.10 total 319.95 last 74.97) - /unlink 28163 (time 1753443824.11 total 329.96 last 80.03) - /unlink 28938 (time 1753443834.15 total 340.00 last 77.16) - /unlink 29751 (time 1753443844.15 total 350.00 last 81.28) - /unlink 30000 (time 1753443847.73 total 353.58 last 69.53) - /unlink 30786 (time 1753443857.73 total 363.58 last 78.60) - /unlink 31498 (time 1753443867.74 total 373.58 last 71.19) - /unlink 32239 (time 1753443877.74 total 383.58 last 74.10) - /unlink 33040 (time 1753443887.75 total 393.59 last 80.03) - /unlink 33785 (time 1753443897.75 total 403.60 last 74.46) - /unlink 34534 (time 1753443907.76 total 413.61 last 74.85) - /unlink 35277 (time 1753443917.76 total 423.61 last 74.29) - /unlink 36022 (time 1753443927.76 total 433.61 last 74.47) - /unlink 36766 (time 1753443937.77 total 443.62 last 74.37) - /unlink 37505 (time 1753443947.77 total 453.62 last 73.86) - /unlink 38196 (time 1753443957.78 total 463.63 last 69.07) - /unlink 38898 (time 1753443967.79 total 473.63 last 70.13) - /unlink 39635 (time 1753443977.79 total 483.64 last 73.66) - /unlink 40000 (time 1753443982.40 total 488.25 last 79.18) - /unlink 40741 (time 1753443992.41 total 498.26 last 74.02) - /unlink 41453 (time 1753444002.42 total 508.27 last 71.11) - /unlink 42172 (time 1753444012.43 total 518.28 last 71.87) - /unlink 42895 (time 1753444022.43 total 528.28 last 72.28) - /unlink 43613 (time 1753444032.44 total 538.29 last 71.76) - /unlink 44359 (time 1753444042.44 total 548.29 last 74.59) - /unlink 45122 (time 1753444052.44 total 558.29 last 76.25) - /unlink 45832 (time 1753444062.46 total 568.30 last 70.92) - /unlink 46505 (time 1753444072.48 total 578.33 last 67.13) - /unlink 47224 (time 1753444082.50 total 588.35 last 71.76) - /unlink 47977 (time 1753444092.52 total 598.37 last 75.17) - /unlink 48722 (time 1753444102.53 total 608.38 last 74.41) - /unlink 49437 (time 1753444112.53 total 618.38 last 71.48) - /unlink 50000 (time 1753444120.38 total 626.23 last 71.75) - /unlink 50753 (time 1753444130.39 total 636.24 last 75.24) - /unlink 51473 (time 1753444140.39 total 646.24 last 71.96) - /unlink 52144 (time 1753444150.41 total 656.26 last 66.96) - /unlink 52870 (time 1753444160.42 total 666.27 last 72.55) - /unlink 53628 (time 1753444170.42 total 676.27 last 75.77) - /unlink 54377 (time 1753444180.43 total 686.28 last 74.84) - /unlink 55158 (time 1753444190.44 total 696.28 last 78.08) - /unlink 55905 (time 1753444200.44 total 706.29 last 74.66) - /unlink 56663 (time 1753444210.45 total 716.30 last 75.74) - /unlink 57396 (time 1753444220.46 total 726.31 last 73.21) - /unlink 58163 (time 1753444230.47 total 736.32 last 76.64) - /unlink 58903 (time 1753444240.48 total 746.33 last 73.88) - /unlink 59672 (time 1753444250.49 total 756.34 last 76.87) - /unlink 60000 (time 1753444254.94 total 760.79 last 73.62) - /unlink 60728 (time 1753444264.96 total 770.81 last 72.71) - /unlink 61420 (time 1753444274.96 total 780.81 last 69.16) - /unlink 62180 (time 1753444284.97 total 790.81 last 75.97) - /unlink 62870 (time 1753444294.99 total 800.83 last 68.86) - /unlink 63641 (time 1753444304.99 total 810.83 last 77.10) - /unlink 64399 (time 1753444315.00 total 820.85 last 75.71) total: 64768 /unlink in 825.68 seconds: 78.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 PASS 106 (3987s) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 oleg137-server: oleg137-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4145 sec ========== 07:53:51 (1753444431) === conf-sanity: start cleanup 07:53:52 (1753444432) === === conf-sanity: finish cleanup 07:53:55 (1753444435) ===