-----============= acceptance-small: conf-sanity ============----- Tue Aug 12 18:57:59 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg459-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg459-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg459-server oleg459-server: oleg459-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg459-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:59:08 (1755039548) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) oleg459-server: oleg459-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg459-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4071 (time 1755039596.90 total 10.00 last 407.09) - open/close 7240 (time 1755039606.90 total 20.00 last 316.87) - open/close 10000 (time 1755039614.49 total 27.59 last 363.77) - open/close 14127 (time 1755039624.49 total 37.59 last 412.62) - open/close 18732 (time 1755039634.49 total 47.59 last 460.47) - open/close 20000 (time 1755039637.87 total 50.97 last 375.21) - open/close 23657 (time 1755039647.87 total 60.97 last 365.59) - open/close 28016 (time 1755039657.87 total 70.98 last 435.81) - open/close 30000 (time 1755039663.08 total 76.18 last 380.97) - open/close 33512 (time 1755039673.08 total 86.18 last 351.15) - open/close 37128 (time 1755039683.08 total 96.19 last 361.56) - open/close 40000 (time 1755039691.05 total 104.15 last 360.43) - open/close 43471 (time 1755039701.05 total 114.16 last 347.05) - open/close 47521 (time 1755039711.05 total 124.16 last 404.96) - open/close 50000 (time 1755039717.00 total 130.10 last 417.17) - open/close 54193 (time 1755039727.00 total 140.10 last 419.28) - open/close 58092 (time 1755039737.00 total 150.10 last 389.82) - open/close 60000 (time 1755039741.36 total 154.46 last 438.00) - open/close 64340 (time 1755039751.36 total 164.46 last 433.97) total: 64768 open/close in 165.59 seconds: 391.14 ops/second - /unlink 7229 (time 1755039763.87 total 10.00 last 722.89) - /unlink 10000 (time 1755039767.37 total 13.50 last 791.08) - /unlink 16952 (time 1755039777.37 total 23.50 last 695.18) - /unlink 20000 (time 1755039781.82 total 27.95 last 685.36) - /unlink 26755 (time 1755039791.82 total 37.95 last 675.44) - /unlink 30000 (time 1755039796.10 total 42.23 last 758.36) - /unlink 36426 (time 1755039806.10 total 52.23 last 642.55) - /unlink 40000 (time 1755039812.17 total 58.30 last 589.29) - /unlink 46292 (time 1755039822.17 total 68.30 last 629.14) - /unlink 50000 (time 1755039827.88 total 74.01 last 648.98) - /unlink 57062 (time 1755039837.88 total 84.01 last 706.14) - /unlink 60000 (time 1755039842.07 total 88.20 last 701.84) total: 64768 /unlink in 94.46 seconds: 685.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4628 (time 1755039882.85 total 10.00 last 462.77) - open/close 8444 (time 1755039892.86 total 20.00 last 381.55) - open/close 10000 (time 1755039896.77 total 23.92 last 397.40) - open/close 14556 (time 1755039906.77 total 33.92 last 455.55) - open/close 19281 (time 1755039916.77 total 43.92 last 472.42) - open/close 23705 (time 1755039926.78 total 53.92 last 442.34) - open/close 28223 (time 1755039936.78 total 63.92 last 451.73) - open/close 30000 (time 1755039941.22 total 68.37 last 399.88) - open/close 34282 (time 1755039951.22 total 78.37 last 428.15) - open/close 38830 (time 1755039961.22 total 88.37 last 454.75) - open/close 40000 (time 1755039963.60 total 90.75 last 491.11) - open/close 44498 (time 1755039973.61 total 100.75 last 449.72) - open/close 48699 (time 1755039983.61 total 110.75 last 420.02) - open/close 50000 (time 1755039986.64 total 113.79 last 428.64) - open/close 54184 (time 1755039996.64 total 123.79 last 418.39) - open/close 58057 (time 1755040006.64 total 133.79 last 387.30) - open/close 60000 (time 1755040011.63 total 138.78 last 389.32) - open/close 64440 (time 1755040021.64 total 148.78 last 443.98) total: 64768 open/close in 149.71 seconds: 432.63 ops/second - /unlink 7327 (time 1755040033.84 total 10.00 last 732.36) - /unlink 10000 (time 1755040037.82 total 13.99 last 671.30) - /unlink 17183 (time 1755040047.82 total 23.99 last 718.25) - /unlink 20000 (time 1755040051.36 total 27.52 last 797.36) - /unlink 27441 (time 1755040061.36 total 37.52 last 744.08) - /unlink 30000 (time 1755040064.95 total 41.11 last 712.83) - /unlink 36863 (time 1755040074.95 total 51.11 last 686.29) - /unlink 40000 (time 1755040078.73 total 54.90 last 828.42) - /unlink 47597 (time 1755040088.73 total 64.90 last 759.67) - /unlink 50000 (time 1755040091.77 total 67.94 last 790.87) - /unlink 57595 (time 1755040101.77 total 77.94 last 759.48) - /unlink 60000 (time 1755040105.29 total 81.45 last 683.55) total: 64768 /unlink in 87.15 seconds: 743.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3713 (time 1755040193.26 total 10.00 last 371.23) - open/close 7067 (time 1755040203.26 total 20.00 last 335.36) - open/close 10000 (time 1755040212.11 total 28.85 last 331.58) - open/close 13547 (time 1755040222.11 total 38.85 last 354.67) - open/close 16576 (time 1755040232.11 total 48.85 last 302.80) - open/close 20000 (time 1755040241.91 total 58.65 last 349.38) - open/close 24272 (time 1755040251.91 total 68.65 last 427.19) - open/close 28317 (time 1755040261.91 total 78.65 last 404.50) - open/close 30000 (time 1755040266.90 total 83.64 last 337.69) - open/close 34043 (time 1755040276.90 total 93.64 last 404.24) - open/close 37764 (time 1755040286.90 total 103.64 last 372.05) - open/close 40000 (time 1755040292.74 total 109.48 last 382.72) - open/close 44168 (time 1755040302.74 total 119.48 last 416.80) - open/close 47507 (time 1755040312.74 total 129.48 last 333.88) - open/close 50000 (time 1755040318.58 total 135.32 last 427.27) - open/close 54192 (time 1755040328.58 total 145.32 last 419.19) - open/close 57449 (time 1755040338.58 total 155.32 last 325.65) - open/close 60000 (time 1755040347.85 total 164.59 last 275.25) - open/close 63939 (time 1755040357.85 total 174.59 last 393.83) total: 64768 open/close in 176.55 seconds: 366.86 ops/second - /unlink 5928 (time 1755040371.33 total 10.00 last 592.77) - /unlink 10000 (time 1755040377.11 total 15.78 last 704.32) - /unlink 15575 (time 1755040387.11 total 25.78 last 557.49) - /unlink 20000 (time 1755040394.39 total 33.06 last 607.72) - /unlink 26248 (time 1755040404.40 total 43.07 last 624.70) - /unlink 30000 (time 1755040409.59 total 48.26 last 721.95) - /unlink 36185 (time 1755040419.59 total 58.26 last 618.38) - /unlink 40000 (time 1755040426.83 total 65.50 last 527.39) - /unlink 46695 (time 1755040436.83 total 75.50 last 669.47) - /unlink 50000 (time 1755040443.06 total 81.73 last 529.98) - /unlink 55555 (time 1755040453.07 total 91.74 last 555.46) - /unlink 60000 (time 1755040461.48 total 100.15 last 528.11) total: 64768 /unlink in 107.46 seconds: 602.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3845 (time 1755040556.61 total 10.00 last 384.45) - open/close 8604 (time 1755040566.61 total 20.00 last 475.87) - open/close 10000 (time 1755040569.56 total 22.95 last 473.70) - open/close 13926 (time 1755040579.56 total 32.95 last 392.59) - open/close 17612 (time 1755040589.56 total 42.95 last 368.46) - open/close 20000 (time 1755040597.14 total 50.53 last 315.12) - open/close 24104 (time 1755040607.14 total 60.53 last 410.26) - open/close 27645 (time 1755040617.14 total 70.53 last 354.09) - open/close 30000 (time 1755040622.19 total 75.58 last 466.94) - open/close 34678 (time 1755040632.19 total 85.58 last 467.74) - open/close 39406 (time 1755040642.19 total 95.58 last 472.72) - open/close 44074 (time 1755040652.19 total 105.58 last 466.70) - open/close 48690 (time 1755040662.19 total 115.59 last 461.51) - open/close 50000 (time 1755040665.04 total 118.43 last 460.56) - open/close 54379 (time 1755040675.04 total 128.43 last 437.85) - open/close 56899 (time 1755040685.17 total 138.56 last 248.79) - open/close 60000 (time 1755040694.60 total 147.99 last 328.67) - open/close 62462 (time 1755040704.61 total 158.00 last 246.16) total: 64768 open/close in 164.42 seconds: 393.92 ops/second - /unlink 5143 (time 1755040722.34 total 10.00 last 514.27) - /unlink 10000 (time 1755040729.49 total 17.15 last 679.21) - /unlink 16571 (time 1755040739.49 total 27.15 last 657.07) - /unlink 20000 (time 1755040744.37 total 32.03 last 703.37) - /unlink 24904 (time 1755040754.37 total 42.03 last 490.40) - /unlink 30000 (time 1755040763.42 total 51.07 last 563.26) - /unlink 36583 (time 1755040773.42 total 61.07 last 658.26) - /unlink 40000 (time 1755040778.20 total 65.86 last 713.87) - /unlink 46335 (time 1755040788.20 total 75.86 last 633.43) - /unlink 50000 (time 1755040793.44 total 81.10 last 699.55) - /unlink 57490 (time 1755040803.44 total 91.10 last 748.94) - /unlink 60000 (time 1755040806.32 total 93.98 last 873.47) total: 64768 /unlink in 99.85 seconds: 648.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4577 (time 1755040858.43 total 10.00 last 457.65) - open/close 9434 (time 1755040868.44 total 20.00 last 485.61) - open/close 14203 (time 1755040878.44 total 30.00 last 476.88) - open/close 18909 (time 1755040888.44 total 40.00 last 470.56) - open/close 20000 (time 1755040890.73 total 42.30 last 475.21) - open/close 24694 (time 1755040900.73 total 52.30 last 469.32) - open/close 29417 (time 1755040910.73 total 62.30 last 472.28) - open/close 34206 (time 1755040920.74 total 72.30 last 478.90) - open/close 39021 (time 1755040930.74 total 82.30 last 481.48) - open/close 40000 (time 1755040932.74 total 84.31 last 487.32) - open/close 44629 (time 1755040942.74 total 94.31 last 462.87) - open/close 48946 (time 1755040952.75 total 104.31 last 431.60) - open/close 50000 (time 1755040955.79 total 107.35 last 346.90) - open/close 54232 (time 1755040965.79 total 117.35 last 423.16) - open/close 58690 (time 1755040975.79 total 127.36 last 445.73) - open/close 60000 (time 1755040978.68 total 130.25 last 452.69) - open/close 64540 (time 1755040988.68 total 140.25 last 453.91) total: 64768 open/close in 140.85 seconds: 459.84 ops/second - /unlink 6278 (time 1755041000.56 total 10.00 last 627.57) - /unlink 10000 (time 1755041006.25 total 15.69 last 654.77) - /unlink 15729 (time 1755041016.25 total 25.69 last 572.84) - /unlink 20000 (time 1755041021.66 total 31.10 last 788.69) - /unlink 27128 (time 1755041031.66 total 41.10 last 712.78) - /unlink 30000 (time 1755041035.35 total 44.79 last 780.09) - /unlink 37439 (time 1755041045.35 total 54.79 last 743.88) - /unlink 40000 (time 1755041049.04 total 58.48 last 693.22) - /unlink 47555 (time 1755041059.04 total 68.48 last 755.49) - /unlink 50000 (time 1755041062.19 total 71.63 last 775.67) - /unlink 57458 (time 1755041072.19 total 81.63 last 745.77) - /unlink 60000 (time 1755041075.73 total 85.17 last 719.55) total: 64768 /unlink in 91.76 seconds: 705.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local oleg459-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 PASS 106 (1623s) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: oleg459-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1714 sec ========== 19:26:34 (1755041194) === conf-sanity: start cleanup 19:26:35 (1755041195) === === conf-sanity: finish cleanup 19:26:36 (1755041196) ===