-----============= acceptance-small: conf-sanity ============----- Fri Oct 10 19:07:51 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg156-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg156-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg156-server oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg156-server: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg156-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:08:55 (1760137735) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg156-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg156-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg156-server: oleg156-server.virtnet: executing set_default_debug -1 all pdsh@oleg156-client: oleg156-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg156-client.virtnet: -o user_xattr,flock 192.168.201.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4564 (time 1760137779.94 total 10.00 last 456.35) - open/close 9004 (time 1760137789.95 total 20.00 last 443.95) - open/close 10000 (time 1760137792.19 total 22.25 last 443.51) - open/close 14152 (time 1760137802.19 total 32.25 last 415.20) - open/close 18619 (time 1760137812.19 total 42.25 last 446.66) - open/close 20000 (time 1760137815.25 total 45.30 last 452.33) - open/close 24604 (time 1760137825.25 total 55.30 last 460.37) - open/close 29121 (time 1760137835.25 total 65.30 last 451.61) - open/close 30000 (time 1760137837.28 total 67.34 last 432.56) - open/close 34195 (time 1760137847.28 total 77.34 last 419.47) - open/close 38577 (time 1760137857.28 total 87.34 last 438.18) - open/close 40000 (time 1760137860.40 total 90.46 last 456.42) - open/close 44393 (time 1760137870.40 total 100.46 last 439.23) - open/close 48633 (time 1760137880.40 total 110.46 last 423.97) - open/close 50000 (time 1760137883.84 total 113.89 last 398.03) - open/close 53430 (time 1760137893.84 total 123.89 last 342.99) - open/close 57204 (time 1760137903.84 total 133.89 last 377.33) - open/close 60000 (time 1760137911.06 total 141.12 last 387.05) - open/close 63765 (time 1760137921.06 total 151.12 last 376.47) total: 64768 open/close in 153.88 seconds: 420.90 ops/second - /unlink 7920 (time 1760137935.26 total 10.00 last 792.00) - /unlink 10000 (time 1760137937.62 total 12.35 last 883.36) - /unlink 17895 (time 1760137947.62 total 22.35 last 789.48) - /unlink 20000 (time 1760137950.54 total 25.28 last 719.19) - /unlink 27459 (time 1760137960.55 total 35.28 last 745.84) - /unlink 30000 (time 1760137963.78 total 38.52 last 784.82) - /unlink 37921 (time 1760137973.81 total 48.55 last 789.82) - /unlink 40000 (time 1760137976.59 total 51.33 last 747.53) - /unlink 47809 (time 1760137986.59 total 61.33 last 780.88) - /unlink 50000 (time 1760137989.95 total 64.68 last 653.24) - /unlink 57406 (time 1760137999.95 total 74.69 last 740.51) - /unlink 60000 (time 1760138003.59 total 78.33 last 712.44) total: 64768 /unlink in 84.35 seconds: 767.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4764 (time 1760138039.77 total 10.00 last 476.32) - open/close 9520 (time 1760138049.77 total 20.00 last 475.57) - open/close 14309 (time 1760138059.77 total 30.00 last 478.82) - open/close 19087 (time 1760138069.77 total 40.01 last 477.72) - open/close 23762 (time 1760138079.77 total 50.01 last 467.42) - open/close 28390 (time 1760138089.77 total 60.01 last 462.77) - open/close 30000 (time 1760138093.12 total 63.36 last 480.88) - open/close 34695 (time 1760138103.12 total 73.36 last 469.48) - open/close 39483 (time 1760138113.12 total 83.36 last 478.74) - open/close 44065 (time 1760138123.12 total 93.36 last 458.15) - open/close 48602 (time 1760138133.13 total 103.36 last 453.69) - open/close 50000 (time 1760138136.23 total 106.47 last 450.11) - open/close 54576 (time 1760138146.23 total 116.47 last 457.54) - open/close 59061 (time 1760138156.23 total 126.47 last 448.46) - open/close 60000 (time 1760138158.29 total 128.52 last 456.55) - open/close 64645 (time 1760138168.29 total 138.52 last 464.47) total: 64768 open/close in 138.89 seconds: 466.33 ops/second - /unlink 7883 (time 1760138180.05 total 10.00 last 788.28) - /unlink 10000 (time 1760138182.59 total 12.54 last 833.36) - /unlink 17702 (time 1760138192.59 total 22.54 last 770.13) - /unlink 20000 (time 1760138195.75 total 25.70 last 727.80) - /unlink 27893 (time 1760138205.75 total 35.70 last 789.23) - /unlink 30000 (time 1760138208.37 total 38.32 last 803.67) - /unlink 37863 (time 1760138218.37 total 48.32 last 786.25) - /unlink 40000 (time 1760138221.25 total 51.21 last 740.64) - /unlink 47862 (time 1760138231.25 total 61.21 last 786.14) - /unlink 50000 (time 1760138233.87 total 63.83 last 816.42) - /unlink 57949 (time 1760138243.87 total 73.83 last 794.87) - /unlink 60000 (time 1760138246.55 total 76.50 last 767.45) total: 64768 /unlink in 82.82 seconds: 782.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4364 (time 1760138284.30 total 10.00 last 436.29) - open/close 8648 (time 1760138294.30 total 20.00 last 428.39) - open/close 10000 (time 1760138297.14 total 22.84 last 475.77) - open/close 13751 (time 1760138307.14 total 32.85 last 375.08) - open/close 17406 (time 1760138317.14 total 42.85 last 365.39) - open/close 20000 (time 1760138324.37 total 50.08 last 358.91) - open/close 24071 (time 1760138334.37 total 60.08 last 406.95) - open/close 27869 (time 1760138344.37 total 70.08 last 379.75) - open/close 30000 (time 1760138350.09 total 75.80 last 372.48) - open/close 33223 (time 1760138360.10 total 85.80 last 322.20) - open/close 37498 (time 1760138370.10 total 95.81 last 427.42) - open/close 40000 (time 1760138376.00 total 101.71 last 423.99) - open/close 44281 (time 1760138386.00 total 111.71 last 428.06) - open/close 48209 (time 1760138396.00 total 121.71 last 392.77) - open/close 50000 (time 1760138400.04 total 125.75 last 443.43) - open/close 54247 (time 1760138410.04 total 135.75 last 424.69) - open/close 58474 (time 1760138420.04 total 145.75 last 422.62) - open/close 60000 (time 1760138423.95 total 149.65 last 391.08) - open/close 63722 (time 1760138433.95 total 159.65 last 372.18) total: 64768 open/close in 162.27 seconds: 399.15 ops/second - /unlink 7727 (time 1760138447.89 total 10.00 last 772.67) - /unlink 10000 (time 1760138451.12 total 13.23 last 702.99) - /unlink 17513 (time 1760138461.12 total 23.23 last 751.25) - /unlink 20000 (time 1760138464.14 total 26.25 last 823.83) - /unlink 25757 (time 1760138474.14 total 36.25 last 575.63) - /unlink 30000 (time 1760138481.63 total 43.74 last 566.49) - /unlink 35749 (time 1760138491.63 total 53.74 last 574.90) - /unlink 40000 (time 1760138499.94 total 62.06 last 511.34) - /unlink 46039 (time 1760138509.94 total 72.06 last 603.87) - /unlink 50000 (time 1760138517.28 total 79.39 last 540.00) - /unlink 56999 (time 1760138527.28 total 89.39 last 699.89) - /unlink 60000 (time 1760138532.06 total 94.17 last 627.72) total: 64768 /unlink in 102.94 seconds: 629.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3017 (time 1760138574.36 total 10.00 last 301.63) - open/close 5940 (time 1760138584.36 total 20.00 last 292.25) - open/close 9942 (time 1760138594.36 total 30.01 last 400.12) - open/close 12899 (time 1760138604.36 total 40.01 last 295.68) - open/close 14773 (time 1760138614.36 total 50.01 last 187.39) - open/close 19437 (time 1760138624.36 total 60.01 last 466.35) - open/close 24068 (time 1760138634.36 total 70.01 last 463.02) - open/close 28673 (time 1760138644.37 total 80.01 last 460.45) - open/close 30000 (time 1760138647.35 total 83.00 last 444.53) - open/close 34505 (time 1760138657.35 total 93.00 last 450.47) - open/close 38914 (time 1760138667.35 total 103.00 last 440.86) - open/close 40000 (time 1760138669.67 total 105.32 last 468.56) - open/close 44496 (time 1760138679.67 total 115.32 last 449.51) - open/close 48731 (time 1760138689.73 total 125.38 last 420.94) - open/close 50000 (time 1760138692.80 total 128.45 last 413.23) - open/close 54574 (time 1760138702.81 total 138.45 last 457.32) - open/close 59014 (time 1760138712.81 total 148.45 last 443.96) - open/close 60000 (time 1760138715.11 total 150.76 last 427.70) - open/close 64420 (time 1760138725.11 total 160.76 last 441.96) total: 64768 open/close in 161.60 seconds: 400.78 ops/second - /unlink 7725 (time 1760138737.32 total 10.00 last 772.48) - /unlink 10000 (time 1760138740.98 total 13.67 last 620.45) - /unlink 15393 (time 1760138750.98 total 23.67 last 539.29) - /unlink 20000 (time 1760138757.96 total 30.65 last 660.16) - /unlink 25011 (time 1760138767.96 total 40.65 last 501.06) - /unlink 30000 (time 1760138775.54 total 48.23 last 658.03) - /unlink 36732 (time 1760138785.55 total 58.23 last 673.06) - /unlink 40000 (time 1760138791.05 total 63.73 last 594.00) - /unlink 45170 (time 1760138801.05 total 73.73 last 516.89) - /unlink 49897 (time 1760138811.05 total 83.73 last 472.68) - /unlink 54400 (time 1760138821.05 total 93.74 last 450.27) - /unlink 59383 (time 1760138831.05 total 103.74 last 498.25) - /unlink 63929 (time 1760138841.05 total 113.74 last 454.50) total: 64768 /unlink in 114.87 seconds: 563.82 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2147 (time 1760138873.93 total 10.00 last 214.62) - open/close 4274 (time 1760138883.93 total 20.00 last 212.70) - open/close 7434 (time 1760138893.93 total 30.01 last 315.89) - open/close 9683 (time 1760138903.94 total 40.01 last 224.83) - open/close 12043 (time 1760138913.94 total 50.01 last 235.99) - open/close 15312 (time 1760138923.94 total 60.01 last 326.84) - open/close 19859 (time 1760138933.94 total 70.01 last 454.69) - open/close 23756 (time 1760138943.94 total 80.01 last 389.66) - open/close 26207 (time 1760138953.94 total 90.02 last 245.07) - open/close 28811 (time 1760138963.94 total 100.02 last 260.33) - open/close 30000 (time 1760138968.86 total 104.93 last 241.99) - open/close 32723 (time 1760138978.86 total 114.94 last 272.19) - open/close 35407 (time 1760138988.86 total 124.94 last 268.36) - open/close 37703 (time 1760138998.87 total 134.95 last 229.36) - open/close 40000 (time 1760139008.06 total 144.14 last 250.01) - open/close 42324 (time 1760139018.06 total 154.14 last 232.34) - open/close 44631 (time 1760139028.06 total 164.14 last 230.65) - open/close 46987 (time 1760139038.07 total 174.14 last 235.51) - open/close 49300 (time 1760139048.07 total 184.15 last 231.26) - open/close 52019 (time 1760139058.07 total 194.15 last 271.79) - open/close 54326 (time 1760139068.08 total 204.15 last 230.67) - open/close 58527 (time 1760139078.08 total 214.15 last 420.03) - open/close 60000 (time 1760139081.21 total 217.28 last 470.93) - open/close 64657 (time 1760139091.21 total 227.28 last 465.63) total: 64768 open/close in 227.63 seconds: 284.53 ops/second - /unlink 8408 (time 1760139102.74 total 10.00 last 840.71) - /unlink 17037 (time 1760139112.74 total 20.00 last 862.84) - /unlink 20000 (time 1760139116.29 total 23.55 last 834.59) - /unlink 27496 (time 1760139126.29 total 33.55 last 749.58) - /unlink 30000 (time 1760139129.65 total 36.91 last 745.76) - /unlink 38186 (time 1760139139.65 total 46.91 last 818.21) - /unlink 40000 (time 1760139142.06 total 49.32 last 754.28) - /unlink 48123 (time 1760139152.06 total 59.32 last 812.26) - /unlink 50000 (time 1760139154.14 total 61.40 last 903.52) - /unlink 56477 (time 1760139164.14 total 71.40 last 647.69) - /unlink 60000 (time 1760139171.57 total 78.83 last 474.19) total: 64768 /unlink in 87.39 seconds: 741.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg156-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg156-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg156-server stop mds service on oleg156-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg156-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing unload_modules_local oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-client: ssh exited with exit code 2 pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 PASS 106 (1487s) Stopping clients: oleg156-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg156-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg156-client: oleg156-server: ssh exited with exit code 2 oleg156-server: oleg156-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg156-server' oleg156-server: oleg156-server.virtnet: executing load_modules_local oleg156-server: Loading modules from /home/green/git/lustre-release/lustre oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg156-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg156-server: detected 4 online CPUs by sysfs oleg156-server: MODOPTS_LIBCFS= oleg156-server: Force libcfs to create 2 CPU partitions oleg156-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg156-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1583 sec ========== 19:34:15 (1760139255) === conf-sanity: start cleanup 19:34:16 (1760139256) === === conf-sanity: finish cleanup 19:34:18 (1760139258) ===