-----============= acceptance-small: conf-sanity ============----- Fri Oct 31 22:07:18 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg408-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg408-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg408-server oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg408-server: oleg408-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg408-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:08:51 (1761962931) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2690 (time 1761962978.47 total 10.00 last 268.94) - open/close 5276 (time 1761962988.48 total 20.00 last 258.57) - open/close 7807 (time 1761962998.48 total 30.01 last 253.04) - open/close 10000 (time 1761963006.94 total 38.47 last 259.05) - open/close 11979 (time 1761963016.94 total 48.47 last 197.90) - open/close 14469 (time 1761963026.95 total 58.47 last 248.97) - open/close 16688 (time 1761963036.95 total 68.48 last 221.83) - open/close 18985 (time 1761963046.95 total 78.48 last 229.67) - open/close 20000 (time 1761963050.80 total 82.33 last 263.37) - open/close 23122 (time 1761963060.81 total 92.33 last 312.15) - open/close 26198 (time 1761963070.81 total 102.33 last 307.60) - open/close 30000 (time 1761963080.61 total 112.14 last 387.79) - open/close 33644 (time 1761963090.61 total 122.14 last 364.37) - open/close 37607 (time 1761963100.61 total 132.14 last 396.23) - open/close 40000 (time 1761963107.23 total 138.76 last 361.49) - open/close 43210 (time 1761963117.23 total 148.76 last 320.97) - open/close 46565 (time 1761963127.23 total 158.76 last 335.47) - open/close 49933 (time 1761963137.24 total 168.76 last 336.72) - open/close 53608 (time 1761963147.24 total 178.76 last 367.46) - open/close 57789 (time 1761963157.24 total 188.77 last 418.03) - open/close 60000 (time 1761963162.91 total 194.44 last 389.89) - open/close 63572 (time 1761963172.91 total 204.44 last 357.15) total: 64768 open/close in 207.32 seconds: 312.40 ops/second - /unlink 7148 (time 1761963187.08 total 10.00 last 714.71) - /unlink 10000 (time 1761963190.62 total 13.54 last 806.47) - /unlink 17015 (time 1761963200.62 total 23.54 last 701.41) - /unlink 20000 (time 1761963205.31 total 28.23 last 636.50) - /unlink 27041 (time 1761963215.31 total 38.23 last 704.06) - /unlink 30000 (time 1761963219.42 total 42.34 last 719.97) - /unlink 37238 (time 1761963229.42 total 52.34 last 723.78) - /unlink 40000 (time 1761963233.09 total 56.02 last 751.03) - /unlink 46877 (time 1761963243.09 total 66.02 last 687.70) - /unlink 50000 (time 1761963247.60 total 70.52 last 692.79) - /unlink 56332 (time 1761963257.60 total 80.53 last 633.18) - /unlink 60000 (time 1761963263.18 total 86.10 last 658.22) total: 64768 /unlink in 92.52 seconds: 700.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4365 (time 1761963299.01 total 10.00 last 436.41) - open/close 9233 (time 1761963309.01 total 20.00 last 486.71) - open/close 13800 (time 1761963319.02 total 30.00 last 456.67) - open/close 18850 (time 1761963329.02 total 40.01 last 504.97) - open/close 20000 (time 1761963331.38 total 42.37 last 486.92) - open/close 24916 (time 1761963341.38 total 52.37 last 491.50) - open/close 29764 (time 1761963351.38 total 62.37 last 484.74) - open/close 34136 (time 1761963361.38 total 72.37 last 437.14) - open/close 38690 (time 1761963371.38 total 82.37 last 455.39) - open/close 40000 (time 1761963374.39 total 85.38 last 435.86) - open/close 44410 (time 1761963384.39 total 95.38 last 440.89) - open/close 48716 (time 1761963394.39 total 105.38 last 430.56) - open/close 50000 (time 1761963397.21 total 108.20 last 455.23) - open/close 54666 (time 1761963407.28 total 118.27 last 463.41) - open/close 59488 (time 1761963417.28 total 128.27 last 482.13) - open/close 64236 (time 1761963427.28 total 138.27 last 474.73) total: 64768 open/close in 139.47 seconds: 464.39 ops/second - /unlink 8493 (time 1761963439.76 total 10.00 last 849.28) - /unlink 16300 (time 1761963449.76 total 20.00 last 780.40) - /unlink 20000 (time 1761963453.90 total 24.15 last 893.36) - /unlink 26953 (time 1761963463.90 total 34.15 last 695.27) - /unlink 30000 (time 1761963468.14 total 38.39 last 718.35) - /unlink 37796 (time 1761963478.14 total 48.39 last 779.59) - /unlink 40000 (time 1761963481.17 total 51.42 last 727.44) - /unlink 48169 (time 1761963491.17 total 61.42 last 816.85) - /unlink 56936 (time 1761963501.18 total 71.42 last 876.64) - /unlink 60000 (time 1761963504.32 total 74.56 last 975.71) total: 64768 /unlink in 80.21 seconds: 807.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4329 (time 1761963544.45 total 10.00 last 432.82) - open/close 8866 (time 1761963554.45 total 20.00 last 453.67) - open/close 10000 (time 1761963557.20 total 22.75 last 412.70) - open/close 14871 (time 1761963567.20 total 32.75 last 487.09) - open/close 19789 (time 1761963577.20 total 42.75 last 491.78) - open/close 24946 (time 1761963587.20 total 52.75 last 515.63) - open/close 30000 (time 1761963597.15 total 62.70 last 508.10) - open/close 34717 (time 1761963607.15 total 72.70 last 471.53) - open/close 39579 (time 1761963617.15 total 82.70 last 486.16) - open/close 44456 (time 1761963627.15 total 92.70 last 487.68) - open/close 49109 (time 1761963637.16 total 102.71 last 465.21) - open/close 53901 (time 1761963647.16 total 112.71 last 479.16) - open/close 58791 (time 1761963657.23 total 122.78 last 485.60) - open/close 60000 (time 1761963659.66 total 125.21 last 496.53) total: 64768 open/close in 134.86 seconds: 480.25 ops/second - /unlink 8478 (time 1761963680.55 total 10.00 last 847.78) - /unlink 16866 (time 1761963690.56 total 20.00 last 838.75) - /unlink 20000 (time 1761963694.10 total 23.55 last 884.06) - /unlink 28776 (time 1761963704.10 total 33.55 last 877.55) - /unlink 36414 (time 1761963714.10 total 43.55 last 763.56) - /unlink 40000 (time 1761963718.89 total 48.33 last 749.69) - /unlink 47204 (time 1761963728.89 total 58.33 last 720.39) - /unlink 50000 (time 1761963732.93 total 62.37 last 692.38) - /unlink 57371 (time 1761963742.93 total 72.37 last 737.09) - /unlink 60000 (time 1761963746.64 total 76.08 last 708.55) total: 64768 /unlink in 82.31 seconds: 786.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4678 (time 1761963786.30 total 10.00 last 467.75) - open/close 9252 (time 1761963796.30 total 20.00 last 457.36) - open/close 13952 (time 1761963806.30 total 30.00 last 469.95) - open/close 18681 (time 1761963816.30 total 40.00 last 472.82) - open/close 20000 (time 1761963819.14 total 42.84 last 465.68) - open/close 24663 (time 1761963829.14 total 52.84 last 466.27) - open/close 29259 (time 1761963839.14 total 62.84 last 459.58) - open/close 33825 (time 1761963849.14 total 72.84 last 456.54) - open/close 38182 (time 1761963859.14 total 82.84 last 435.63) - open/close 40000 (time 1761963863.46 total 87.17 last 420.31) - open/close 44498 (time 1761963873.46 total 97.17 last 449.80) - open/close 48950 (time 1761963883.47 total 107.17 last 445.19) - open/close 50000 (time 1761963885.89 total 109.59 last 432.39) - open/close 54439 (time 1761963895.89 total 119.60 last 443.84) - open/close 58573 (time 1761963905.90 total 129.60 last 413.36) - open/close 60000 (time 1761963909.24 total 132.94 last 426.48) - open/close 64398 (time 1761963919.24 total 142.95 last 439.70) total: 64768 open/close in 143.93 seconds: 450.01 ops/second - /unlink 7134 (time 1761963931.61 total 10.00 last 713.37) - /unlink 10000 (time 1761963936.05 total 14.44 last 645.88) - /unlink 17126 (time 1761963946.16 total 24.55 last 704.71) - /unlink 20000 (time 1761963950.55 total 28.94 last 654.56) - /unlink 27328 (time 1761963960.56 total 38.94 last 732.73) - /unlink 30000 (time 1761963964.70 total 43.09 last 644.12) - /unlink 36629 (time 1761963974.70 total 53.09 last 662.87) - /unlink 40000 (time 1761963979.54 total 57.93 last 696.87) - /unlink 46145 (time 1761963989.54 total 67.93 last 614.48) - /unlink 50000 (time 1761963995.01 total 73.40 last 704.92) - /unlink 57383 (time 1761964005.01 total 83.40 last 738.26) - /unlink 60000 (time 1761964008.73 total 87.11 last 704.33) total: 64768 /unlink in 93.48 seconds: 692.87 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4791 (time 1761964046.00 total 10.00 last 479.05) - open/close 9429 (time 1761964056.00 total 20.00 last 463.72) - open/close 14006 (time 1761964066.01 total 30.00 last 457.67) - open/close 18685 (time 1761964076.01 total 40.00 last 467.85) - open/close 20000 (time 1761964078.88 total 42.88 last 457.48) - open/close 24424 (time 1761964088.88 total 52.88 last 442.39) - open/close 29065 (time 1761964098.88 total 62.88 last 464.01) - open/close 33576 (time 1761964108.88 total 72.88 last 451.08) - open/close 38011 (time 1761964118.88 total 82.88 last 443.50) - open/close 40000 (time 1761964123.53 total 87.53 last 427.61) - open/close 44491 (time 1761964133.54 total 97.53 last 449.05) - open/close 49128 (time 1761964143.54 total 107.54 last 463.60) - open/close 53725 (time 1761964153.54 total 117.54 last 459.65) - open/close 58149 (time 1761964163.54 total 127.54 last 442.36) - open/close 60000 (time 1761964168.02 total 132.02 last 413.30) - open/close 64364 (time 1761964178.02 total 142.02 last 436.34) total: 64768 open/close in 142.89 seconds: 453.26 ops/second - /unlink 7683 (time 1761964190.18 total 10.00 last 768.26) - /unlink 10000 (time 1761964193.51 total 13.32 last 697.04) - /unlink 17386 (time 1761964203.51 total 23.32 last 738.58) - /unlink 20000 (time 1761964206.41 total 26.22 last 901.37) - /unlink 27617 (time 1761964216.41 total 36.23 last 761.63) - /unlink 30000 (time 1761964219.43 total 39.25 last 788.55) - /unlink 37293 (time 1761964229.43 total 49.25 last 729.27) - /unlink 40000 (time 1761964234.17 total 53.99 last 571.10) - /unlink 46781 (time 1761964244.17 total 63.99 last 678.03) - /unlink 50000 (time 1761964248.03 total 67.85 last 834.67) - /unlink 57513 (time 1761964258.03 total 77.85 last 751.30) - /unlink 60000 (time 1761964261.79 total 81.61 last 660.87) total: 64768 /unlink in 88.38 seconds: 732.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 PASS 106 (1373s) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1491 sec ========== 22:32:09 (1761964329) === conf-sanity: start cleanup 22:32:10 (1761964330) === === conf-sanity: finish cleanup 22:32:11 (1761964331) ===