-----============= acceptance-small: conf-sanity ============----- Wed Aug 13 10:39:33 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg351-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg351-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg351-server oleg351-server: oleg351-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg351-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg351-server: oleg351-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg351-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:40:49 (1755096049) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) oleg351-server: oleg351-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg351-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4515 (time 1755096095.09 total 10.00 last 451.44) - open/close 7729 (time 1755096105.09 total 20.00 last 321.38) - open/close 10000 (time 1755096112.15 total 27.06 last 321.59) - open/close 13759 (time 1755096122.16 total 37.06 last 375.89) - open/close 17615 (time 1755096132.16 total 47.07 last 385.55) - open/close 20000 (time 1755096140.02 total 54.93 last 303.40) - open/close 23400 (time 1755096150.02 total 64.93 last 339.94) - open/close 26938 (time 1755096160.02 total 74.93 last 353.71) - open/close 30000 (time 1755096169.97 total 84.88 last 307.73) - open/close 33526 (time 1755096179.97 total 94.88 last 352.52) - open/close 37013 (time 1755096189.97 total 104.88 last 348.70) - open/close 40000 (time 1755096198.56 total 113.47 last 347.96) - open/close 43550 (time 1755096208.56 total 123.47 last 354.91) - open/close 46729 (time 1755096218.56 total 133.47 last 317.85) - open/close 49828 (time 1755096228.56 total 143.47 last 309.85) - open/close 53710 (time 1755096238.57 total 153.48 last 388.12) - open/close 56754 (time 1755096248.57 total 163.48 last 304.35) - open/close 59733 (time 1755096258.57 total 173.48 last 297.90) - open/close 63176 (time 1755096268.57 total 183.48 last 344.27) total: 64768 open/close in 188.32 seconds: 343.93 ops/second - /unlink 4441 (time 1755096284.96 total 10.00 last 444.04) - /unlink 9366 (time 1755096294.97 total 20.00 last 492.40) - /unlink 13953 (time 1755096304.97 total 30.01 last 458.62) - /unlink 17770 (time 1755096314.97 total 40.01 last 381.70) - /unlink 20000 (time 1755096320.49 total 45.52 last 404.02) - /unlink 24216 (time 1755096330.49 total 55.53 last 421.51) - /unlink 29209 (time 1755096340.49 total 65.53 last 499.15) - /unlink 34210 (time 1755096350.49 total 75.53 last 499.98) - /unlink 38759 (time 1755096360.50 total 85.53 last 454.82) - /unlink 40000 (time 1755096362.69 total 87.73 last 565.94) - /unlink 44119 (time 1755096372.69 total 97.73 last 411.83) - /unlink 48328 (time 1755096382.69 total 107.73 last 420.85) - /unlink 50000 (time 1755096386.93 total 111.96 last 394.81) - /unlink 54360 (time 1755096396.93 total 121.97 last 435.97) - /unlink 58714 (time 1755096406.93 total 131.97 last 435.34) - /unlink 60000 (time 1755096410.24 total 135.28 last 388.29) total: 64768 /unlink in 144.76 seconds: 447.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3428 (time 1755096448.11 total 10.00 last 342.75) - open/close 6969 (time 1755096458.11 total 20.00 last 354.02) - open/close 10000 (time 1755096466.03 total 27.93 last 382.50) - open/close 13553 (time 1755096476.03 total 37.93 last 355.23) - open/close 17685 (time 1755096486.04 total 47.93 last 413.08) - open/close 20000 (time 1755096492.08 total 53.98 last 382.81) - open/close 23537 (time 1755096502.09 total 63.98 last 353.68) - open/close 27190 (time 1755096512.09 total 73.98 last 365.29) - open/close 30000 (time 1755096520.25 total 82.14 last 344.31) - open/close 32803 (time 1755096530.25 total 92.14 last 280.23) - open/close 36298 (time 1755096540.25 total 102.15 last 349.42) - open/close 39426 (time 1755096550.28 total 112.18 last 311.85) - open/close 40000 (time 1755096552.51 total 114.40 last 257.85) - open/close 43362 (time 1755096562.51 total 124.40 last 336.19) - open/close 46861 (time 1755096572.51 total 134.41 last 349.80) - open/close 49927 (time 1755096582.51 total 144.41 last 306.50) - open/close 53481 (time 1755096592.52 total 154.41 last 355.33) - open/close 56718 (time 1755096602.52 total 164.41 last 323.67) - open/close 60000 (time 1755096612.12 total 174.02 last 341.76) - open/close 63889 (time 1755096622.12 total 184.02 last 388.85) total: 64768 open/close in 186.48 seconds: 347.31 ops/second - /unlink 5905 (time 1755096636.11 total 10.00 last 590.49) - /unlink 10000 (time 1755096642.51 total 16.40 last 640.19) - /unlink 16405 (time 1755096652.51 total 26.40 last 640.47) - /unlink 20000 (time 1755096657.39 total 31.27 last 737.06) - /unlink 26706 (time 1755096667.39 total 41.27 last 670.59) - /unlink 30000 (time 1755096672.37 total 46.26 last 661.25) - /unlink 36628 (time 1755096682.37 total 56.26 last 662.75) - /unlink 40000 (time 1755096686.93 total 60.82 last 739.52) - /unlink 47255 (time 1755096696.93 total 70.82 last 725.41) - /unlink 50000 (time 1755096701.42 total 75.30 last 611.88) - /unlink 56916 (time 1755096711.42 total 85.30 last 691.59) - /unlink 60000 (time 1755096715.81 total 89.70 last 701.38) total: 64768 /unlink in 96.67 seconds: 670.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4717 (time 1755096773.14 total 10.00 last 471.65) - open/close 9560 (time 1755096783.14 total 20.00 last 484.27) - open/close 14372 (time 1755096793.14 total 30.00 last 481.13) - open/close 19291 (time 1755096803.14 total 40.00 last 491.84) - open/close 24166 (time 1755096813.14 total 50.01 last 487.46) - open/close 28780 (time 1755096823.14 total 60.01 last 461.34) - open/close 30000 (time 1755096825.68 total 62.54 last 480.76) - open/close 34784 (time 1755096835.68 total 72.55 last 478.34) - open/close 39544 (time 1755096845.68 total 82.55 last 475.97) - open/close 44194 (time 1755096855.69 total 92.55 last 464.94) - open/close 48697 (time 1755096865.69 total 102.55 last 450.26) - open/close 50000 (time 1755096868.26 total 105.12 last 506.37) - open/close 54982 (time 1755096878.26 total 115.12 last 498.16) - open/close 59549 (time 1755096888.26 total 125.12 last 456.66) - open/close 64567 (time 1755096898.26 total 135.12 last 501.76) total: 64768 open/close in 135.52 seconds: 477.92 ops/second - /unlink 8327 (time 1755096910.01 total 10.00 last 832.69) - /unlink 10000 (time 1755096912.22 total 12.21 last 757.36) - /unlink 18167 (time 1755096922.22 total 22.21 last 816.60) - /unlink 20000 (time 1755096924.63 total 24.62 last 760.47) - /unlink 27953 (time 1755096934.63 total 34.62 last 795.23) - /unlink 30000 (time 1755096936.88 total 36.87 last 909.63) - /unlink 37002 (time 1755096946.88 total 46.87 last 700.15) - /unlink 40000 (time 1755096951.69 total 51.68 last 623.53) - /unlink 46203 (time 1755096961.69 total 61.68 last 620.26) - /unlink 50000 (time 1755096966.89 total 66.87 last 731.07) - /unlink 57825 (time 1755096976.89 total 76.88 last 782.44) - /unlink 60000 (time 1755096979.84 total 79.82 last 737.48) total: 64768 /unlink in 85.53 seconds: 757.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4440 (time 1755097015.37 total 10.00 last 443.93) - open/close 9028 (time 1755097025.37 total 20.00 last 458.72) - open/close 13763 (time 1755097035.37 total 30.00 last 473.46) - open/close 18818 (time 1755097045.37 total 40.00 last 505.48) - open/close 20000 (time 1755097047.69 total 42.33 last 508.22) - open/close 25096 (time 1755097057.70 total 52.33 last 509.52) - open/close 30000 (time 1755097067.15 total 61.79 last 518.72) - open/close 34436 (time 1755097077.15 total 71.79 last 443.55) - open/close 38845 (time 1755097087.15 total 81.79 last 440.85) - open/close 40000 (time 1755097089.56 total 84.19 last 479.97) - open/close 44965 (time 1755097099.56 total 94.20 last 496.47) - open/close 50000 (time 1755097109.39 total 104.02 last 512.28) - open/close 54503 (time 1755097119.39 total 114.02 last 450.28) - open/close 58615 (time 1755097129.39 total 124.03 last 411.13) - open/close 60000 (time 1755097132.47 total 127.10 last 450.27) total: 64768 open/close in 136.63 seconds: 474.05 ops/second - /unlink 8150 (time 1755097153.22 total 10.00 last 814.91) - /unlink 10000 (time 1755097155.45 total 12.23 last 828.59) - /unlink 18013 (time 1755097165.45 total 22.23 last 801.30) - /unlink 20000 (time 1755097167.97 total 24.75 last 789.69) - /unlink 28161 (time 1755097177.97 total 34.75 last 816.04) - /unlink 36322 (time 1755097187.97 total 44.75 last 816.02) - /unlink 40000 (time 1755097193.50 total 50.28 last 665.54) - /unlink 48036 (time 1755097203.50 total 60.28 last 803.57) - /unlink 50000 (time 1755097205.66 total 62.44 last 906.55) - /unlink 58204 (time 1755097215.66 total 72.45 last 820.37) - /unlink 60000 (time 1755097217.73 total 74.51 last 868.27) total: 64768 /unlink in 80.43 seconds: 805.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5077 (time 1755097294.29 total 10.00 last 507.69) - open/close 10000 (time 1755097303.90 total 19.61 last 512.11) - open/close 15231 (time 1755097313.90 total 29.61 last 523.03) - open/close 19907 (time 1755097323.91 total 39.62 last 467.51) - open/close 24930 (time 1755097333.91 total 49.62 last 502.26) - open/close 29423 (time 1755097343.91 total 59.62 last 449.29) - open/close 33198 (time 1755097353.91 total 69.62 last 377.49) - open/close 37353 (time 1755097363.91 total 79.62 last 415.43) - open/close 40000 (time 1755097369.54 total 85.25 last 469.88) - open/close 44484 (time 1755097379.54 total 95.25 last 448.38) - open/close 48609 (time 1755097389.54 total 105.26 last 412.41) - open/close 50000 (time 1755097393.55 total 109.27 last 346.89) - open/close 53460 (time 1755097403.56 total 119.27 last 345.98) - open/close 57316 (time 1755097413.56 total 129.27 last 385.59) - open/close 60000 (time 1755097419.89 total 135.60 last 423.61) - open/close 63907 (time 1755097429.89 total 145.60 last 390.69) total: 64768 open/close in 147.66 seconds: 438.62 ops/second - /unlink 7017 (time 1755097443.26 total 10.00 last 701.44) - /unlink 10000 (time 1755097447.65 total 14.40 last 679.22) - /unlink 17185 (time 1755097457.69 total 24.44 last 715.48) - /unlink 20000 (time 1755097461.59 total 28.33 last 722.86) - /unlink 27576 (time 1755097471.59 total 38.33 last 757.57) - /unlink 30000 (time 1755097475.09 total 41.84 last 691.67) - /unlink 37515 (time 1755097485.09 total 51.84 last 751.46) - /unlink 40000 (time 1755097488.14 total 54.88 last 815.47) - /unlink 47147 (time 1755097498.14 total 64.89 last 714.66) - /unlink 50000 (time 1755097503.03 total 69.77 last 583.53) - /unlink 56567 (time 1755097513.03 total 79.78 last 656.63) - /unlink 60000 (time 1755097518.84 total 85.59 last 590.60) total: 64768 /unlink in 92.77 seconds: 698.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 PASS 106 (1514s) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 oleg351-server: oleg351-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions oleg351-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg351-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1614 sec ========== 11:06:28 (1755097588) === conf-sanity: start cleanup 11:06:28 (1755097588) === === conf-sanity: finish cleanup 11:06:29 (1755097589) ===