-----============= acceptance-small: conf-sanity ============----- Sun Aug 31 17:32:04 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg252-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg252-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg252-server oleg252-server: oleg252-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg252-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg252-server: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg252-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:33:18 (1756675998) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) oleg252-server: oleg252-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg252-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4057 (time 1756676040.80 total 10.00 last 405.64) - open/close 9036 (time 1756676050.80 total 20.00 last 497.81) - open/close 13792 (time 1756676060.81 total 30.00 last 475.52) - open/close 18138 (time 1756676070.81 total 40.01 last 434.57) - open/close 20000 (time 1756676074.66 total 43.85 last 483.72) - open/close 24727 (time 1756676084.66 total 53.86 last 472.67) - open/close 29802 (time 1756676094.66 total 63.86 last 507.42) - open/close 34403 (time 1756676104.66 total 73.86 last 460.05) - open/close 38932 (time 1756676114.66 total 83.86 last 452.83) - open/close 40000 (time 1756676117.67 total 86.87 last 354.78) - open/close 43941 (time 1756676127.67 total 96.87 last 394.05) - open/close 48402 (time 1756676137.67 total 106.87 last 446.03) - open/close 50000 (time 1756676141.55 total 110.75 last 412.19) - open/close 54691 (time 1756676151.57 total 120.77 last 467.95) - open/close 59479 (time 1756676161.65 total 130.85 last 475.27) - open/close 63934 (time 1756676171.65 total 140.85 last 445.44) total: 64768 open/close in 142.79 seconds: 453.59 ops/second - /unlink 7282 (time 1756676185.07 total 10.00 last 728.17) - /unlink 10000 (time 1756676188.39 total 13.32 last 819.81) - /unlink 16854 (time 1756676198.39 total 23.32 last 685.34) - /unlink 20000 (time 1756676202.93 total 27.86 last 692.70) - /unlink 28273 (time 1756676212.93 total 37.86 last 827.05) - /unlink 36888 (time 1756676222.93 total 47.86 last 861.46) - /unlink 40000 (time 1756676226.91 total 51.83 last 783.54) - /unlink 47758 (time 1756676236.91 total 61.83 last 775.74) - /unlink 50000 (time 1756676240.34 total 65.27 last 652.24) - /unlink 57007 (time 1756676250.34 total 75.27 last 700.68) - /unlink 60000 (time 1756676254.61 total 79.54 last 701.98) total: 64768 /unlink in 85.49 seconds: 757.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4553 (time 1756676290.90 total 10.00 last 455.28) - open/close 8962 (time 1756676300.90 total 20.00 last 440.87) - open/close 10000 (time 1756676303.21 total 22.31 last 449.82) - open/close 14752 (time 1756676313.21 total 32.31 last 475.11) - open/close 20000 (time 1756676323.06 total 42.17 last 532.55) - open/close 25142 (time 1756676333.06 total 52.17 last 514.12) - open/close 29973 (time 1756676343.06 total 62.17 last 483.03) - open/close 34737 (time 1756676353.07 total 72.17 last 476.34) - open/close 39772 (time 1756676363.07 total 82.17 last 503.39) - open/close 44524 (time 1756676373.07 total 92.17 last 475.13) - open/close 49199 (time 1756676383.07 total 102.17 last 467.44) - open/close 54190 (time 1756676393.07 total 112.17 last 499.06) - open/close 58776 (time 1756676403.07 total 122.18 last 458.58) - open/close 60000 (time 1756676405.64 total 124.75 last 476.10) total: 64768 open/close in 134.71 seconds: 480.80 ops/second - /unlink 7326 (time 1756676426.93 total 10.00 last 732.55) - /unlink 10000 (time 1756676430.56 total 13.63 last 737.05) - /unlink 17370 (time 1756676440.56 total 23.63 last 736.90) - /unlink 20000 (time 1756676444.13 total 27.20 last 737.22) - /unlink 27307 (time 1756676454.13 total 37.20 last 730.64) - /unlink 30000 (time 1756676457.90 total 40.96 last 715.34) - /unlink 36912 (time 1756676467.90 total 50.96 last 691.15) - /unlink 40000 (time 1756676472.10 total 55.16 last 735.31) - /unlink 47562 (time 1756676482.10 total 65.16 last 756.18) - /unlink 50000 (time 1756676485.73 total 68.79 last 671.51) - /unlink 57053 (time 1756676495.73 total 78.80 last 705.19) - /unlink 60000 (time 1756676500.08 total 83.14 last 677.87) total: 64768 /unlink in 89.52 seconds: 723.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4643 (time 1756676535.45 total 10.00 last 464.25) - open/close 9221 (time 1756676545.45 total 20.00 last 457.75) - open/close 13498 (time 1756676555.45 total 30.00 last 427.63) - open/close 16548 (time 1756676565.45 total 40.00 last 304.99) - open/close 20000 (time 1756676573.81 total 48.36 last 413.19) - open/close 24146 (time 1756676583.81 total 58.36 last 414.46) - open/close 28398 (time 1756676593.81 total 68.36 last 425.19) - open/close 30000 (time 1756676597.55 total 72.10 last 428.52) - open/close 34623 (time 1756676607.55 total 82.10 last 462.23) - open/close 39115 (time 1756676617.55 total 92.10 last 449.18) - open/close 43620 (time 1756676627.55 total 102.10 last 450.49) - open/close 48084 (time 1756676637.55 total 112.10 last 446.34) - open/close 50000 (time 1756676641.76 total 116.32 last 454.68) - open/close 54448 (time 1756676651.77 total 126.32 last 444.80) - open/close 58667 (time 1756676661.77 total 136.32 last 421.84) - open/close 60000 (time 1756676664.79 total 139.35 last 440.57) - open/close 64447 (time 1756676674.79 total 149.35 last 444.67) total: 64768 open/close in 150.00 seconds: 431.79 ops/second - /unlink 7514 (time 1756676686.89 total 10.00 last 751.38) - /unlink 10000 (time 1756676690.36 total 13.47 last 716.63) - /unlink 17408 (time 1756676700.36 total 23.47 last 740.74) - /unlink 20000 (time 1756676703.54 total 26.65 last 814.34) - /unlink 27143 (time 1756676713.55 total 36.65 last 714.24) - /unlink 30000 (time 1756676717.53 total 40.64 last 717.62) - /unlink 37426 (time 1756676727.53 total 50.64 last 742.59) - /unlink 40000 (time 1756676731.12 total 54.23 last 717.01) - /unlink 47214 (time 1756676741.12 total 64.23 last 721.34) - /unlink 50000 (time 1756676744.69 total 67.80 last 779.57) - /unlink 58082 (time 1756676754.69 total 77.80 last 808.17) - /unlink 60000 (time 1756676757.62 total 80.73 last 655.10) total: 64768 /unlink in 87.63 seconds: 739.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4655 (time 1756676799.15 total 10.00 last 465.44) - open/close 9255 (time 1756676809.15 total 20.00 last 459.99) - open/close 10000 (time 1756676811.33 total 22.18 last 342.29) - open/close 14182 (time 1756676821.33 total 32.18 last 418.18) - open/close 18882 (time 1756676831.33 total 42.18 last 469.96) - open/close 20000 (time 1756676833.78 total 44.63 last 456.33) - open/close 24376 (time 1756676843.78 total 54.63 last 437.52) - open/close 28852 (time 1756676853.78 total 64.63 last 447.53) - open/close 30000 (time 1756676856.23 total 67.08 last 468.45) - open/close 33992 (time 1756676866.23 total 77.09 last 399.12) - open/close 38551 (time 1756676876.23 total 87.09 last 455.88) - open/close 40000 (time 1756676879.37 total 90.23 last 461.42) - open/close 44490 (time 1756676889.37 total 100.23 last 448.97) - open/close 48932 (time 1756676899.38 total 110.23 last 444.17) - open/close 50000 (time 1756676901.87 total 112.73 last 427.46) - open/close 54012 (time 1756676911.88 total 122.73 last 401.14) - open/close 58421 (time 1756676921.88 total 132.73 last 440.86) - open/close 60000 (time 1756676925.75 total 136.60 last 407.77) - open/close 64448 (time 1756676935.83 total 146.69 last 441.06) total: 64768 open/close in 147.39 seconds: 439.44 ops/second - /unlink 7751 (time 1756676947.89 total 10.00 last 775.05) - /unlink 10000 (time 1756676951.24 total 13.35 last 671.15) - /unlink 16798 (time 1756676961.24 total 23.35 last 679.75) - /unlink 20000 (time 1756676965.74 total 27.85 last 712.17) - /unlink 27395 (time 1756676975.74 total 37.85 last 739.45) - /unlink 30000 (time 1756676979.11 total 41.21 last 774.12) - /unlink 37583 (time 1756676989.11 total 51.22 last 757.86) - /unlink 40000 (time 1756676992.41 total 54.51 last 733.83) - /unlink 47314 (time 1756677002.41 total 64.51 last 731.35) - /unlink 50000 (time 1756677006.26 total 68.37 last 697.27) - /unlink 56898 (time 1756677016.26 total 78.37 last 689.78) - /unlink 60000 (time 1756677020.63 total 82.74 last 709.58) total: 64768 /unlink in 89.30 seconds: 725.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4440 (time 1756677057.54 total 10.00 last 443.97) - open/close 8815 (time 1756677067.54 total 20.00 last 437.47) - open/close 10000 (time 1756677070.25 total 22.71 last 437.36) - open/close 14174 (time 1756677080.25 total 32.71 last 417.32) - open/close 18924 (time 1756677090.25 total 42.71 last 474.92) - open/close 20000 (time 1756677092.50 total 44.97 last 477.55) - open/close 24667 (time 1756677102.50 total 54.97 last 466.66) - open/close 29322 (time 1756677112.51 total 64.97 last 465.44) - open/close 33892 (time 1756677122.51 total 74.97 last 456.91) - open/close 38509 (time 1756677132.51 total 84.97 last 461.70) - open/close 40000 (time 1756677135.77 total 88.24 last 456.45) - open/close 44589 (time 1756677145.77 total 98.24 last 458.88) - open/close 49036 (time 1756677155.78 total 108.24 last 444.66) - open/close 50000 (time 1756677157.99 total 110.45 last 435.97) - open/close 54456 (time 1756677167.99 total 120.45 last 445.53) - open/close 58892 (time 1756677177.99 total 130.45 last 443.53) - open/close 60000 (time 1756677180.43 total 132.89 last 454.76) - open/close 64416 (time 1756677190.43 total 142.89 last 441.59) total: 64768 open/close in 143.65 seconds: 450.88 ops/second - /unlink 7309 (time 1756677202.64 total 10.00 last 730.88) - /unlink 10000 (time 1756677206.41 total 13.77 last 714.19) - /unlink 17381 (time 1756677216.41 total 23.77 last 738.03) - /unlink 20000 (time 1756677220.04 total 27.40 last 720.93) - /unlink 27482 (time 1756677230.04 total 37.40 last 748.16) - /unlink 30000 (time 1756677233.22 total 40.58 last 791.57) - /unlink 37213 (time 1756677243.22 total 50.58 last 721.22) - /unlink 40000 (time 1756677246.74 total 54.10 last 792.13) - /unlink 47031 (time 1756677256.74 total 64.10 last 703.02) - /unlink 50000 (time 1756677260.42 total 67.79 last 806.28) - /unlink 57605 (time 1756677270.42 total 77.79 last 760.48) - /unlink 60000 (time 1756677273.72 total 81.08 last 727.49) total: 64768 /unlink in 87.23 seconds: 742.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 PASS 106 (1318s) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 oleg252-server: oleg252-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions oleg252-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg252-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1417 sec ========== 17:55:41 (1756677341) === conf-sanity: start cleanup 17:55:41 (1756677341) === === conf-sanity: finish cleanup 17:55:42 (1756677342) ===