-----============= acceptance-small: conf-sanity ============----- Sun Nov 23 00:36:17 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg104-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg104-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg104-server oleg104-server: oleg104-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg104-server: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg104-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:37:30 (1763876250) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) oleg104-server: oleg104-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5133 (time 1763876296.70 total 10.00 last 513.23) - open/close 10000 (time 1763876306.45 total 19.76 last 498.99) - open/close 15054 (time 1763876316.45 total 29.76 last 505.33) - open/close 19762 (time 1763876326.45 total 39.76 last 470.66) - open/close 24609 (time 1763876336.45 total 49.76 last 484.69) - open/close 29157 (time 1763876346.45 total 59.76 last 454.76) - open/close 33884 (time 1763876356.46 total 69.76 last 472.68) - open/close 38763 (time 1763876366.46 total 79.76 last 487.90) - open/close 40000 (time 1763876369.12 total 82.43 last 463.38) - open/close 44690 (time 1763876379.13 total 92.43 last 468.96) - open/close 49268 (time 1763876389.13 total 102.43 last 457.72) - open/close 54243 (time 1763876399.13 total 112.43 last 497.42) - open/close 58979 (time 1763876409.13 total 122.44 last 473.50) - open/close 60000 (time 1763876411.23 total 124.53 last 486.75) total: 64768 open/close in 134.21 seconds: 482.59 ops/second - /unlink 8611 (time 1763876432.13 total 10.00 last 861.08) - /unlink 15953 (time 1763876442.13 total 20.00 last 734.15) - /unlink 20000 (time 1763876448.60 total 26.47 last 625.56) - /unlink 27548 (time 1763876458.60 total 36.47 last 754.79) - /unlink 30000 (time 1763876461.34 total 39.21 last 895.07) - /unlink 37670 (time 1763876471.34 total 49.21 last 766.91) - /unlink 40000 (time 1763876474.11 total 51.98 last 841.39) - /unlink 47884 (time 1763876484.11 total 61.98 last 788.34) - /unlink 50000 (time 1763876487.00 total 64.87 last 732.65) - /unlink 57651 (time 1763876497.00 total 74.87 last 765.09) - /unlink 60000 (time 1763876499.83 total 77.70 last 829.21) total: 64768 /unlink in 84.40 seconds: 767.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4789 (time 1763876542.04 total 10.00 last 478.87) - open/close 9308 (time 1763876552.04 total 20.00 last 451.87) - open/close 14155 (time 1763876562.04 total 30.00 last 484.68) - open/close 18608 (time 1763876572.04 total 40.00 last 445.22) - open/close 20000 (time 1763876574.67 total 42.63 last 530.03) - open/close 25237 (time 1763876584.67 total 52.63 last 523.62) - open/close 30000 (time 1763876594.54 total 62.50 last 482.56) - open/close 34633 (time 1763876604.55 total 72.51 last 462.78) - open/close 38454 (time 1763876614.55 total 82.51 last 382.04) - open/close 40000 (time 1763876618.46 total 86.42 last 396.00) - open/close 43903 (time 1763876628.46 total 96.42 last 390.30) - open/close 48256 (time 1763876638.46 total 106.42 last 435.29) - open/close 50000 (time 1763876642.80 total 110.76 last 401.53) - open/close 53653 (time 1763876652.80 total 120.76 last 365.29) - open/close 58228 (time 1763876662.80 total 130.76 last 457.39) - open/close 60000 (time 1763876666.88 total 134.84 last 434.66) - open/close 64574 (time 1763876676.88 total 144.84 last 457.39) total: 64768 open/close in 145.41 seconds: 445.40 ops/second - /unlink 7275 (time 1763876688.85 total 10.00 last 727.46) - /unlink 10000 (time 1763876692.31 total 13.46 last 788.42) - /unlink 17807 (time 1763876702.31 total 23.46 last 780.63) - /unlink 20000 (time 1763876705.72 total 26.87 last 643.06) - /unlink 27494 (time 1763876715.72 total 36.87 last 749.38) - /unlink 30000 (time 1763876719.34 total 40.49 last 691.51) - /unlink 37370 (time 1763876729.35 total 50.49 last 736.95) - /unlink 40000 (time 1763876732.66 total 53.81 last 792.71) - /unlink 47800 (time 1763876742.66 total 63.81 last 779.87) - /unlink 50000 (time 1763876745.50 total 66.65 last 775.36) - /unlink 58009 (time 1763876755.50 total 76.65 last 800.90) - /unlink 60000 (time 1763876757.71 total 78.86 last 900.07) total: 64768 /unlink in 85.07 seconds: 761.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4768 (time 1763876796.35 total 10.00 last 476.78) - open/close 9362 (time 1763876806.35 total 20.00 last 459.38) - open/close 14419 (time 1763876816.35 total 30.00 last 505.68) - open/close 19618 (time 1763876826.35 total 40.00 last 519.83) - open/close 25029 (time 1763876836.35 total 50.00 last 541.04) - open/close 30000 (time 1763876846.27 total 59.92 last 501.46) - open/close 35008 (time 1763876856.27 total 69.92 last 500.71) - open/close 40000 (time 1763876866.26 total 79.91 last 499.82) - open/close 44219 (time 1763876876.26 total 89.91 last 421.84) - open/close 48919 (time 1763876886.26 total 99.91 last 469.98) - open/close 50000 (time 1763876888.56 total 102.21 last 470.09) - open/close 55056 (time 1763876898.56 total 112.21 last 505.60) - open/close 59800 (time 1763876908.56 total 122.21 last 474.36) - open/close 64397 (time 1763876918.56 total 132.21 last 459.63) total: 64768 open/close in 133.16 seconds: 486.38 ops/second - /unlink 6680 (time 1763876930.85 total 10.00 last 667.96) - /unlink 10000 (time 1763876935.63 total 14.78 last 694.74) - /unlink 17047 (time 1763876945.63 total 24.78 last 704.65) - /unlink 20000 (time 1763876949.42 total 28.56 last 780.21) - /unlink 27209 (time 1763876959.42 total 38.57 last 720.82) - /unlink 30000 (time 1763876963.19 total 42.34 last 740.01) - /unlink 37365 (time 1763876973.19 total 52.34 last 736.46) - /unlink 40000 (time 1763876976.35 total 55.49 last 835.12) - /unlink 47609 (time 1763876986.35 total 65.49 last 760.84) - /unlink 50000 (time 1763876989.10 total 68.25 last 867.90) - /unlink 58133 (time 1763876999.10 total 78.25 last 813.27) total: 64768 /unlink in 86.43 seconds: 749.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4869 (time 1763877045.18 total 10.00 last 486.85) - open/close 9531 (time 1763877055.18 total 20.00 last 466.10) - open/close 14614 (time 1763877065.18 total 30.00 last 508.26) - open/close 19621 (time 1763877075.18 total 40.01 last 500.60) - open/close 24910 (time 1763877085.18 total 50.01 last 528.88) - open/close 30000 (time 1763877094.73 total 59.55 last 533.12) - open/close 35294 (time 1763877104.73 total 69.55 last 529.38) - open/close 40000 (time 1763877113.27 total 78.09 last 551.16) - open/close 44560 (time 1763877123.27 total 88.09 last 455.96) - open/close 48981 (time 1763877133.27 total 98.10 last 442.02) - open/close 50000 (time 1763877135.59 total 100.41 last 439.43) - open/close 54123 (time 1763877145.59 total 110.41 last 412.27) - open/close 59195 (time 1763877155.59 total 120.42 last 507.13) - open/close 63563 (time 1763877165.59 total 130.42 last 436.72) total: 64768 open/close in 133.22 seconds: 486.17 ops/second - /unlink 7287 (time 1763877179.75 total 10.00 last 728.56) - /unlink 10000 (time 1763877183.18 total 13.43 last 792.20) - /unlink 17308 (time 1763877193.18 total 23.43 last 730.78) - /unlink 20000 (time 1763877196.87 total 27.12 last 728.62) - /unlink 26724 (time 1763877206.87 total 37.12 last 672.34) - /unlink 30000 (time 1763877211.37 total 41.62 last 728.70) - /unlink 37310 (time 1763877221.37 total 51.62 last 730.94) - /unlink 40000 (time 1763877224.65 total 54.90 last 820.33) - /unlink 47091 (time 1763877234.65 total 64.90 last 709.09) - /unlink 50000 (time 1763877238.65 total 68.90 last 727.65) - /unlink 57198 (time 1763877248.65 total 78.90 last 719.76) - /unlink 60000 (time 1763877252.56 total 82.81 last 716.44) total: 64768 /unlink in 89.10 seconds: 726.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4560 (time 1763877289.29 total 10.00 last 455.99) - open/close 9137 (time 1763877299.29 total 20.00 last 457.63) - open/close 13597 (time 1763877309.29 total 30.00 last 445.94) - open/close 18153 (time 1763877319.30 total 40.01 last 455.52) - open/close 20000 (time 1763877323.31 total 44.02 last 459.95) - open/close 24597 (time 1763877333.31 total 54.02 last 459.63) - open/close 29206 (time 1763877343.31 total 64.02 last 460.90) - open/close 33853 (time 1763877353.32 total 74.02 last 464.63) - open/close 38359 (time 1763877363.32 total 84.02 last 450.59) - open/close 40000 (time 1763877367.05 total 87.76 last 439.24) - open/close 44328 (time 1763877377.05 total 97.76 last 432.75) - open/close 48605 (time 1763877387.05 total 107.76 last 427.66) - open/close 50000 (time 1763877390.24 total 110.95 last 437.68) - open/close 54345 (time 1763877400.24 total 120.95 last 434.48) - open/close 58708 (time 1763877410.24 total 130.95 last 436.22) - open/close 60000 (time 1763877413.26 total 133.97 last 428.04) - open/close 64274 (time 1763877423.26 total 143.97 last 427.31) total: 64768 open/close in 145.32 seconds: 445.68 ops/second - /unlink 7003 (time 1763877436.05 total 10.00 last 700.30) - /unlink 10000 (time 1763877440.27 total 14.22 last 709.75) - /unlink 17321 (time 1763877450.30 total 24.25 last 730.33) - /unlink 20000 (time 1763877454.35 total 28.30 last 661.40) - /unlink 27051 (time 1763877464.35 total 38.30 last 705.09) - /unlink 30000 (time 1763877468.58 total 42.53 last 697.09) - /unlink 37258 (time 1763877478.58 total 52.53 last 725.75) - /unlink 40000 (time 1763877482.52 total 56.47 last 694.90) - /unlink 47501 (time 1763877492.52 total 66.48 last 750.01) - /unlink 50000 (time 1763877495.56 total 69.51 last 822.80) - /unlink 57124 (time 1763877505.57 total 79.52 last 711.68) - /unlink 60000 (time 1763877509.63 total 83.58 last 709.44) total: 64768 /unlink in 90.26 seconds: 717.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 PASS 106 (1302s) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 oleg104-server: oleg104-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg104-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions oleg104-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg104-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1400 sec ========== 00:59:37 (1763877577) === conf-sanity: start cleanup 00:59:38 (1763877578) === === conf-sanity: finish cleanup 00:59:39 (1763877579) ===