-----============= acceptance-small: conf-sanity ============----- Wed Dec 3 01:51:54 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg237-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg237-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:53:08 (1764744788) Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg237-client.virtnet: -o user_xattr,flock 192.168.202.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1811 (time 1764744841.80 total 10.00 last 181.02) - open/close 3773 (time 1764744851.80 total 20.01 last 196.17) - open/close 6014 (time 1764744861.81 total 30.01 last 223.96) - open/close 7762 (time 1764744871.81 total 40.01 last 174.75) - open/close 10000 (time 1764744881.60 total 49.80 last 228.64) - open/close 12520 (time 1764744891.60 total 59.81 last 251.91) - open/close 14938 (time 1764744901.60 total 69.81 last 241.70) - open/close 18191 (time 1764744911.61 total 79.81 last 325.27) - open/close 20000 (time 1764744918.25 total 86.46 last 272.30) - open/close 22861 (time 1764744928.25 total 96.46 last 286.06) - open/close 26169 (time 1764744938.25 total 106.46 last 330.77) - open/close 29944 (time 1764744948.25 total 116.46 last 377.47) - open/close 33773 (time 1764744958.25 total 126.46 last 382.88) - open/close 38000 (time 1764744968.25 total 136.46 last 422.68) - open/close 40000 (time 1764744973.43 total 141.63 last 386.55) - open/close 44390 (time 1764744983.43 total 151.63 last 438.95) - open/close 48513 (time 1764744993.43 total 161.64 last 412.23) - open/close 50000 (time 1764744996.74 total 164.94 last 449.96) - open/close 54182 (time 1764745006.74 total 174.94 last 418.20) - open/close 58262 (time 1764745016.74 total 184.94 last 407.97) - open/close 60000 (time 1764745020.69 total 188.90 last 439.28) - open/close 63686 (time 1764745030.69 total 198.90 last 368.53) total: 64768 open/close in 202.48 seconds: 319.87 ops/second - /unlink 7778 (time 1764745045.90 total 10.00 last 777.75) - /unlink 10000 (time 1764745048.77 total 12.87 last 774.20) - /unlink 17043 (time 1764745058.77 total 22.87 last 704.24) - /unlink 20000 (time 1764745063.06 total 27.16 last 689.74) - /unlink 26984 (time 1764745073.06 total 37.16 last 698.37) - /unlink 30000 (time 1764745076.94 total 41.03 last 778.31) - /unlink 37345 (time 1764745086.94 total 51.04 last 734.42) - /unlink 40000 (time 1764745090.38 total 54.48 last 770.81) - /unlink 47463 (time 1764745100.38 total 64.48 last 746.23) - /unlink 50000 (time 1764745103.66 total 67.75 last 774.93) - /unlink 58240 (time 1764745113.66 total 77.75 last 823.96) - /unlink 60000 (time 1764745115.96 total 80.05 last 765.20) total: 64768 /unlink in 85.68 seconds: 755.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4065 (time 1764745157.04 total 10.00 last 406.43) - open/close 8894 (time 1764745167.04 total 20.00 last 482.84) - open/close 10000 (time 1764745169.21 total 22.17 last 509.70) - open/close 13949 (time 1764745179.21 total 32.17 last 394.84) - open/close 18206 (time 1764745189.21 total 42.18 last 425.65) - open/close 20000 (time 1764745193.13 total 46.09 last 458.27) - open/close 23951 (time 1764745203.13 total 56.09 last 395.06) - open/close 28892 (time 1764745213.13 total 66.09 last 494.04) - open/close 30000 (time 1764745215.18 total 68.14 last 540.56) - open/close 34690 (time 1764745225.18 total 78.14 last 468.95) - open/close 39505 (time 1764745235.18 total 88.14 last 481.49) - open/close 43876 (time 1764745245.18 total 98.14 last 437.10) - open/close 48315 (time 1764745255.20 total 108.16 last 443.08) - open/close 50000 (time 1764745258.84 total 111.81 last 462.37) - open/close 54491 (time 1764745268.84 total 121.81 last 449.08) - open/close 59347 (time 1764745278.84 total 131.81 last 485.53) - open/close 63731 (time 1764745288.85 total 141.81 last 438.34) total: 64768 open/close in 144.12 seconds: 449.41 ops/second - /unlink 7427 (time 1764745302.51 total 10.00 last 742.68) - /unlink 10000 (time 1764745305.75 total 13.24 last 793.82) - /unlink 17988 (time 1764745315.75 total 23.24 last 798.77) - /unlink 20000 (time 1764745318.87 total 26.36 last 644.74) - /unlink 27900 (time 1764745328.88 total 36.36 last 789.93) - /unlink 30000 (time 1764745331.06 total 38.55 last 962.32) - /unlink 38793 (time 1764745341.06 total 48.55 last 879.26) - /unlink 47393 (time 1764745351.06 total 58.55 last 860.00) - /unlink 50000 (time 1764745354.20 total 61.69 last 830.14) - /unlink 58354 (time 1764745364.20 total 71.69 last 835.35) total: 64768 /unlink in 79.21 seconds: 817.72 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4666 (time 1764745471.56 total 10.00 last 466.59) - open/close 9331 (time 1764745481.56 total 20.00 last 466.45) - open/close 13866 (time 1764745491.56 total 30.00 last 453.40) - open/close 18194 (time 1764745501.56 total 40.01 last 432.70) - open/close 20000 (time 1764745505.01 total 43.45 last 524.67) - open/close 24587 (time 1764745515.01 total 53.45 last 458.67) - open/close 29109 (time 1764745525.01 total 63.45 last 452.05) - open/close 30000 (time 1764745527.06 total 65.50 last 435.25) - open/close 34500 (time 1764745537.06 total 75.50 last 449.91) - open/close 39060 (time 1764745547.06 total 85.50 last 455.87) - open/close 40000 (time 1764745549.33 total 87.77 last 414.01) - open/close 44867 (time 1764745559.33 total 97.78 last 486.66) - open/close 49872 (time 1764745569.33 total 107.78 last 500.43) - open/close 54326 (time 1764745579.34 total 117.78 last 445.10) - open/close 58793 (time 1764745589.34 total 127.78 last 446.70) - open/close 60000 (time 1764745592.01 total 130.45 last 451.93) - open/close 64448 (time 1764745602.01 total 140.45 last 444.80) total: 64768 open/close in 141.24 seconds: 458.58 ops/second - /unlink 7362 (time 1764745614.21 total 10.00 last 736.20) - /unlink 10000 (time 1764745618.11 total 13.90 last 675.80) - /unlink 17069 (time 1764745628.11 total 23.90 last 706.83) - /unlink 20000 (time 1764745631.96 total 27.75 last 761.35) - /unlink 27189 (time 1764745641.96 total 37.76 last 718.84) - /unlink 30000 (time 1764745645.74 total 41.53 last 743.77) - /unlink 37408 (time 1764745655.74 total 51.54 last 740.76) - /unlink 40000 (time 1764745659.37 total 55.16 last 714.61) - /unlink 47507 (time 1764745669.37 total 65.16 last 750.66) - /unlink 50000 (time 1764745672.47 total 68.26 last 804.62) - /unlink 57529 (time 1764745682.47 total 78.26 last 752.88) - /unlink 60000 (time 1764745686.02 total 81.81 last 696.84) total: 64768 /unlink in 88.13 seconds: 734.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4761 (time 1764745725.41 total 10.00 last 476.01) - open/close 9380 (time 1764745735.41 total 20.00 last 461.88) - open/close 13664 (time 1764745745.41 total 30.00 last 428.39) - open/close 17919 (time 1764745755.41 total 40.00 last 425.50) - open/close 20000 (time 1764745760.26 total 44.85 last 429.02) - open/close 24072 (time 1764745770.26 total 54.85 last 407.17) - open/close 28211 (time 1764745780.26 total 64.85 last 413.87) - open/close 30000 (time 1764745784.56 total 69.15 last 416.64) - open/close 34365 (time 1764745794.56 total 79.15 last 436.45) - open/close 38856 (time 1764745804.58 total 89.17 last 448.03) - open/close 40000 (time 1764745807.31 total 91.90 last 419.29) - open/close 44062 (time 1764745817.31 total 101.90 last 406.18) - open/close 48263 (time 1764745827.31 total 111.90 last 420.02) - open/close 50000 (time 1764745831.59 total 116.18 last 406.22) - open/close 54152 (time 1764745841.59 total 126.18 last 415.15) - open/close 58143 (time 1764745851.59 total 136.18 last 399.09) - open/close 60000 (time 1764745856.03 total 140.62 last 418.55) - open/close 64366 (time 1764745866.03 total 150.62 last 436.54) total: 64768 open/close in 151.56 seconds: 427.33 ops/second - /unlink 7968 (time 1764745878.31 total 10.00 last 796.78) - /unlink 10000 (time 1764745881.19 total 12.88 last 704.42) - /unlink 13318 (time 1764745891.19 total 22.89 last 331.76) - /unlink 15445 (time 1764745901.20 total 32.89 last 212.60) - /unlink 20000 (time 1764745909.78 total 41.48 last 530.54) - /unlink 28161 (time 1764745919.78 total 51.48 last 816.10) - /unlink 30000 (time 1764745922.38 total 54.07 last 708.97) - /unlink 38408 (time 1764745932.38 total 64.07 last 840.63) - /unlink 45586 (time 1764745942.38 total 74.07 last 717.78) - /unlink 50000 (time 1764745948.34 total 80.03 last 740.78) - /unlink 57942 (time 1764745958.34 total 90.03 last 794.14) - /unlink 60000 (time 1764745960.53 total 92.23 last 937.22) total: 64768 /unlink in 97.98 seconds: 661.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4752 (time 1764745994.16 total 10.00 last 475.09) - open/close 9943 (time 1764746004.16 total 20.00 last 519.09) - open/close 14519 (time 1764746014.17 total 30.00 last 457.58) - open/close 19173 (time 1764746024.17 total 40.00 last 465.32) - open/close 24398 (time 1764746034.17 total 50.01 last 522.41) - open/close 29820 (time 1764746044.17 total 60.01 last 542.17) - open/close 34936 (time 1764746054.17 total 70.01 last 511.60) - open/close 39438 (time 1764746064.18 total 80.01 last 449.87) - open/close 43703 (time 1764746074.18 total 90.02 last 426.45) - open/close 48016 (time 1764746084.18 total 100.02 last 431.25) - open/close 50000 (time 1764746088.94 total 104.78 last 416.55) - open/close 54345 (time 1764746098.94 total 114.78 last 434.41) - open/close 58613 (time 1764746108.94 total 124.78 last 426.78) - open/close 60000 (time 1764746112.42 total 128.26 last 398.58) - open/close 64602 (time 1764746122.43 total 138.26 last 460.13) total: 64768 open/close in 138.58 seconds: 467.38 ops/second - /unlink 8443 (time 1764746134.08 total 10.00 last 844.28) - /unlink 16973 (time 1764746144.08 total 20.00 last 852.97) - /unlink 20000 (time 1764746148.21 total 24.13 last 733.13) - /unlink 27902 (time 1764746158.21 total 34.13 last 790.17) - /unlink 30000 (time 1764746160.94 total 36.86 last 768.49) - /unlink 37778 (time 1764746170.94 total 46.86 last 777.72) - /unlink 40000 (time 1764746173.59 total 49.52 last 836.50) - /unlink 47949 (time 1764746183.59 total 59.52 last 794.89) - /unlink 50000 (time 1764746186.21 total 62.14 last 782.87) - /unlink 57845 (time 1764746196.21 total 72.14 last 784.39) - /unlink 60000 (time 1764746199.00 total 74.93 last 772.96) total: 64768 /unlink in 80.71 seconds: 802.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg237-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 PASS 106 (1460s) Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1559 sec ========== 02:17:54 (1764746274) === conf-sanity: start cleanup 02:17:54 (1764746274) === === conf-sanity: finish cleanup 02:17:55 (1764746275) ===