-----============= acceptance-small: conf-sanity ============----- Sun Nov 23 00:36:38 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg146-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg146-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg146-server oleg146-server: oleg146-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg146-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg146-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg146-server: oleg146-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg146-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 00:38:19 (1763876299) Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f) oleg146-server: oleg146-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg146-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg146-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg146-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4265 (time 1763876350.06 total 10.00 last 426.37) - open/close 8779 (time 1763876360.07 total 20.01 last 451.30) - open/close 10000 (time 1763876362.72 total 22.66 last 460.26) - open/close 14667 (time 1763876372.72 total 32.66 last 466.64) - open/close 19157 (time 1763876382.72 total 42.66 last 448.95) - open/close 23967 (time 1763876392.72 total 52.66 last 480.80) - open/close 28777 (time 1763876402.73 total 62.67 last 480.97) - open/close 30000 (time 1763876405.14 total 65.08 last 507.43) - open/close 34997 (time 1763876415.14 total 75.08 last 499.62) - open/close 40000 (time 1763876425.04 total 84.98 last 505.26) - open/close 44959 (time 1763876435.04 total 94.98 last 495.87) - open/close 49189 (time 1763876445.04 total 104.98 last 422.94) - open/close 53312 (time 1763876455.04 total 114.98 last 412.27) - open/close 58075 (time 1763876465.04 total 124.98 last 476.21) - open/close 60000 (time 1763876469.13 total 129.07 last 471.43) total: 64768 open/close in 138.94 seconds: 466.16 ops/second - /unlink 8637 (time 1763876490.34 total 10.00 last 863.65) - /unlink 17011 (time 1763876500.35 total 20.00 last 837.34) - /unlink 20000 (time 1763876503.78 total 23.43 last 871.22) - /unlink 28504 (time 1763876513.78 total 33.43 last 850.38) - /unlink 37165 (time 1763876523.78 total 43.43 last 866.09) - /unlink 40000 (time 1763876526.86 total 46.52 last 918.38) - /unlink 48997 (time 1763876536.86 total 56.52 last 899.63) - /unlink 57524 (time 1763876546.86 total 66.52 last 852.68) - /unlink 60000 (time 1763876549.64 total 69.30 last 890.84) total: 64768 /unlink in 74.70 seconds: 867.09 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4138 (time 1763876617.61 total 10.00 last 413.77) - open/close 8581 (time 1763876627.61 total 20.00 last 444.20) - open/close 10000 (time 1763876630.65 total 23.05 last 465.98) - open/close 14820 (time 1763876640.66 total 33.05 last 481.92) - open/close 19156 (time 1763876650.66 total 43.05 last 433.57) - open/close 23972 (time 1763876660.66 total 53.05 last 481.53) - open/close 28935 (time 1763876670.66 total 63.05 last 496.27) - open/close 30000 (time 1763876672.89 total 65.28 last 477.50) - open/close 34384 (time 1763876682.89 total 75.28 last 438.32) - open/close 39245 (time 1763876692.89 total 85.29 last 486.08) - open/close 44032 (time 1763876702.89 total 95.29 last 478.58) - open/close 48721 (time 1763876712.89 total 105.29 last 468.85) - open/close 50000 (time 1763876715.56 total 107.95 last 480.57) - open/close 54516 (time 1763876725.56 total 117.95 last 451.55) - open/close 59085 (time 1763876735.56 total 127.95 last 456.82) - open/close 64311 (time 1763876745.56 total 137.96 last 522.48) total: 64768 open/close in 138.83 seconds: 466.52 ops/second - /unlink 8404 (time 1763876757.81 total 10.00 last 840.27) - /unlink 10000 (time 1763876759.92 total 12.11 last 755.69) - /unlink 19028 (time 1763876769.92 total 22.11 last 902.75) - /unlink 28155 (time 1763876779.92 total 32.11 last 912.62) - /unlink 30000 (time 1763876782.05 total 34.24 last 867.20) - /unlink 39155 (time 1763876792.05 total 44.24 last 915.49) - /unlink 47463 (time 1763876802.05 total 54.24 last 830.77) - /unlink 50000 (time 1763876804.90 total 57.10 last 889.09) - /unlink 59152 (time 1763876814.90 total 67.10 last 915.17) total: 64768 /unlink in 73.17 seconds: 885.18 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4711 (time 1763876920.60 total 10.00 last 471.05) - open/close 8734 (time 1763876930.60 total 20.00 last 402.23) - open/close 10000 (time 1763876933.56 total 22.96 last 427.68) - open/close 14641 (time 1763876943.56 total 32.96 last 464.02) - open/close 19114 (time 1763876953.56 total 42.97 last 447.26) - open/close 23662 (time 1763876963.56 total 52.97 last 454.79) - open/close 28196 (time 1763876973.57 total 62.97 last 453.35) - open/close 30000 (time 1763876977.05 total 66.45 last 517.63) - open/close 34579 (time 1763876987.05 total 76.45 last 457.84) - open/close 39462 (time 1763876997.05 total 86.45 last 488.26) - open/close 44348 (time 1763877007.05 total 96.46 last 488.53) - open/close 49479 (time 1763877017.05 total 106.46 last 513.08) - open/close 54513 (time 1763877027.06 total 116.46 last 503.33) - open/close 59665 (time 1763877037.06 total 126.46 last 515.12) - open/close 64337 (time 1763877047.06 total 136.46 last 467.16) total: 64768 open/close in 137.33 seconds: 471.61 ops/second - /unlink 7982 (time 1763877059.32 total 10.00 last 798.15) - /unlink 10000 (time 1763877061.60 total 12.28 last 884.81) - /unlink 18187 (time 1763877071.60 total 22.28 last 818.64) - /unlink 20000 (time 1763877073.88 total 24.57 last 793.51) - /unlink 28737 (time 1763877083.88 total 34.57 last 873.65) - /unlink 38169 (time 1763877093.88 total 44.57 last 943.20) - /unlink 47359 (time 1763877103.88 total 54.57 last 918.99) - /unlink 50000 (time 1763877106.64 total 57.32 last 959.55) - /unlink 59435 (time 1763877116.64 total 67.32 last 943.42) total: 64768 /unlink in 73.96 seconds: 875.75 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4665 (time 1763877186.55 total 10.00 last 466.41) - open/close 9309 (time 1763877196.55 total 20.00 last 464.31) - open/close 13769 (time 1763877206.56 total 30.01 last 445.89) - open/close 18272 (time 1763877216.56 total 40.01 last 450.20) - open/close 20000 (time 1763877220.55 total 44.00 last 432.48) - open/close 24398 (time 1763877230.55 total 54.00 last 439.78) - open/close 28753 (time 1763877240.56 total 64.01 last 435.46) - open/close 30000 (time 1763877243.48 total 66.93 last 426.13) - open/close 34365 (time 1763877253.48 total 76.93 last 436.46) - open/close 38846 (time 1763877263.48 total 86.93 last 448.04) - open/close 40000 (time 1763877266.17 total 89.62 last 429.78) - open/close 44612 (time 1763877276.17 total 99.62 last 461.12) - open/close 49124 (time 1763877286.17 total 109.62 last 451.17) - open/close 50000 (time 1763877288.19 total 111.64 last 433.76) - open/close 54556 (time 1763877298.19 total 121.64 last 455.52) - open/close 59067 (time 1763877308.19 total 131.64 last 451.03) - open/close 60000 (time 1763877310.29 total 133.74 last 445.41) - open/close 64504 (time 1763877320.29 total 143.74 last 450.37) total: 64768 open/close in 144.32 seconds: 448.78 ops/second - /unlink 7879 (time 1763877332.35 total 10.00 last 787.84) - /unlink 10000 (time 1763877335.32 total 12.97 last 714.97) - /unlink 17672 (time 1763877345.32 total 22.97 last 767.16) - /unlink 20000 (time 1763877348.22 total 25.87 last 801.58) - /unlink 27515 (time 1763877358.22 total 35.87 last 751.46) - /unlink 30000 (time 1763877361.18 total 38.83 last 839.77) - /unlink 37853 (time 1763877371.18 total 48.83 last 785.29) - /unlink 40000 (time 1763877373.73 total 51.38 last 842.57) - /unlink 48072 (time 1763877383.73 total 61.38 last 807.16) - /unlink 50000 (time 1763877386.33 total 63.98 last 741.62) - /unlink 57954 (time 1763877396.33 total 73.98 last 795.38) - /unlink 60000 (time 1763877398.81 total 76.46 last 826.16) total: 64768 /unlink in 82.43 seconds: 785.70 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 - open/close 4551 (time 1763877457.32 total 10.00 last 455.01) - open/close 8975 (time 1763877467.32 total 20.00 last 442.34) - open/close 10000 (time 1763877469.55 total 22.24 last 458.76) - open/close 14533 (time 1763877479.55 total 32.24 last 453.24) - open/close 19072 (time 1763877489.55 total 42.24 last 453.89) - open/close 20000 (time 1763877491.58 total 44.27 last 458.14) - open/close 24571 (time 1763877501.58 total 54.27 last 457.06) - open/close 29125 (time 1763877511.58 total 64.27 last 455.38) - open/close 33386 (time 1763877521.58 total 74.27 last 426.06) - open/close 36840 (time 1763877531.58 total 84.27 last 345.35) - open/close 40000 (time 1763877538.93 total 91.61 last 430.45) - open/close 44230 (time 1763877548.93 total 101.62 last 422.72) - open/close 48667 (time 1763877558.93 total 111.62 last 443.66) - open/close 50000 (time 1763877561.91 total 114.60 last 447.15) - open/close 54455 (time 1763877571.92 total 124.60 last 445.42) - open/close 58854 (time 1763877581.92 total 134.60 last 439.88) - open/close 60000 (time 1763877584.49 total 137.18 last 444.34) - open/close 64358 (time 1763877594.50 total 147.18 last 435.72) total: 64768 open/close in 148.16 seconds: 437.16 ops/second - /unlink 7673 (time 1763877606.92 total 10.00 last 767.23) - /unlink 10000 (time 1763877610.43 total 13.51 last 662.92) - /unlink 17767 (time 1763877620.43 total 23.51 last 776.61) - /unlink 20000 (time 1763877623.20 total 26.28 last 807.88) - /unlink 27382 (time 1763877633.20 total 36.28 last 738.17) - /unlink 30000 (time 1763877636.53 total 39.61 last 784.99) - /unlink 37540 (time 1763877646.53 total 49.61 last 753.98) - /unlink 40000 (time 1763877649.57 total 52.65 last 809.10) - /unlink 47956 (time 1763877659.57 total 62.65 last 795.51) - /unlink 50000 (time 1763877662.57 total 65.65 last 681.34) - /unlink 58067 (time 1763877672.57 total 75.65 last 806.68) - /unlink 60000 (time 1763877675.03 total 78.11 last 788.30) total: 64768 /unlink in 84.01 seconds: 770.97 ops/second Waiting for MDT destroys to complete oleg146-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local oleg146-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg146-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 PASS 106 (1475s) Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 oleg146-server: oleg146-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg146-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg146-server: detected 4 online CPUs by sysfs oleg146-server: MODOPTS_LIBCFS= oleg146-server: Force libcfs to create 2 CPU partitions oleg146-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg146-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1600 sec ========== 01:03:19 (1763877799) === conf-sanity: start cleanup 01:03:19 (1763877799) === === conf-sanity: finish cleanup 01:03:20 (1763877800) ===