-----============= acceptance-small: conf-sanity ============----- Wed Aug 13 10:39:52 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg325-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg325-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg325-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg325-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg325-server oleg325-server: oleg325-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg325-server: detected 4 online CPUs by sysfs oleg325-server: MODOPTS_LIBCFS= oleg325-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg325-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg325-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg325-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg325-server: oleg325-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg325-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg325-server: oleg325-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg325-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:41:39 (1755096099) Stopping clients: oleg325-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg325-client.virtnet /mnt/lustre2 (opts:-f) oleg325-server: oleg325-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg325-server: detected 4 online CPUs by sysfs oleg325-server: MODOPTS_LIBCFS= oleg325-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg325-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg325-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg325-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3454 (time 1755096153.66 total 10.00 last 345.37) - open/close 7185 (time 1755096163.66 total 20.00 last 373.00) - open/close 10000 (time 1755096173.30 total 29.64 last 292.15) - open/close 13890 (time 1755096183.30 total 39.64 last 388.94) - open/close 17353 (time 1755096193.30 total 49.64 last 346.22) - open/close 20000 (time 1755096200.40 total 56.75 last 372.59) - open/close 23749 (time 1755096210.41 total 66.75 last 374.88) - open/close 27011 (time 1755096220.41 total 76.75 last 326.11) - open/close 30000 (time 1755096229.08 total 85.42 last 344.72) - open/close 34308 (time 1755096239.08 total 95.42 last 430.70) - open/close 37472 (time 1755096249.08 total 105.42 last 316.37) - open/close 40000 (time 1755096257.08 total 113.42 last 316.07) - open/close 43719 (time 1755096267.08 total 123.43 last 371.81) - open/close 47161 (time 1755096277.08 total 133.43 last 344.18) - open/close 49999 (time 1755096287.09 total 143.43 last 283.74) - open/close 53073 (time 1755096297.09 total 153.43 last 307.40) - open/close 55588 (time 1755096307.09 total 163.43 last 251.40) - open/close 57912 (time 1755096317.09 total 173.43 last 232.38) - open/close 60000 (time 1755096325.39 total 181.73 last 251.72) - open/close 62594 (time 1755096335.39 total 191.73 last 259.37) total: 64768 open/close in 198.24 seconds: 326.71 ops/second - /unlink 5187 (time 1755096353.40 total 10.00 last 518.69) - /unlink 10000 (time 1755096362.48 total 19.08 last 530.14) - /unlink 14972 (time 1755096372.48 total 29.08 last 497.11) - /unlink 19915 (time 1755096382.48 total 39.08 last 494.21) - /unlink 24543 (time 1755096392.48 total 49.08 last 462.75) - /unlink 29773 (time 1755096402.48 total 59.09 last 522.89) - /unlink 34336 (time 1755096412.48 total 69.09 last 456.24) - /unlink 39957 (time 1755096422.49 total 79.09 last 562.08) - /unlink 46732 (time 1755096432.49 total 89.09 last 677.48) - /unlink 50000 (time 1755096436.89 total 93.49 last 741.66) - /unlink 56079 (time 1755096446.89 total 103.49 last 607.89) - /unlink 60000 (time 1755096452.49 total 109.09 last 701.01) total: 64768 /unlink in 116.18 seconds: 557.50 ops/second Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 - open/close 3956 (time 1755096480.83 total 10.00 last 395.52) - open/close 8172 (time 1755096490.83 total 20.00 last 421.57) - open/close 10000 (time 1755096495.09 total 24.26 last 428.93) - open/close 14173 (time 1755096505.09 total 34.27 last 417.22) - open/close 18183 (time 1755096515.09 total 44.27 last 400.95) - open/close 20000 (time 1755096519.78 total 48.96 last 387.57) - open/close 23152 (time 1755096529.79 total 58.96 last 315.09) - open/close 27034 (time 1755096539.79 total 68.96 last 388.10) - open/close 30000 (time 1755096548.32 total 77.49 last 347.62) - open/close 33540 (time 1755096558.32 total 87.49 last 353.96) - open/close 37520 (time 1755096568.32 total 97.50 last 397.96) - open/close 40000 (time 1755096574.43 total 103.60 last 406.30) - open/close 43531 (time 1755096584.43 total 113.60 last 353.02) - open/close 47738 (time 1755096594.43 total 123.60 last 420.67) - open/close 50000 (time 1755096600.16 total 129.33 last 394.65) - open/close 53840 (time 1755096610.16 total 139.34 last 383.95) - open/close 58146 (time 1755096620.16 total 149.34 last 430.53) - open/close 60000 (time 1755096624.65 total 153.83 last 412.97) - open/close 63892 (time 1755096634.65 total 163.83 last 389.13) total: 64768 open/close in 166.22 seconds: 389.64 ops/second - /unlink 7211 (time 1755096648.48 total 10.00 last 721.04) - /unlink 10000 (time 1755096651.92 total 13.44 last 811.92) - /unlink 18297 (time 1755096661.92 total 23.44 last 829.64) - /unlink 20000 (time 1755096664.12 total 25.64 last 772.72) - /unlink 27979 (time 1755096674.13 total 35.64 last 797.87) - /unlink 30000 (time 1755096676.68 total 38.19 last 791.55) - /unlink 38146 (time 1755096686.68 total 48.20 last 814.51) - /unlink 40000 (time 1755096688.90 total 50.42 last 833.43) - /unlink 48483 (time 1755096698.90 total 60.42 last 848.27) - /unlink 56902 (time 1755096708.90 total 70.42 last 841.88) - /unlink 60000 (time 1755096713.28 total 74.80 last 708.13) total: 64768 /unlink in 80.64 seconds: 803.21 ops/second Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 - open/close 5137 (time 1755096811.09 total 10.00 last 513.65) - open/close 9649 (time 1755096821.09 total 20.00 last 451.18) - open/close 14384 (time 1755096831.09 total 30.00 last 473.48) - open/close 19137 (time 1755096841.09 total 40.00 last 475.24) - open/close 23883 (time 1755096851.09 total 50.00 last 474.57) - open/close 28771 (time 1755096861.09 total 60.00 last 488.75) - open/close 30000 (time 1755096864.09 total 63.00 last 410.73) - open/close 35219 (time 1755096874.09 total 73.00 last 521.85) - open/close 40000 (time 1755096883.25 total 82.16 last 522.00) - open/close 44994 (time 1755096893.25 total 92.16 last 499.32) - open/close 50000 (time 1755096903.16 total 102.07 last 504.99) - open/close 55184 (time 1755096913.16 total 112.07 last 518.39) - open/close 60000 (time 1755096922.37 total 121.28 last 523.06) total: 64768 open/close in 130.47 seconds: 496.43 ops/second - /unlink 8352 (time 1755096942.90 total 10.00 last 835.16) - /unlink 10000 (time 1755096945.17 total 12.27 last 726.24) - /unlink 17430 (time 1755096955.17 total 22.27 last 742.99) - /unlink 20000 (time 1755096959.17 total 26.27 last 642.79) - /unlink 27781 (time 1755096969.17 total 36.27 last 778.03) - /unlink 30000 (time 1755096972.00 total 39.10 last 783.88) - /unlink 38737 (time 1755096982.00 total 49.10 last 873.63) - /unlink 47949 (time 1755096992.00 total 59.10 last 921.17) - /unlink 50000 (time 1755096994.55 total 61.65 last 803.28) - /unlink 57737 (time 1755097004.55 total 71.65 last 773.65) - /unlink 60000 (time 1755097007.31 total 74.41 last 820.24) total: 64768 /unlink in 80.53 seconds: 804.26 ops/second Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 - open/close 5246 (time 1755097069.93 total 10.00 last 524.58) - open/close 9793 (time 1755097079.93 total 20.00 last 454.67) - open/close 14525 (time 1755097089.93 total 30.00 last 473.10) - open/close 19560 (time 1755097099.94 total 40.00 last 503.42) - open/close 25020 (time 1755097109.94 total 50.01 last 545.95) - open/close 29698 (time 1755097119.94 total 60.01 last 467.79) - open/close 33971 (time 1755097129.94 total 70.01 last 427.22) - open/close 39064 (time 1755097139.94 total 80.01 last 509.23) - open/close 44303 (time 1755097149.94 total 90.01 last 523.89) - open/close 49101 (time 1755097159.94 total 100.01 last 479.79) - open/close 54170 (time 1755097169.94 total 110.01 last 506.88) - open/close 59632 (time 1755097179.94 total 120.01 last 546.16) total: 64768 open/close in 129.82 seconds: 498.91 ops/second - /unlink 8138 (time 1755097201.08 total 10.00 last 813.77) - /unlink 16956 (time 1755097211.10 total 20.01 last 880.51) - /unlink 20000 (time 1755097214.23 total 23.14 last 972.84) - /unlink 28437 (time 1755097224.23 total 33.15 last 843.58) - /unlink 30000 (time 1755097226.45 total 35.37 last 702.53) - /unlink 39344 (time 1755097236.45 total 45.37 last 934.30) - /unlink 47142 (time 1755097246.45 total 55.37 last 779.77) - /unlink 50000 (time 1755097249.45 total 58.37 last 954.27) - /unlink 58953 (time 1755097259.45 total 68.37 last 895.21) total: 64768 /unlink in 75.81 seconds: 854.40 ops/second Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 - open/close 5164 (time 1755097318.82 total 10.00 last 516.39) - open/close 9590 (time 1755097328.82 total 20.00 last 442.55) - open/close 14193 (time 1755097338.83 total 30.00 last 460.22) - open/close 18300 (time 1755097348.83 total 40.01 last 410.43) - open/close 20000 (time 1755097353.83 total 45.01 last 339.93) - open/close 23970 (time 1755097363.84 total 55.01 last 396.93) - open/close 28873 (time 1755097373.84 total 65.01 last 490.24) - open/close 30000 (time 1755097376.24 total 67.41 last 469.41) - open/close 34166 (time 1755097386.24 total 77.42 last 416.53) - open/close 38379 (time 1755097396.24 total 87.42 last 421.23) - open/close 40000 (time 1755097400.92 total 92.10 last 346.13) - open/close 43964 (time 1755097410.93 total 102.10 last 396.33) - open/close 48225 (time 1755097420.93 total 112.10 last 426.08) - open/close 50000 (time 1755097425.21 total 116.38 last 414.71) - open/close 53916 (time 1755097435.21 total 126.38 last 391.58) - open/close 57863 (time 1755097445.21 total 136.38 last 394.67) - open/close 60000 (time 1755097450.82 total 141.99 last 381.08) - open/close 64512 (time 1755097460.82 total 151.99 last 451.17) total: 64768 open/close in 152.59 seconds: 424.47 ops/second - /unlink 8234 (time 1755097472.79 total 10.00 last 823.40) - /unlink 10000 (time 1755097475.25 total 12.47 last 715.48) - /unlink 18218 (time 1755097485.25 total 22.47 last 821.72) - /unlink 20000 (time 1755097487.30 total 24.52 last 869.57) - /unlink 27729 (time 1755097497.30 total 34.52 last 772.85) - /unlink 30000 (time 1755097500.79 total 38.00 last 652.28) - /unlink 36696 (time 1755097510.79 total 48.00 last 669.53) - /unlink 40000 (time 1755097515.39 total 52.60 last 718.05) - /unlink 46441 (time 1755097525.39 total 62.60 last 644.07) - /unlink 50000 (time 1755097529.64 total 66.86 last 837.20) - /unlink 59032 (time 1755097539.64 total 76.86 last 903.19) total: 64768 /unlink in 83.63 seconds: 774.50 ops/second Waiting for MDT destroys to complete oleg325-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg325-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing unload_modules_local oleg325-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg325-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 PASS 106 (1529s) Stopping clients: oleg325-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg325-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 oleg325-server: oleg325-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg325-server: detected 4 online CPUs by sysfs oleg325-server: MODOPTS_LIBCFS= oleg325-server: Force libcfs to create 2 CPU partitions oleg325-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg325-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1661 sec ========== 11:07:34 (1755097654) === conf-sanity: start cleanup 11:07:35 (1755097655) === === conf-sanity: finish cleanup 11:07:36 (1755097656) ===