-----============= acceptance-small: conf-sanity ============----- Mon Nov 17 12:51:22 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg257-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg257-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg257-server oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server == conf-sanity test 45: long unlink handling in ptlrpcd == 12:54:31 (1763402071) start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg257-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 PASS 45 (262s) == conf-sanity test 69: replace an OST with the same index ========================================================== 12:58:54 (1763402334) start mds service on oleg257-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 842 (time 1763402430.97 total 10.01 last 84.15) - open/close 1659 (time 1763402440.97 total 20.01 last 81.69) - open/close 2510 (time 1763402451.00 total 30.04 last 84.83) - open/close 3267 (time 1763402461.01 total 40.04 last 75.67) - open/close 4055 (time 1763402471.02 total 50.05 last 78.72) - open/close 4908 (time 1763402481.02 total 60.06 last 85.25) - open/close 5802 (time 1763402491.03 total 70.07 last 89.33) - open/close 6630 (time 1763402501.03 total 80.07 last 82.78) - open/close 7416 (time 1763402511.03 total 90.07 last 78.60) - open/close 8163 (time 1763402521.04 total 100.07 last 74.69) - open/close 8932 (time 1763402531.04 total 110.08 last 76.83) - open/close 9813 (time 1763402541.05 total 120.08 last 88.07) total: 10000 open/close in 122.20 seconds: 81.84 ops/second - unlinked 0 (time 1763402546 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 871 (time 1763402624.80 total 10.02 last 86.95) - open/close 1783 (time 1763402634.80 total 20.02 last 91.17) - open/close 2678 (time 1763402644.81 total 30.03 last 89.44) - open/close 3537 (time 1763402654.81 total 40.03 last 85.87) - open/close 4397 (time 1763402664.82 total 50.04 last 85.93) - open/close 5309 (time 1763402674.82 total 60.05 last 91.13) - open/close 6046 (time 1763402684.83 total 70.05 last 73.68) - open/close 6957 (time 1763402694.83 total 80.05 last 91.08) - open/close 7862 (time 1763402704.83 total 90.06 last 90.46) - open/close 8831 (time 1763402714.85 total 100.07 last 96.77) - open/close 9771 (time 1763402724.86 total 110.08 last 93.85) total: 10000 open/close in 112.60 seconds: 88.81 ops/second - unlinked 0 (time 1763402731 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 890 (time 1763402806.84 total 10.01 last 88.95) - open/close 1722 (time 1763402816.85 total 20.01 last 83.13) - open/close 2590 (time 1763402826.85 total 30.02 last 86.72) - open/close 3470 (time 1763402836.86 total 40.03 last 87.95) - open/close 4434 (time 1763402846.87 total 50.04 last 96.33) - open/close 5343 (time 1763402856.87 total 60.04 last 90.90) - open/close 6366 (time 1763402866.89 total 70.05 last 102.12) - open/close 7307 (time 1763402876.90 total 80.07 last 93.99) - open/close 8238 (time 1763402886.90 total 90.07 last 93.04) - open/close 9172 (time 1763402896.91 total 100.08 last 93.37) total: 10000 open/close in 108.88 seconds: 91.84 ops/second - unlinked 0 (time 1763402908 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 790 (time 1763402991.26 total 10.01 last 78.96) - open/close 1608 (time 1763403001.27 total 20.01 last 81.78) - open/close 2554 (time 1763403011.27 total 30.01 last 94.59) - open/close 3465 (time 1763403021.28 total 40.02 last 90.96) - open/close 4384 (time 1763403031.30 total 50.04 last 91.78) - open/close 5316 (time 1763403041.30 total 60.04 last 93.18) - open/close 6325 (time 1763403051.30 total 70.05 last 100.84) - open/close 7204 (time 1763403061.31 total 80.05 last 87.82) - open/close 8160 (time 1763403071.32 total 90.06 last 95.54) - open/close 9172 (time 1763403081.33 total 100.07 last 101.13) total: 10000 open/close in 109.10 seconds: 91.66 ops/second - unlinked 0 (time 1763403093 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 947 (time 1763403169.15 total 10.01 last 94.64) - open/close 1857 (time 1763403179.15 total 20.01 last 90.99) - open/close 2736 (time 1763403189.16 total 30.01 last 87.85) - open/close 3654 (time 1763403199.16 total 40.02 last 91.75) - open/close 4621 (time 1763403209.17 total 50.02 last 96.67) - open/close 5524 (time 1763403219.18 total 60.03 last 90.21) - open/close 6395 (time 1763403229.18 total 70.04 last 87.04) - open/close 7285 (time 1763403239.19 total 80.04 last 88.94) - open/close 8174 (time 1763403249.20 total 90.05 last 88.83) - open/close 9047 (time 1763403259.20 total 100.06 last 87.24) - open/close 9896 (time 1763403269.21 total 110.07 last 84.83) total: 10000 open/close in 111.10 seconds: 90.01 ops/second - unlinked 0 (time 1763403274 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 862 (time 1763403353.31 total 10.02 last 86.07) - open/close 1810 (time 1763403363.33 total 20.03 last 94.66) - open/close 2712 (time 1763403373.33 total 30.04 last 90.13) - open/close 3649 (time 1763403383.34 total 40.05 last 93.62) - open/close 4512 (time 1763403393.35 total 50.05 last 86.27) - open/close 5290 (time 1763403403.36 total 60.06 last 77.72) - open/close 6121 (time 1763403413.36 total 70.07 last 83.04) - open/close 7066 (time 1763403423.37 total 80.07 last 94.48) - open/close 7962 (time 1763403433.37 total 90.07 last 89.58) - open/close 8918 (time 1763403443.37 total 100.07 last 95.59) - open/close 9885 (time 1763403453.38 total 110.08 last 96.63) total: 10000 open/close in 111.43 seconds: 89.74 ops/second - unlinked 0 (time 1763403457 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 895 (time 1763403533.91 total 10.00 last 89.48) - open/close 1784 (time 1763403543.91 total 20.00 last 88.88) - open/close 2657 (time 1763403553.92 total 30.01 last 87.29) - open/close 3498 (time 1763403563.92 total 40.01 last 84.09) - open/close 4365 (time 1763403573.93 total 50.02 last 86.60) - open/close 5187 (time 1763403583.93 total 60.02 last 82.20) - open/close 5997 (time 1763403593.94 total 70.03 last 80.95) - open/close 6799 (time 1763403603.94 total 80.03 last 80.14) - open/close 7601 (time 1763403613.95 total 90.04 last 80.16) - open/close 8421 (time 1763403623.95 total 100.04 last 81.97) - open/close 9260 (time 1763403633.96 total 110.05 last 83.86) total: 10000 open/close in 118.99 seconds: 84.04 ops/second - unlinked 0 (time 1763403645 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 810 (time 1763403731.24 total 10.00 last 80.97) - open/close 1701 (time 1763403741.24 total 20.01 last 89.07) - open/close 2665 (time 1763403751.25 total 30.01 last 96.35) - open/close 3631 (time 1763403761.26 total 40.02 last 96.49) - open/close 4433 (time 1763403771.27 total 50.04 last 80.09) - open/close 5291 (time 1763403781.29 total 60.06 last 85.63) - open/close 6190 (time 1763403791.30 total 70.06 last 89.89) - open/close 7116 (time 1763403801.30 total 80.06 last 92.59) - open/close 8031 (time 1763403811.30 total 90.06 last 91.45) - open/close 8943 (time 1763403821.30 total 100.06 last 91.19) - open/close 9829 (time 1763403831.31 total 110.07 last 88.58) total: 10000 open/close in 111.96 seconds: 89.32 ops/second - unlinked 0 (time 1763403837 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 907 (time 1763403917.83 total 10.01 last 90.62) - open/close 1756 (time 1763403927.83 total 20.01 last 84.89) - open/close 2567 (time 1763403937.83 total 30.01 last 81.07) - open/close 3363 (time 1763403947.85 total 40.03 last 79.43) - open/close 4171 (time 1763403957.86 total 50.04 last 80.74) - open/close 5037 (time 1763403967.87 total 60.04 last 86.57) - open/close 5941 (time 1763403977.87 total 70.05 last 90.32) - open/close 6899 (time 1763403987.88 total 80.06 last 95.76) - open/close 7775 (time 1763403997.89 total 90.07 last 87.50) - open/close 8570 (time 1763404007.90 total 100.08 last 79.40) - open/close 9459 (time 1763404017.90 total 110.08 last 88.87) total: 10000 open/close in 115.69 seconds: 86.44 ops/second - unlinked 0 (time 1763404027 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 935 (time 1763404104.95 total 10.01 last 93.42) - open/close 1780 (time 1763404114.96 total 20.02 last 84.45) - open/close 2631 (time 1763404124.96 total 30.02 last 85.02) - open/close 3565 (time 1763404134.97 total 40.03 last 93.36) - open/close 4498 (time 1763404144.97 total 50.03 last 93.29) - open/close 5447 (time 1763404154.97 total 60.03 last 94.89) - open/close 6336 (time 1763404164.98 total 70.04 last 88.84) - open/close 7230 (time 1763404174.98 total 80.04 last 89.35) - open/close 8177 (time 1763404184.98 total 90.04 last 94.70) - open/close 9055 (time 1763404194.99 total 100.05 last 87.78) - open/close 9946 (time 1763404204.99 total 110.05 last 89.07) total: 10000 open/close in 110.64 seconds: 90.39 ops/second - unlinked 0 (time 1763404209 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg257-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 PASS 69 (2070s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 13:33:25 (1763404405) oleg257-server: debugfs 1.47.3-wc1 (8-Aug-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg257-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server SKIP 111 (193s) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: MODOPTS_LIBCFS= oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2764 sec ========== 13:37:28 (1763404648) === conf-sanity: start cleanup 13:37:30 (1763404650) === === conf-sanity: finish cleanup 13:37:33 (1763404653) ===