-----============= acceptance-small: conf-sanity ============----- Sun Nov 16 18:32:47 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg206-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg206-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg206-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg206-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg206-server oleg206-server: oleg206-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg206-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server == conf-sanity test 45: long unlink handling in ptlrpcd == 18:35:24 (1763336124) start mds service on oleg206-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg206-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg206-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 PASS 45 (259s) == conf-sanity test 69: replace an OST with the same index ========================================================== 18:39:43 (1763336383) start mds service on oleg206-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg206-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 760 (time 1763336477.65 total 10.01 last 75.94) - open/close 1565 (time 1763336487.66 total 20.02 last 80.44) - open/close 2361 (time 1763336497.67 total 30.02 last 79.54) - open/close 3134 (time 1763336507.67 total 40.02 last 77.28) - open/close 3885 (time 1763336517.67 total 50.03 last 75.06) - open/close 4696 (time 1763336527.68 total 60.04 last 81.04) - open/close 5442 (time 1763336537.69 total 70.04 last 74.57) - open/close 6218 (time 1763336547.69 total 80.04 last 77.58) - open/close 7040 (time 1763336557.70 total 90.05 last 82.12) - open/close 7827 (time 1763336567.71 total 100.06 last 78.64) - open/close 8604 (time 1763336577.71 total 110.07 last 77.66) - open/close 9360 (time 1763336587.72 total 120.08 last 75.52) total: 10000 open/close in 129.22 seconds: 77.39 ops/second - unlinked 0 (time 1763336600 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 877 (time 1763336691.08 total 10.01 last 87.59) - open/close 1691 (time 1763336701.08 total 20.02 last 81.37) - open/close 2419 (time 1763336711.09 total 30.03 last 72.72) - open/close 3180 (time 1763336721.10 total 40.03 last 76.06) - open/close 3917 (time 1763336731.10 total 50.04 last 73.66) - open/close 4769 (time 1763336741.11 total 60.04 last 85.17) - open/close 5543 (time 1763336751.12 total 70.06 last 77.28) - open/close 6179 (time 1763336761.14 total 80.07 last 63.49) - open/close 6826 (time 1763336771.16 total 90.09 last 64.59) - open/close 7542 (time 1763336781.17 total 100.10 last 71.52) - open/close 8311 (time 1763336791.17 total 110.11 last 76.87) - open/close 9131 (time 1763336801.17 total 120.11 last 81.98) - open/close 9998 (time 1763336811.18 total 130.11 last 86.67) total: 10000 open/close in 130.12 seconds: 76.85 ops/second - unlinked 0 (time 1763336813 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second - open/close 829 (time 1763336896.98 total 10.01 last 82.82) - open/close 1430 (time 1763336906.99 total 20.02 last 60.05) - open/close 2043 (time 1763336916.99 total 30.02 last 61.29) - open/close 2861 (time 1763336927.00 total 40.03 last 81.70) - open/close 3638 (time 1763336937.01 total 50.03 last 77.69) - open/close 4439 (time 1763336947.01 total 60.04 last 80.07) - open/close 5222 (time 1763336957.01 total 70.04 last 78.26) - open/close 5980 (time 1763336967.02 total 80.05 last 75.72) - open/close 6701 (time 1763336977.03 total 90.05 last 72.09) - open/close 7495 (time 1763336987.04 total 100.07 last 79.28) - open/close 8266 (time 1763336997.04 total 110.07 last 77.09) - open/close 9068 (time 1763337007.06 total 120.08 last 80.10) - open/close 9846 (time 1763337017.06 total 130.09 last 77.73) total: 10000 open/close in 132.03 seconds: 75.74 ops/second - unlinked 0 (time 1763337022 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 820 (time 1763337104.81 total 10.01 last 81.92) - open/close 1593 (time 1763337114.82 total 20.02 last 77.25) - open/close 2410 (time 1763337124.82 total 30.02 last 81.70) - open/close 3212 (time 1763337134.83 total 40.03 last 80.11) - open/close 4028 (time 1763337144.83 total 50.03 last 81.59) - open/close 4853 (time 1763337154.83 total 60.03 last 82.49) - open/close 5727 (time 1763337164.84 total 70.04 last 87.31) - open/close 6586 (time 1763337174.86 total 80.06 last 85.77) - open/close 7439 (time 1763337184.86 total 90.06 last 85.28) - open/close 8254 (time 1763337194.86 total 100.06 last 81.50) - open/close 9035 (time 1763337204.87 total 110.06 last 78.06) - open/close 9818 (time 1763337214.87 total 120.07 last 78.23) total: 10000 open/close in 122.22 seconds: 81.82 ops/second - unlinked 0 (time 1763337220 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 861 (time 1763337310.15 total 10.01 last 86.05) - open/close 1722 (time 1763337320.15 total 20.01 last 86.06) - open/close 2588 (time 1763337330.17 total 30.03 last 86.47) - open/close 3393 (time 1763337340.18 total 40.04 last 80.42) - open/close 4197 (time 1763337350.18 total 50.04 last 80.39) - open/close 4970 (time 1763337360.18 total 60.04 last 77.29) - open/close 5695 (time 1763337370.19 total 70.04 last 72.46) - open/close 6519 (time 1763337380.20 total 80.06 last 82.30) - open/close 7318 (time 1763337390.20 total 90.06 last 79.89) - open/close 8088 (time 1763337400.20 total 100.06 last 76.95) - open/close 8911 (time 1763337410.21 total 110.07 last 82.25) - open/close 9688 (time 1763337420.21 total 120.07 last 77.69) total: 10000 open/close in 123.87 seconds: 80.73 ops/second - unlinked 0 (time 1763337427 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 711 (time 1763337518.35 total 10.01 last 71.04) - open/close 1517 (time 1763337528.36 total 20.02 last 80.47) - open/close 2227 (time 1763337538.37 total 30.03 last 70.96) - open/close 2961 (time 1763337548.38 total 40.04 last 73.34) - open/close 3625 (time 1763337558.39 total 50.05 last 66.32) - open/close 4311 (time 1763337568.40 total 60.06 last 68.50) - open/close 5024 (time 1763337578.42 total 70.08 last 71.21) - open/close 5727 (time 1763337588.43 total 80.09 last 70.23) - open/close 6406 (time 1763337598.44 total 90.10 last 67.83) - open/close 7133 (time 1763337608.44 total 100.10 last 72.67) - open/close 7853 (time 1763337618.45 total 110.11 last 71.97) - open/close 8622 (time 1763337628.46 total 120.12 last 76.81) - open/close 9449 (time 1763337638.46 total 130.12 last 82.65) total: 10000 open/close in 136.70 seconds: 73.15 ops/second - unlinked 0 (time 1763337649 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 790 (time 1763337741.30 total 10.01 last 78.96) - open/close 1508 (time 1763337751.31 total 20.02 last 71.71) - open/close 2220 (time 1763337761.32 total 30.03 last 71.14) - open/close 3053 (time 1763337771.32 total 40.03 last 83.30) - open/close 3851 (time 1763337781.32 total 50.03 last 79.76) - open/close 4620 (time 1763337791.33 total 60.04 last 76.85) - open/close 5400 (time 1763337801.33 total 70.04 last 77.99) - open/close 6226 (time 1763337811.33 total 80.04 last 82.59) - open/close 6988 (time 1763337821.33 total 90.04 last 76.18) - open/close 7757 (time 1763337831.34 total 100.05 last 76.82) - open/close 8573 (time 1763337841.35 total 110.06 last 81.55) - open/close 9302 (time 1763337851.36 total 120.06 last 72.86) total: 10000 open/close in 128.78 seconds: 77.65 ops/second - unlinked 0 (time 1763337863 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 803 (time 1763337958.06 total 10.01 last 80.23) - open/close 1447 (time 1763337968.07 total 20.02 last 64.31) - open/close 2152 (time 1763337978.07 total 30.03 last 70.48) - open/close 2841 (time 1763337988.08 total 40.03 last 68.87) - open/close 3578 (time 1763337998.10 total 50.05 last 73.56) - open/close 4319 (time 1763338008.12 total 60.07 last 73.92) - open/close 5084 (time 1763338018.12 total 70.08 last 76.48) - open/close 5879 (time 1763338028.12 total 80.08 last 79.50) - open/close 6613 (time 1763338038.13 total 90.08 last 73.36) - open/close 7385 (time 1763338048.14 total 100.09 last 77.14) - open/close 8136 (time 1763338058.14 total 110.09 last 75.10) - open/close 8927 (time 1763338068.15 total 120.10 last 79.02) - open/close 9754 (time 1763338078.15 total 130.10 last 82.68) total: 10000 open/close in 133.46 seconds: 74.93 ops/second - unlinked 0 (time 1763338085 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 792 (time 1763338172.33 total 10.01 last 79.13) - open/close 1529 (time 1763338182.33 total 20.02 last 73.65) - open/close 2299 (time 1763338192.34 total 30.02 last 76.97) - open/close 3099 (time 1763338202.34 total 40.02 last 79.99) - open/close 3868 (time 1763338212.35 total 50.04 last 76.78) - open/close 4654 (time 1763338222.35 total 60.04 last 78.57) - open/close 5436 (time 1763338232.36 total 70.04 last 78.16) - open/close 6208 (time 1763338242.37 total 80.06 last 77.11) - open/close 6955 (time 1763338252.37 total 90.06 last 74.70) - open/close 7751 (time 1763338262.38 total 100.06 last 79.55) - open/close 8601 (time 1763338272.40 total 110.08 last 84.84) - open/close 9392 (time 1763338282.41 total 120.09 last 79.01) total: 10000 open/close in 127.36 seconds: 78.52 ops/second - unlinked 0 (time 1763338293 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 730 (time 1763338378.81 total 10.01 last 72.94) - open/close 1503 (time 1763338388.81 total 20.01 last 77.29) - open/close 2242 (time 1763338398.82 total 30.01 last 73.85) - open/close 3050 (time 1763338408.82 total 40.02 last 80.78) - open/close 3808 (time 1763338418.83 total 50.02 last 75.74) - open/close 4659 (time 1763338428.83 total 60.03 last 85.04) - open/close 5466 (time 1763338438.84 total 70.04 last 80.62) - open/close 6303 (time 1763338448.85 total 80.05 last 83.64) - open/close 7118 (time 1763338458.85 total 90.05 last 81.47) - open/close 7894 (time 1763338468.86 total 100.05 last 77.59) - open/close 8665 (time 1763338478.86 total 110.06 last 77.07) - open/close 9381 (time 1763338488.86 total 120.06 last 71.59) total: 10000 open/close in 129.51 seconds: 77.22 ops/second - unlinked 0 (time 1763338503 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.106@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.106@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.106@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg206-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 PASS 69 (2324s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 19:18:27 (1763338707) oleg206-server: debugfs 1.47.3-wc1 (8-Aug-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg206-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg206-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 oleg206-server: oleg206-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg206-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg206-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg206-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg206-client.virtnet /mnt/lustre2 (opts:-f) oleg206-server: oleg206-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg206-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg206-server: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg206-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server SKIP 111 (183s) Stopping clients: oleg206-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg206-client.virtnet /mnt/lustre2 (opts:-f) oleg206-server: oleg206-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2974 sec ========== 19:22:22 (1763338942) === conf-sanity: start cleanup 19:22:24 (1763338944) === === conf-sanity: finish cleanup 19:22:27 (1763338947) ===