-----============= acceptance-small: conf-sanity ============----- Fri Jan 9 20:24:52 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg403-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg403-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg403-server oleg403-server: oleg403-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server == conf-sanity test 45: long unlink handling in ptlrpcd == 20:27:05 (1768008425) start mds service on oleg403-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg403-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 PASS 45 (193s) == conf-sanity test 69: replace an OST with the same index ========================================================== 20:30:19 (1768008619) start mds service on oleg403-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 1469 (time 1768008677.61 total 10.00 last 146.85) - open/close 3026 (time 1768008687.62 total 20.01 last 155.58) - open/close 4517 (time 1768008697.64 total 30.02 last 148.90) - open/close 6183 (time 1768008707.64 total 40.03 last 166.57) - open/close 7648 (time 1768008717.64 total 50.03 last 146.47) - open/close 9328 (time 1768008727.65 total 60.04 last 167.88) total: 10000 open/close in 62.96 seconds: 158.83 ops/second - unlinked 0 (time 1768008732 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1584 (time 1768008775.24 total 10.00 last 158.38) - open/close 3044 (time 1768008785.24 total 20.00 last 145.99) - open/close 4642 (time 1768008795.25 total 30.01 last 159.64) - open/close 6354 (time 1768008805.25 total 40.02 last 171.15) - open/close 7745 (time 1768008815.25 total 50.02 last 139.06) - open/close 9206 (time 1768008825.26 total 60.02 last 146.03) total: 10000 open/close in 64.70 seconds: 154.57 ops/second - unlinked 0 (time 1768008831 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1392 (time 1768008879.64 total 10.01 last 139.09) - open/close 3031 (time 1768008889.64 total 20.01 last 163.89) - open/close 4404 (time 1768008899.65 total 30.02 last 137.13) - open/close 5781 (time 1768008909.65 total 40.02 last 137.69) - open/close 7045 (time 1768008919.66 total 50.03 last 126.28) - open/close 8860 (time 1768008929.66 total 60.03 last 181.47) total: 10000 open/close in 67.05 seconds: 149.15 ops/second - unlinked 0 (time 1768008938 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1601 (time 1768008985.16 total 10.01 last 159.99) - open/close 3255 (time 1768008995.16 total 20.01 last 165.39) - open/close 5142 (time 1768009005.17 total 30.01 last 188.62) - open/close 6945 (time 1768009015.17 total 40.01 last 180.27) - open/close 8614 (time 1768009025.18 total 50.02 last 166.77) total: 10000 open/close in 56.72 seconds: 176.32 ops/second - unlinked 0 (time 1768009033 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2110 (time 1768009070.82 total 10.01 last 210.87) - open/close 4251 (time 1768009080.82 total 20.01 last 214.01) - open/close 6775 (time 1768009090.82 total 30.01 last 252.40) - open/close 9243 (time 1768009100.82 total 40.01 last 246.78) total: 10000 open/close in 43.06 seconds: 232.24 ops/second - unlinked 0 (time 1768009105 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2032 (time 1768009143.83 total 10.00 last 203.12) - open/close 4653 (time 1768009153.84 total 20.01 last 261.97) - open/close 6818 (time 1768009163.84 total 30.01 last 216.44) - open/close 8740 (time 1768009173.84 total 40.01 last 192.16) total: 10000 open/close in 46.34 seconds: 215.80 ops/second - unlinked 0 (time 1768009181 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1841 (time 1768009219.39 total 10.00 last 184.04) - open/close 3805 (time 1768009229.39 total 20.00 last 196.37) - open/close 6057 (time 1768009239.39 total 30.01 last 225.18) - open/close 8417 (time 1768009249.39 total 40.01 last 235.88) total: 10000 open/close in 46.39 seconds: 215.57 ops/second - unlinked 0 (time 1768009257 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2198 (time 1768009293.14 total 10.00 last 219.80) - open/close 5260 (time 1768009303.14 total 20.00 last 306.18) - open/close 7810 (time 1768009313.14 total 30.00 last 254.97) total: 10000 open/close in 38.94 seconds: 256.82 ops/second - unlinked 0 (time 1768009323 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2820 (time 1768009354.15 total 10.00 last 281.91) - open/close 5685 (time 1768009364.15 total 20.00 last 286.48) - open/close 8735 (time 1768009374.15 total 30.01 last 304.96) total: 10000 open/close in 34.76 seconds: 287.70 ops/second - unlinked 0 (time 1768009380 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 3009 (time 1768009407.80 total 10.00 last 300.87) - open/close 5697 (time 1768009417.80 total 20.00 last 268.75) - open/close 9211 (time 1768009427.80 total 30.00 last 351.36) total: 10000 open/close in 32.24 seconds: 310.19 ops/second - unlinked 0 (time 1768009431 ; total 0 ; last 0) total: 10000 unlinks in 13 seconds: 769.230774 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 15 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre On OST0, 12245 used inodes rc=0 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.103@tcp sys.timeout=20 Writing CONFIGS/mountdata start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 PASS 69 (913s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 20:45:33 (1768009533) oleg403-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: oleg403-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) oleg403-server: oleg403-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server SKIP 111 (85s) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) oleg403-server: oleg403-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1345 sec ========== 20:47:18 (1768009638) === conf-sanity: start cleanup 20:47:19 (1768009639) === === conf-sanity: finish cleanup 20:47:20 (1768009640) ===