-----============= acceptance-small: conf-sanity ============----- Tue Nov 25 16:26:37 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg331-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg331-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server oleg331-server: oleg331-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:28:08 (1764106088) start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg331-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 PASS 45 (147s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:30:35 (1764106235) start mds service on oleg331-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1756 (time 1764106281.84 total 10.00 last 175.56) - open/close 3345 (time 1764106291.85 total 20.01 last 158.74) - open/close 4874 (time 1764106301.85 total 30.02 last 152.83) - open/close 6495 (time 1764106311.86 total 40.02 last 162.08) - open/close 7914 (time 1764106321.86 total 50.02 last 141.89) - open/close 9456 (time 1764106331.86 total 60.02 last 154.17) total: 10000 open/close in 62.54 seconds: 159.90 ops/second - unlinked 0 (time 1764106336 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 2156 (time 1764106374.84 total 10.00 last 215.50) - open/close 4215 (time 1764106384.84 total 20.01 last 205.88) - open/close 6232 (time 1764106394.85 total 30.01 last 201.61) - open/close 8225 (time 1764106404.85 total 40.02 last 199.20) total: 10000 open/close in 48.29 seconds: 207.08 ops/second - unlinked 0 (time 1764106414 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1953 (time 1764106458.77 total 10.00 last 195.26) - open/close 4283 (time 1764106468.77 total 20.01 last 232.89) - open/close 7151 (time 1764106478.78 total 30.01 last 286.69) - open/close 8913 (time 1764106488.78 total 40.01 last 176.14) total: 10000 open/close in 44.43 seconds: 225.08 ops/second - unlinked 0 (time 1764106494 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2484 (time 1764106529.77 total 10.01 last 248.16) - open/close 4659 (time 1764106539.77 total 20.01 last 217.42) - open/close 6067 (time 1764106549.78 total 30.02 last 140.76) - open/close 7742 (time 1764106559.78 total 40.02 last 167.41) - open/close 9249 (time 1764106569.78 total 50.02 last 150.69) total: 10000 open/close in 55.10 seconds: 181.49 ops/second - unlinked 0 (time 1764106576 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 2321 (time 1764106618.80 total 10.00 last 232.09) - open/close 4699 (time 1764106628.80 total 20.00 last 237.79) - open/close 7658 (time 1764106638.81 total 30.00 last 295.83) total: 10000 open/close in 38.08 seconds: 262.62 ops/second - unlinked 0 (time 1764106648 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2918 (time 1764106677.58 total 10.00 last 291.73) - open/close 5977 (time 1764106687.58 total 20.00 last 305.88) - open/close 9133 (time 1764106697.58 total 30.00 last 315.59) total: 10000 open/close in 33.63 seconds: 297.34 ops/second - unlinked 0 (time 1764106702 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3485 (time 1764106732.04 total 10.00 last 348.47) - open/close 7253 (time 1764106742.04 total 20.00 last 376.77) total: 10000 open/close in 26.87 seconds: 372.10 ops/second - unlinked 0 (time 1764106750 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3600 (time 1764106780.30 total 10.00 last 359.94) - open/close 6920 (time 1764106790.30 total 20.00 last 331.99) total: 10000 open/close in 29.93 seconds: 334.10 ops/second - unlinked 0 (time 1764106801 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2598 (time 1764106838.25 total 10.00 last 259.78) - open/close 4809 (time 1764106848.25 total 20.00 last 221.09) - open/close 7241 (time 1764106858.25 total 30.00 last 243.18) - open/close 9890 (time 1764106868.25 total 40.00 last 264.88) total: 10000 open/close in 40.52 seconds: 246.77 ops/second - unlinked 0 (time 1764106870 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2921 (time 1764106904.12 total 10.00 last 292.04) - open/close 6020 (time 1764106914.12 total 20.00 last 309.82) - open/close 8374 (time 1764106924.13 total 30.02 last 235.11) - open/close 9142 (time 1764106934.14 total 40.03 last 76.72) - open/close 9888 (time 1764106944.16 total 50.04 last 74.50) total: 10000 open/close in 52.33 seconds: 191.11 ops/second - unlinked 0 (time 1764106951 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.131@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.131@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre On OST0, 10493 used inodes rc=0 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.131@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.131@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.131@tcp sys.timeout=20 autodegrade=on Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.131@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.131@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg331-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 69 (912s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:45:47 (1764107147) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (3s) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 oleg331-server: oleg331-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg331-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1199 sec ========== 16:46:37 (1764107197) === conf-sanity: start cleanup 16:46:39 (1764107199) === === conf-sanity: finish cleanup 16:46:41 (1764107201) ===