-----============= acceptance-small: conf-sanity ============----- Mon Dec 29 13:57:02 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg318-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg318-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg318-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg318-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg318-server oleg318-server: oleg318-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg318-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg318-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server == conf-sanity test 45: long unlink handling in ptlrpcd == 13:58:29 (1767034709) start mds service on oleg318-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg318-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 PASS 45 (151s) == conf-sanity test 69: replace an OST with the same index ========================================================== 14:01:00 (1767034860) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1197 (time 1767034909.53 total 10.00 last 119.66) - open/close 2595 (time 1767034919.53 total 20.01 last 139.78) - open/close 4120 (time 1767034929.54 total 30.01 last 152.44) - open/close 5437 (time 1767034939.54 total 40.01 last 131.69) - open/close 6550 (time 1767034949.54 total 50.01 last 111.29) - open/close 7928 (time 1767034959.54 total 60.01 last 137.76) - open/close 9659 (time 1767034969.54 total 70.01 last 173.07) total: 10000 open/close in 71.87 seconds: 139.14 ops/second - unlinked 0 (time 1767034973 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1984 (time 1767035015.31 total 10.00 last 198.34) - open/close 4085 (time 1767035025.31 total 20.00 last 210.10) - open/close 6355 (time 1767035035.32 total 30.01 last 226.92) - open/close 8802 (time 1767035045.32 total 40.01 last 244.64) total: 10000 open/close in 46.19 seconds: 216.51 ops/second - unlinked 0 (time 1767035053 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 2003 (time 1767035092.87 total 10.00 last 200.24) - open/close 4154 (time 1767035102.87 total 20.00 last 215.06) - open/close 6782 (time 1767035112.87 total 30.01 last 262.76) - open/close 9042 (time 1767035122.87 total 40.01 last 225.96) total: 10000 open/close in 43.98 seconds: 227.40 ops/second - unlinked 0 (time 1767035128 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1922 (time 1767035171.24 total 10.00 last 192.18) - open/close 3953 (time 1767035181.24 total 20.00 last 203.04) - open/close 5785 (time 1767035191.24 total 30.01 last 183.12) - open/close 8064 (time 1767035201.25 total 40.01 last 227.81) total: 10000 open/close in 48.08 seconds: 207.97 ops/second - unlinked 0 (time 1767035210 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2287 (time 1767035244.95 total 10.00 last 228.64) - open/close 4269 (time 1767035254.95 total 20.00 last 198.19) - open/close 5999 (time 1767035264.96 total 30.00 last 172.98) - open/close 8163 (time 1767035274.96 total 40.01 last 216.32) total: 10000 open/close in 49.89 seconds: 200.46 ops/second - unlinked 0 (time 1767035286 ; total 0 ; last 0) total: 10000 unlinks in 29 seconds: 344.827576 unlinks/second - open/close 2113 (time 1767035327.43 total 10.00 last 211.28) - open/close 4244 (time 1767035337.43 total 20.00 last 213.09) - open/close 6417 (time 1767035347.44 total 30.01 last 217.18) - open/close 8252 (time 1767035357.44 total 40.01 last 183.44) total: 10000 open/close in 48.11 seconds: 207.85 ops/second - unlinked 0 (time 1767035367 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2902 (time 1767035403.04 total 10.00 last 290.16) - open/close 5509 (time 1767035413.05 total 20.00 last 260.62) - open/close 7993 (time 1767035423.05 total 30.01 last 248.37) total: 10000 open/close in 37.29 seconds: 268.20 ops/second - unlinked 0 (time 1767035431 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3078 (time 1767035463.27 total 10.00 last 307.71) - open/close 5601 (time 1767035473.28 total 20.01 last 252.19) - open/close 8525 (time 1767035483.28 total 30.01 last 292.33) total: 10000 open/close in 34.76 seconds: 287.68 ops/second - unlinked 0 (time 1767035489 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2290 (time 1767035519.73 total 10.00 last 228.93) - open/close 4698 (time 1767035529.73 total 20.00 last 240.77) - open/close 7373 (time 1767035539.74 total 30.01 last 267.43) total: 10000 open/close in 38.84 seconds: 257.48 ops/second - unlinked 0 (time 1767035550 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2589 (time 1767035586.49 total 10.00 last 258.81) - open/close 4998 (time 1767035596.49 total 20.01 last 240.86) - open/close 7903 (time 1767035606.49 total 30.01 last 290.42) total: 10000 open/close in 36.62 seconds: 273.08 ops/second - unlinked 0 (time 1767035614 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.118@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg318-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre On OST0, 18749 used inodes rc=0 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.118@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.118@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.118@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.118@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.203.118@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg318-server: oleg318-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg318-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 PASS 69 (832s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 14:14:53 (1767035693) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg318-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg318-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 oleg318-server: oleg318-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1097 sec ========== 14:15:20 (1767035720) === conf-sanity: start cleanup 14:15:21 (1767035721) === === conf-sanity: finish cleanup 14:15:22 (1767035722) ===