-----============= acceptance-small: conf-sanity ============----- Thu Sep 25 14:08:47 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg259-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg259-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg259-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg259-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg259-server oleg259-server: oleg259-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg259-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg259-server: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg259-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server == conf-sanity test 45: long unlink handling in ptlrpcd == 14:10:03 (1758823803) start mds service on oleg259-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg259-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 PASS 45 (143s) == conf-sanity test 69: replace an OST with the same index ========================================================== 14:12:26 (1758823946) start mds service on oleg259-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions oleg259-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg259-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1822 (time 1758823992.53 total 10.00 last 182.20) - open/close 3654 (time 1758824002.53 total 20.00 last 183.17) - open/close 6291 (time 1758824012.54 total 30.01 last 263.52) - open/close 8511 (time 1758824022.54 total 40.01 last 222.00) total: 10000 open/close in 48.35 seconds: 206.81 ops/second - unlinked 0 (time 1758824033 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1037 (time 1758824088.04 total 10.01 last 103.62) - open/close 2299 (time 1758824098.04 total 20.01 last 126.16) - open/close 3418 (time 1758824108.05 total 30.02 last 111.85) - open/close 4595 (time 1758824118.06 total 40.02 last 117.59) - open/close 5942 (time 1758824128.07 total 50.03 last 134.60) - open/close 7931 (time 1758824138.07 total 60.03 last 198.88) - open/close 9729 (time 1758824148.07 total 70.04 last 179.77) total: 10000 open/close in 71.27 seconds: 140.31 ops/second - unlinked 0 (time 1758824150 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1684 (time 1758824193.26 total 10.01 last 168.31) - open/close 3136 (time 1758824203.26 total 20.01 last 145.12) - open/close 4694 (time 1758824213.26 total 30.01 last 155.77) - open/close 6262 (time 1758824223.27 total 40.02 last 156.65) - open/close 7794 (time 1758824233.28 total 50.02 last 153.18) - open/close 9265 (time 1758824243.28 total 60.03 last 147.03) total: 10000 open/close in 65.07 seconds: 153.67 ops/second - unlinked 0 (time 1758824250 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1924 (time 1758824293.76 total 10.01 last 192.24) - open/close 3719 (time 1758824303.76 total 20.01 last 179.45) - open/close 5620 (time 1758824313.77 total 30.01 last 190.07) - open/close 7514 (time 1758824323.77 total 40.01 last 189.37) - open/close 9211 (time 1758824333.77 total 50.01 last 169.70) total: 10000 open/close in 53.92 seconds: 185.46 ops/second - unlinked 0 (time 1758824339 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1891 (time 1758824377.04 total 10.00 last 189.07) - open/close 4316 (time 1758824387.04 total 20.00 last 242.45) - open/close 6664 (time 1758824397.04 total 30.01 last 234.75) - open/close 8965 (time 1758824407.04 total 40.01 last 230.05) total: 10000 open/close in 44.97 seconds: 222.36 ops/second - unlinked 0 (time 1758824413 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2512 (time 1758824449.56 total 10.00 last 251.13) - open/close 4849 (time 1758824459.56 total 20.00 last 233.68) - open/close 6906 (time 1758824469.56 total 30.01 last 205.64) - open/close 9620 (time 1758824479.57 total 40.01 last 271.33) total: 10000 open/close in 41.79 seconds: 239.31 ops/second - unlinked 0 (time 1758824482 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2701 (time 1758824518.59 total 10.00 last 270.03) - open/close 4680 (time 1758824528.59 total 20.00 last 197.87) - open/close 6447 (time 1758824538.59 total 30.01 last 176.68) - open/close 8342 (time 1758824548.59 total 40.01 last 189.47) - open/close 9986 (time 1758824558.59 total 50.01 last 164.37) total: 10000 open/close in 50.08 seconds: 199.67 ops/second - unlinked 0 (time 1758824560 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2141 (time 1758824595.88 total 10.00 last 214.08) - open/close 4718 (time 1758824605.88 total 20.00 last 257.64) - open/close 7463 (time 1758824615.88 total 30.01 last 274.39) total: 10000 open/close in 39.85 seconds: 250.92 ops/second - unlinked 0 (time 1758824627 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2113 (time 1758824663.27 total 10.00 last 211.29) - open/close 4863 (time 1758824673.27 total 20.00 last 274.94) - open/close 7566 (time 1758824683.28 total 30.00 last 270.28) total: 10000 open/close in 39.36 seconds: 254.09 ops/second - unlinked 0 (time 1758824694 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 2903 (time 1758824724.39 total 10.00 last 290.28) - open/close 6144 (time 1758824734.39 total 20.00 last 324.09) total: 10000 open/close in 29.23 seconds: 342.15 ops/second - unlinked 0 (time 1758824744 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.159@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.159@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg259-server: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg259-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.159@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.159@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.159@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.159@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.159@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg259-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg259-server: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg259-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 PASS 69 (886s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 14:27:13 (1758824833) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg259-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg259-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 oleg259-server: oleg259-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg259-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions oleg259-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg259-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1132 sec ========== 14:27:39 (1758824859) === conf-sanity: start cleanup 14:27:39 (1758824859) === === conf-sanity: finish cleanup 14:27:40 (1758824860) ===